bvandeusenandClaude Opus 5.5
3c24c633df
- One run per host: start_run claims every host it touches (addresses read from the inventory Steward wrote; a source's own inventory file is claimed as that file) before it takes a run slot. The claim is all-or-nothing. A run waiting on a host shows queued and names the run it waits for, honours a cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run. - A due schedule whose previous run is still queued or running skips the firing: last_run_at moves on, last_run_id still names the run going, and last_error says "Skipped: the previous run was still going". The Schedules page shows that beside the run (muted), and a launch error in red. - An alert rule that fires while the action run from an earlier firing is still going starts no new run; the event says so and names the run. An in-process set closes the gap between the check and the run being linked. - docs/core/ansible.md: queued/cancelled statuses and the overlap rules. Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing, cancel, deadline) and tests/integration/test_ansible_overlap.py (two real runs on one host take turns; schedule skip then fire; alert action skip then start). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
151 lines
5.6 KiB
Markdown
151 lines
5.6 KiB
Markdown
# Ansible Integration
|
|
|
|
Steward can browse, trigger, and stream output from Ansible playbooks directly from the web UI. Runs execute as asyncio tasks inside the same process — no Celery, no external workers.
|
|
|
|
---
|
|
|
|
## Playbook Sources
|
|
|
|
Sources are configured at `/settings/` under Ansible. Two source types are supported:
|
|
|
|
### Local Filesystem
|
|
|
|
Points to a directory on the host that already contains playbooks.
|
|
|
|
```yaml
|
|
# In app_settings (configured via UI)
|
|
ansible.sources:
|
|
- name: "homelab"
|
|
type: local
|
|
path: "/opt/playbooks"
|
|
```
|
|
|
|
### Git Repository
|
|
|
|
The app clones or pulls the repo into a local cache directory on a configurable schedule. All execution uses the local cache.
|
|
|
|
```yaml
|
|
ansible.sources:
|
|
- name: "infra-repo"
|
|
type: git
|
|
url: "https://github.com/example/infra.git"
|
|
branch: "main"
|
|
pull_interval_seconds: 300
|
|
```
|
|
|
|
Each git source is checked out under `/data/ansible/sources/<name>` (override with `STEWARD_ANSIBLE_SOURCE_DIR`), on the persistent volume, so a redeploy keeps it. One scheduled task, `ansible_git_pulls`, ticks every 30 s and pulls each source once its `pull_interval_seconds` has passed; a source added in Settings is pulled within a tick, with no restart. A clone or pull that runs past 120 s is cut (its process group terminated), a clone that fails is removed so the next attempt starts clean, and git never prompts for credentials.
|
|
|
|
---
|
|
|
|
## Inventory Discovery
|
|
|
|
The app discovers inventory files within the root of the playbook source directory (non-recursive). It looks for files named:
|
|
|
|
- `hosts`
|
|
- `inventory`
|
|
- `inventory.yml`
|
|
- `inventory.ini`
|
|
|
|
A manual relative path can also be entered in the UI (e.g. `inventories/production/hosts`) for inventories in subdirectories.
|
|
|
|
---
|
|
|
|
## Triggering a Run
|
|
|
|
From the UI at `/ansible/`, you can:
|
|
|
|
1. Browse available playbooks across all sources
|
|
2. View playbook contents before running
|
|
3. Select an inventory file
|
|
4. Trigger a run (requires `operator` or `viewer` role — execution is restricted to `operator`/`admin`)
|
|
|
|
The run flow:
|
|
|
|
1. UI submits `POST /ansible/runs` with `playbook_path`, `source_name`, and `inventory_path`
|
|
2. An `AnsibleRun` row is created with `status = running`
|
|
3. Playbook execution starts as `asyncio.create_task()`
|
|
4. The response returns the `run_id` and an HTMX partial that wires the SSE subscription
|
|
5. The browser connects to `GET /ansible/runs/<run_id>/stream` to receive live output
|
|
|
|
---
|
|
|
|
## SSE Streaming
|
|
|
|
Run output is streamed to the browser via Server-Sent Events (SSE) at:
|
|
|
|
```
|
|
GET /ansible/runs/<run_id>/stream
|
|
```
|
|
|
|
Each output line is sent as:
|
|
|
|
```
|
|
event: output
|
|
data: <line of stdout/stderr>
|
|
```
|
|
|
|
Run completion:
|
|
|
|
```
|
|
event: done
|
|
data: success|failed|interrupted
|
|
```
|
|
|
|
Output is flushed to the `ansible_runs.output` DB column every 50 lines or every 5 seconds (whichever comes first) and always on completion. This means partial output survives a process crash.
|
|
|
|
If a client connects after the run has already completed, the endpoint immediately sends `event: done` with the final status and closes the stream. To view stored output, use the run history view at `GET /ansible/runs/<run_id>`.
|
|
|
|
---
|
|
|
|
## Run Lifecycle
|
|
|
|
| Status | Description |
|
|
|---|---|
|
|
| `queued` | Waiting for a host another run holds, or for a run slot |
|
|
| `running` | Execution in progress |
|
|
| `success` | Playbook exited 0 |
|
|
| `failed` | Playbook exited non-zero |
|
|
| `interrupted` | App restarted while run was in progress |
|
|
| `cancelled` | Stopped by an operator, or by Steward shutting down mid-run |
|
|
|
|
On startup, the app marks any runs still `running` or `queued` as `interrupted` (see `_mark_interrupted_runs()` in `app.py`).
|
|
|
|
### Overlapping runs
|
|
|
|
- **One run per host.** Before it takes a run slot, a run claims every host it touches. Hosts come from the inventory Steward writes, keyed by address. A run on a source's own inventory file claims that file, because Steward doesn't read its hosts. While another run holds one of its hosts, the run shows `queued` and says which run it waits for. After an hour of waiting it fails and says so.
|
|
- **One run at a time per schedule.** A schedule that falls due while its previous run is still `queued` or `running` skips that firing. The Schedules page shows "Skipped: the previous run was still going" beside that run. The next firing comes an interval later.
|
|
- **One action run at a time per alert rule.** When a rule fires again while the action run from an earlier firing is still going, no new run starts. The firing's event says it was skipped and names the run.
|
|
|
|
Output stored in the DB is capped at 1 MB. If truncated, `[output truncated]` is appended to the DB column, but the live SSE stream continues unaffected.
|
|
|
|
---
|
|
|
|
## Data Model
|
|
|
|
`ansible_runs` table (defined in `steward/models/ansible.py`):
|
|
|
|
| Column | Type | Description |
|
|
|---|---|---|
|
|
| `id` | UUID | Primary key |
|
|
| `playbook_path` | str | Relative path to playbook |
|
|
| `inventory_path` | str | Inventory path used |
|
|
| `source_name` | str | Name of the playbook source |
|
|
| `triggered_by` | FK → users | |
|
|
| `status` | enum | `running`, `success`, `failed`, `interrupted` |
|
|
| `started_at` | timestamp UTC | |
|
|
| `finished_at` | timestamp UTC | Null if still running |
|
|
| `output` | text | Captured stdout/stderr (capped at 1 MB) |
|
|
|
|
Runs older than `data.retention_days` (default 90) are pruned by the `data_cleanup` scheduled task.
|
|
|
|
---
|
|
|
|
## Source Files
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `steward/ansible/sources.py` | Source discovery, git pull logic |
|
|
| `steward/ansible/executor.py` | Subprocess execution and output streaming |
|
|
| `steward/ansible/routes.py` | HTTP routes (browse, trigger, stream, history) |
|
|
| `steward/models/ansible.py` | `AnsibleRun` model |
|