fix(ansible): overlapping runs: one per host, one per schedule, one action run per alert rule (#5158)
- One run per host: start_run claims every host it touches (addresses read from the inventory Steward wrote; a source's own inventory file is claimed as that file) before it takes a run slot. The claim is all-or-nothing. A run waiting on a host shows queued and names the run it waits for, honours a cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run. - A due schedule whose previous run is still queued or running skips the firing: last_run_at moves on, last_run_id still names the run going, and last_error says "Skipped: the previous run was still going". The Schedules page shows that beside the run (muted), and a launch error in red. - An alert rule that fires while the action run from an earlier firing is still going starts no new run; the event says so and names the run. An in-process set closes the gap between the check and the run being linked. - docs/core/ansible.md: queued/cancelled statuses and the overlap rules. Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing, cancel, deadline) and tests/integration/test_ansible_overlap.py (two real runs on one host take turns; schedule skip then fire; alert action skip then start). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -101,12 +101,20 @@ If a client connects after the run has already completed, the endpoint immediate
|
||||
|
||||
| Status | Description |
|
||||
|---|---|
|
||||
| `queued` | Waiting for a host another run holds, or for a run slot |
|
||||
| `running` | Execution in progress |
|
||||
| `success` | Playbook exited 0 |
|
||||
| `failed` | Playbook exited non-zero |
|
||||
| `interrupted` | App restarted while run was in progress |
|
||||
| `cancelled` | Stopped by an operator, or by Steward shutting down mid-run |
|
||||
|
||||
On startup, the app marks any runs still in `running` state as `interrupted` (see `_mark_interrupted_runs()` in `app.py`).
|
||||
On startup, the app marks any runs still `running` or `queued` as `interrupted` (see `_mark_interrupted_runs()` in `app.py`).
|
||||
|
||||
### Overlapping runs
|
||||
|
||||
- **One run per host.** Before it takes a run slot, a run claims every host it touches. Hosts come from the inventory Steward writes, keyed by address. A run on a source's own inventory file claims that file, because Steward doesn't read its hosts. While another run holds one of its hosts, the run shows `queued` and says which run it waits for. After an hour of waiting it fails and says so.
|
||||
- **One run at a time per schedule.** A schedule that falls due while its previous run is still `queued` or `running` skips that firing. The Schedules page shows "Skipped: the previous run was still going" beside that run. The next firing comes an interval later.
|
||||
- **One action run at a time per alert rule.** When a rule fires again while the action run from an earlier firing is still going, no new run starts. The firing's event says it was skipped and names the run.
|
||||
|
||||
Output stored in the DB is capped at 1 MB. If truncated, `[output truncated]` is appended to the DB column, but the live SSE stream continues unaffected.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user