fix(ansible): overlapping runs: one per host, one per schedule, one action run per alert rule (#5158)
- One run per host: start_run claims every host it touches (addresses read from the inventory Steward wrote; a source's own inventory file is claimed as that file) before it takes a run slot. The claim is all-or-nothing. A run waiting on a host shows queued and names the run it waits for, honours a cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run. - A due schedule whose previous run is still queued or running skips the firing: last_run_at moves on, last_run_id still names the run going, and last_error says "Skipped: the previous run was still going". The Schedules page shows that beside the run (muted), and a launch error in red. - An alert rule that fires while the action run from an earlier firing is still going starts no new run; the event says so and names the run. An in-process set closes the gap between the check and the run being linked. - docs/core/ansible.md: queued/cancelled statuses and the overlap rules. Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing, cancel, deadline) and tests/integration/test_ansible_overlap.py (two real runs on one host take turns; schedule skip then fire; alert action skip then start). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -16,6 +16,19 @@ from steward.ansible.inventory_gen import (
|
||||
)
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
|
||||
# A run in one of these hasn't finished: a schedule or an alert action whose
|
||||
# previous run is still in one skips its next firing (#5158).
|
||||
IN_FLIGHT = (AnsibleRunStatus.queued, AnsibleRunStatus.running)
|
||||
|
||||
|
||||
async def run_in_flight(db, run_id: str | None) -> bool:
|
||||
"""Whether run_id names a run that hasn't finished. Runs left running by a
|
||||
restart are marked interrupted at startup, so this can't stick."""
|
||||
if not run_id:
|
||||
return False
|
||||
run = await db.get(AnsibleRun, run_id)
|
||||
return run is not None and run.status in IN_FLIGHT
|
||||
|
||||
|
||||
async def trigger_run(
|
||||
app,
|
||||
|
||||
Reference in New Issue
Block a user