fix(ansible): overlapping runs: one per host, one per schedule, one action run per alert rule (#5158)
- One run per host: start_run claims every host it touches (addresses read from the inventory Steward wrote; a source's own inventory file is claimed as that file) before it takes a run slot. The claim is all-or-nothing. A run waiting on a host shows queued and names the run it waits for, honours a cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run. - A due schedule whose previous run is still queued or running skips the firing: last_run_at moves on, last_run_id still names the run going, and last_error says "Skipped: the previous run was still going". The Schedules page shows that beside the run (muted), and a launch error in red. - An alert rule that fires while the action run from an earlier firing is still going starts no new run; the event says so and names the run. An in-process set closes the gap between the check and the run being linked. - docs/core/ansible.md: queued/cancelled statuses and the overlap rules. Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing, cancel, deadline) and tests/integration/test_ansible_overlap.py (two real runs on one host take turns; schedule skip then fire; alert action skip then start). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,7 +7,7 @@ from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from steward.ansible.runner import trigger_run
|
||||
from steward.ansible.runner import run_in_flight, trigger_run
|
||||
from steward.models.ansible_schedule import AnsibleSchedule
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -27,7 +27,8 @@ def is_due(schedule: AnsibleSchedule, now: datetime) -> bool:
|
||||
|
||||
|
||||
async def run_due_schedules(app) -> int:
|
||||
"""Trigger every due schedule once. Returns the number launched."""
|
||||
"""Trigger every due schedule once. Returns the number launched. A due
|
||||
schedule whose previous run hasn't finished skips this firing."""
|
||||
now = datetime.now(timezone.utc)
|
||||
async with app.db_sessionmaker() as db:
|
||||
schedules = (await db.execute(
|
||||
@@ -38,6 +39,21 @@ async def run_due_schedules(app) -> int:
|
||||
for s in schedules:
|
||||
if not is_due(s, now):
|
||||
continue
|
||||
async with app.db_sessionmaker() as db:
|
||||
busy = await run_in_flight(db, s.last_run_id)
|
||||
if busy:
|
||||
# One run at a time per schedule (#5158). This firing is skipped,
|
||||
# not deferred, so the cadence holds; last_run_id still names the
|
||||
# run that's going.
|
||||
async with app.db_sessionmaker() as db:
|
||||
async with db.begin():
|
||||
fresh = await db.get(AnsibleSchedule, s.id)
|
||||
if fresh:
|
||||
fresh.last_run_at = now
|
||||
fresh.last_error = "Skipped: the previous run was still going"
|
||||
logger.info("Schedule %s (%s) skipped: run %s still going",
|
||||
s.id, s.name, s.last_run_id)
|
||||
continue
|
||||
run = None
|
||||
err: str | None = None
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user