feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -48,7 +48,8 @@ def test_firing_triggers_ansible_action(app, tmp_path):
|
||||
- debug:
|
||||
msg: "FIRED-{{ steward_alert_metric }}"
|
||||
"""))
|
||||
(tmp_path / "inv.ini").write_text("localhost ansible_connection=local\n")
|
||||
# inventory.ini: an inventory name the source lists, as trigger_run's guard requires.
|
||||
(tmp_path / "inventory.ini").write_text("localhost ansible_connection=local\n")
|
||||
|
||||
# An ansible source the action resolves against.
|
||||
app.config["ANSIBLE"] = {
|
||||
@@ -57,7 +58,7 @@ def test_firing_triggers_ansible_action(app, tmp_path):
|
||||
|
||||
uid = str(uuid.uuid4())
|
||||
rule_id = str(uuid.uuid4())
|
||||
action = {"source": "t", "playbook": "fire.yml", "inventory": "inv.ini"}
|
||||
action = {"source": "t", "playbook": "fire.yml", "inventory": "inventory.ini"}
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
@@ -82,9 +83,13 @@ def test_firing_triggers_ansible_action(app, tmp_path):
|
||||
async with s.begin():
|
||||
await record_metric(s, "host_agent", "srv1", "cpu_pct", 95.0)
|
||||
|
||||
# Let the fire-and-forget dispatch + action tasks complete.
|
||||
pending = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
|
||||
await asyncio.gather(*pending, return_exceptions=True)
|
||||
# Let the dispatch, the action and the run it starts finish. The run's
|
||||
# task is created by the action's task, so wait until none are left.
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + 120
|
||||
while pending := [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]:
|
||||
assert loop.time() < deadline, f"tasks still running: {pending}"
|
||||
await asyncio.wait(pending, timeout=deadline - loop.time())
|
||||
|
||||
async with app.db_sessionmaker() as s:
|
||||
run = (await s.execute(
|
||||
|
||||
Reference in New Issue
Block a user