feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m11s
CI / integration (push) Failing after 2m56s
CI / publish (push) Skipped

Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.

A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.

Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.

Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
  for the commit; maintenance mutes while the state moves and notifies once
  when the window ends; an action whose playbook is outside its source is
  refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
  deadline, for the run the action starts;
- unit: _muted_by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 13:45:20 -04:00
co-authored by Claude Opus 5.5
parent b009a8dd30
commit 49607a2441
7 changed files with 371 additions and 82 deletions
+11 -6
View File
@@ -344,9 +344,13 @@ def _register_core_tasks(app: Quart) -> None:
from .core.cleanup import run_cleanup as _cleanup
await _cleanup(app)
async def run_silence_check():
from .core.alerts import check_silence
await check_silence(app)
async def run_alert_reconcile():
from .core.alerts import check_silence, notify_after_maintenance
for step in (check_silence, notify_after_maintenance):
try:
await step(app)
except Exception:
app.logger.exception("alert_reconcile: %s failed", step.__name__)
async def run_report_check():
from .core.reports import check_and_send
@@ -372,10 +376,11 @@ def _register_core_tasks(app: Quart) -> None:
interval_seconds=60,
run_on_startup=True,
),
# Rules whose series has gone quiet get a visible no_data mark.
# Rules whose series has gone quiet get a visible no_data mark; alerts
# a maintenance window muted notify once it ends.
ScheduledTask(
name="alert_silence_check",
coro_factory=run_silence_check,
name="alert_reconcile",
coro_factory=run_alert_reconcile,
interval_seconds=60,
run_on_startup=False,
),