feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -96,3 +96,14 @@ def test_a_sample_after_silence_clears_it_and_records_the_resume():
|
||||
assert value == 42.0 and note == "data resumed after 15m silent"
|
||||
[event] = session.added
|
||||
assert (event.id, event.from_state, event.to_state) == (event_id, "no_data", "firing")
|
||||
|
||||
|
||||
def test_muted_by_names_the_window_that_covers_the_rule():
|
||||
from steward.core.alerts import _muted_by
|
||||
rule = NS(source_module="host_agent", resource_name="nas")
|
||||
windows = [NS(name="other", scope_source="docker", scope_resource=None),
|
||||
NS(name="nas work", scope_source="host_agent", scope_resource="nas")]
|
||||
assert _muted_by(windows, rule) == "notification muted: maintenance window nas work"
|
||||
assert _muted_by(windows[:1], rule) is None
|
||||
assert _muted_by([NS(name="all", scope_source=None, scope_resource=None)], rule) == (
|
||||
"notification muted: maintenance window all")
|
||||
|
||||
Reference in New Issue
Block a user