3 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 da81010a75 fix(alerts): an alert that fired and ended inside maintenance isn't mailed resolved (#6097)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m9s
During Docker02's maintenance reboot its Agent offline alert fired muted by
the job's window, then resolved just after the window closed: the resolution
was mailed for a firing nobody had been told of. A resolution whose firing
was held and never released is now recorded with a note, not sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 07:41:51 -04:00
bvandeusenandClaude Opus 5.5 d5601575d8 test(alerts): seed metrics the way ingest does, through record_metrics (#5473)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m10s
CI / unit (push) Successful in 1m37s
CI / publish (push) Successful in 1m11s
#5473 "Opening an alert rule takes up to 15 s". With the rule pickers now
reading plugin_metric_latest, nine integration tests failed on rule_fields:
"host_agent has no resource 'box-…'". They seeded a resource by adding a raw
PluginMetric row, which no production path does: host-agent ingest and
record_metric both go through record_metrics, which keeps the latest-value
table. The seeds now go through record_metrics too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:28:53 -04:00
bvandeusenandClaude Opus 5.5 49607a2441 feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m11s
CI / integration (push) Failing after 2m56s
CI / publish (push) Skipped
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.

A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.

Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.

Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
  for the commit; maintenance mutes while the state moves and notifies once
  when the window ends; an action whose playbook is outside its source is
  refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
  deadline, for the run the action starts;
- unit: _muted_by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:45:20 -04:00