Commit Graph
4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 0ec7ce4472 test(alerts): wait for the alert's tasks and the run, not every task (#5063)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 46s
CI / unit (push) Successful in 1m4s
CI / publish (push) Successful in 1m25s
CI 8343: the wait-for-all-tasks loop also waited on the executor's
5-minute run-output cleanup task (_schedule_cleanup) and hit its 120s
deadline. Wait on alerts._background, then poll the run to a final status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:48:50 -04:00
bvandeusenandClaude Opus 5.5 49607a2441 feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m11s
CI / integration (push) Failing after 2m56s
CI / publish (push) Skipped
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.

A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.

Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.

Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
  for the commit; maintenance mutes while the state moves and notifies once
  when the window ends; an action whose playbook is outside its source is
  refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
  deadline, for the run the action starts;
- unit: _muted_by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:45:20 -04:00
bvandeusenandClaude Opus 4.8 534ed030b8 test(alerts): commit the test user before the rule's created_by FK
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 7s
CI / integration (push) Successful in 2m14s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 14:31:25 -04:00
bvandeusenandClaude Opus 4.8 4771d17f6d feat(alerts): run an Ansible playbook on alert firing (task 250)
CI / unit (push) Successful in 8s
CI / integration (push) Failing after 2m16s
CI / lint (push) Successful in 2s
Rebuilds the deleted NUT/UPS automation as a general alert action: any metric
can drive a playbook run on transition-to-firing.

- models/alerts.py + migration 0014: AlertRule.ansible_action (JSON, admin-only,
  reuses the #546 param shape); AlertEvent.ansible_run_id links a firing event to
  the run it triggered
- core/alerts.py: pure alert_extra_vars() injects steward_alert_* context; on
  ('firing', event) with an action set, schedule _run_ansible_action (deferred
  after commit, same pattern as notifications) — fires once per transition,
  consecutive_failures_required is the debounce; system-triggered AnsibleRun
- alerts/routes.py: admin-only parse/validate of the action (source must be
  configured, playbook must exist); operators keep editing rules, action preserved
- rules_form.html: admin-only 'On firing -> run a playbook' section
- tests: unit for alert_extra_vars; integration drives record_metric to firing
  and asserts a system AnsibleRun ran the playbook with the injected var and the
  event linked to it

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 14:22:29 -04:00