feat(alerts): alert email waits 2 minutes in the outbox and each alert records its outcome (#5507)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m13s
CI / unit (push) Successful in 1m25s
CI / publish (push) Skipped

#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.

- outbox.hold_alert: an alert joins each recipient's held alert email, or
  opens one due 2 minutes later (System 156 ruling). A Postgres advisory
  lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
  fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
  for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
  has settled: email_sent, or email_error naming each reader it didn't
  reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 15:59:17 -04:00
co-authored by Claude Opus 5.5
parent 4e77ad0e7f
commit 5237ffc663
12 changed files with 356 additions and 212 deletions
+20 -7
View File
@@ -162,13 +162,26 @@ Configured at `/settings/` under SMTP. Settings:
Email is skipped if `smtp.host` is empty.
Alerts that arrive together go out as one email (`core/alert_burst.py`). The
first alert opens a 30-second window. When it closes, each recipient gets one
message listing every alert in the window, oldest first, in their own zone,
with a subject like "[Steward] 3 alerts: 2 firing, 1 resolved". One alert on
its own keeps its usual email. Every alert's event records whether the email
went. A burst still waiting when Steward stops is not sent; its alerts stay on
the Alerts page and in the history.
Alerts that arrive together go out as one email. The first alert opens a
2-minute window. When it closes, each recipient gets one message listing every
alert in the window, oldest first, in their own zone, with a subject like
"[Steward] 3 alerts: 2 firing, 1 resolved". One alert on its own keeps its
usual email.
Email waits in the database (`email_outbox`, `core/outbox.py`) until the mail
server accepts it, so nothing waiting is lost when Steward restarts: the
sender picks it up at boot. The sender runs every 15 seconds and settles each
email on the server's answer:
- **Accepted:** sent.
- **Refused** (an SMTP 5xx, such as "no such mailbox"): failed at once, with
the server's words. Trying again would only repeat the refusal.
- **Temporary** (no connection, a timeout, a 4xx): tried again after 30 s,
1, 2, 5, 10 and then every 15 minutes. An email the server still hasn't
taken an hour after its first try is failed with the last error.
Once every email carrying an alert has settled, that alert's event records
whether the email went, and to whom it didn't.
### Webhook