feat(alerts): alert email waits 2 minutes in the outbox and each alert records its outcome (#5507)
#5507 "Alert emails through the outbox, gathered for 2 minutes" (milestone 501). Replaces the in-memory core/alert_burst.py. - outbox.hold_alert: an alert joins each recipient's held alert email, or opens one due 2 minutes later (System 156 ruling). A Postgres advisory lock keeps two alerts at once from opening two windows for one reader. - email_outbox_events.alert_json (migration 0043) keeps the alert as it fired, so the email written when the window closes says what it said. - write_held writes it then, in the reader's zone: the usual alert email for one alert, the list for several (outbox.alert_words). - after_settle records on each AlertEvent, once every email carrying it has settled: email_sent, or email_error naming each reader it didn't reach and the server's reason. - A held email survives a restart and goes out when its window closes. - A refusal of a message's one recipient no longer repeats the address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+20
-7
@@ -162,13 +162,26 @@ Configured at `/settings/` under SMTP. Settings:
|
||||
|
||||
Email is skipped if `smtp.host` is empty.
|
||||
|
||||
Alerts that arrive together go out as one email (`core/alert_burst.py`). The
|
||||
first alert opens a 30-second window. When it closes, each recipient gets one
|
||||
message listing every alert in the window, oldest first, in their own zone,
|
||||
with a subject like "[Steward] 3 alerts: 2 firing, 1 resolved". One alert on
|
||||
its own keeps its usual email. Every alert's event records whether the email
|
||||
went. A burst still waiting when Steward stops is not sent; its alerts stay on
|
||||
the Alerts page and in the history.
|
||||
Alerts that arrive together go out as one email. The first alert opens a
|
||||
2-minute window. When it closes, each recipient gets one message listing every
|
||||
alert in the window, oldest first, in their own zone, with a subject like
|
||||
"[Steward] 3 alerts: 2 firing, 1 resolved". One alert on its own keeps its
|
||||
usual email.
|
||||
|
||||
Email waits in the database (`email_outbox`, `core/outbox.py`) until the mail
|
||||
server accepts it, so nothing waiting is lost when Steward restarts: the
|
||||
sender picks it up at boot. The sender runs every 15 seconds and settles each
|
||||
email on the server's answer:
|
||||
|
||||
- **Accepted:** sent.
|
||||
- **Refused** (an SMTP 5xx, such as "no such mailbox"): failed at once, with
|
||||
the server's words. Trying again would only repeat the refusal.
|
||||
- **Temporary** (no connection, a timeout, a 4xx): tried again after 30 s,
|
||||
1, 2, 5, 10 and then every 15 minutes. An email the server still hasn't
|
||||
taken an hour after its first try is failed with the last error.
|
||||
|
||||
Once every email carrying an alert has settled, that alert's event records
|
||||
whether the email went, and to whom it didn't.
|
||||
|
||||
### Webhook
|
||||
|
||||
|
||||
Reference in New Issue
Block a user