feat(alerts): alert email waits 2 minutes in the outbox and each alert records its outcome (#5507)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m13s
CI / unit (push) Successful in 1m25s
CI / publish (push) Skipped

#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.

- outbox.hold_alert: an alert joins each recipient's held alert email, or
  opens one due 2 minutes later (System 156 ruling). A Postgres advisory
  lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
  fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
  for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
  has settled: email_sent, or email_error naming each reader it didn't
  reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 15:59:17 -04:00
co-authored by Claude Opus 5.5
parent 4e77ad0e7f
commit 5237ffc663
12 changed files with 356 additions and 212 deletions
+11 -7
View File
@@ -495,9 +495,10 @@ async def _dispatch_notification(
event_id: str,
note: str | None = None,
) -> None:
"""Run after the transaction commits. Holds the alert's email for its
burst (core.alert_burst), posts its webhook, and records the webhook's
outcome on the event; the burst records the email's."""
"""Run after the transaction commits. Queues the alert's email in the
outbox, gathered for 2 minutes with any others (core.outbox.hold_alert),
posts its webhook, and records the webhook's outcome on the event; the
outbox records the email's once it settles."""
from steward.core.notifications import dispatch_notifications
alert_data = {
@@ -513,11 +514,14 @@ async def _dispatch_notification(
"timestamp": datetime.now(timezone.utc).isoformat(),
}
# Email waits a moment so alerts that arrive together go out as one
# message per recipient (#5465); the webhook posts this alert now.
from steward.core import alert_burst
# Email waits so alerts that arrive together go out as one message per
# recipient (#5465, #5507); the webhook posts this alert now.
from steward.core import outbox
if app.config.get("SMTP", {}).get("host"):
alert_burst.hold(app, alert_data, event_id)
try:
await outbox.hold_alert(app, alert_data, event_id)
except Exception:
logger.exception("Alert %s: its email could not be queued", event_id)
results = await dispatch_notifications({}, app.config.get("WEBHOOK", {}), alert_data)
webhook_r = results.get("webhook")