fix(alerts): a silence ends when Steward hears of it, once (#5720)

An agent's flushed backlog brings samples hours old. The resume was stamped
at the first old sample's time, inside the silence it ended, and silence was
still judged by the backlog's sample times, so the series was marked silent
again between posts and resumed a second time. A resumed series now counts as
heard at ingest time, its events from that batch are stamped no earlier, and
the rest of the backlog reads as late samples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:47:09 -04:00
co-authored by Claude Opus 5.5
parent 264d44ac81
commit 76d75961f8
2 changed files with 30 additions and 2 deletions
+14 -2
View File
@@ -250,9 +250,14 @@ async def record_metrics(session: AsyncSession, rows: list[PluginMetric], phases
key=lambda r: r.recorded_at)
muted = _muted_by(windows, rule)
resumed = _track_samples(alert_state, samples, now)
# A silence ends when Steward hears of it: an agent's flushed backlog
# brings samples hours old, and stamping their events at their own
# times put the resume inside the silence it ends (#5720).
back = resumed is not None
for sample in samples:
at = max(sample.recorded_at, now) if back else sample.recorded_at
notifications = await _evaluate_rule(session, rule, alert_state, sample.value,
sample.recorded_at, note=muted, resumed=resumed)
at, note=muted, resumed=resumed)
resumed = None # said once, with the first sample back
if muted:
continue
@@ -319,7 +324,13 @@ def _track_samples(state: AlertState, samples: list[PluginMetric], now: datetime
series had been marked no_data, clear the mark and return the note saying
so ("data resumed after 15m silent"); the first sample's evaluation
records it, in one event with whatever that sample does to the rule
(#5442)."""
(#5442).
A resumed series counts as last heard `now`, when Steward heard it, not at
its newest sample's time: a backlog flushed a few samples a post is hours
old, and judging silence by it marked the series silent again between
posts, so one silence was resumed twice (#5720). The rest of the backlog
then reads as late samples, which teach the cadence nothing."""
for sample in samples:
if state.last_sample_at is not None:
if sample.recorded_at <= state.last_sample_at:
@@ -331,6 +342,7 @@ def _track_samples(state: AlertState, samples: list[PluginMetric], now: datetime
return None
note = f"data resumed after {human_duration((now - state.no_data_since).total_seconds())} silent"
state.no_data_since = None
state.last_sample_at = max(state.last_sample_at or now, now)
return note