fix(alerts): Steward's own outage no longer reads as every agent going silent (#5441)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m26s
CI / unit (push) Successful in 1m31s
CI / publish (push) Skipped

After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.

A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 21:45:47 -04:00
co-authored by Claude Opus 5.5
parent 5a44afe86e
commit a9bf4b7f16
7 changed files with 183 additions and 6 deletions
+13
View File
@@ -60,6 +60,19 @@ def test_silent_since(last_ago, interval, watch_ago, silent_after):
assert got == (last or watch) + timedelta(seconds=silent_after)
def test_steward_s_own_outage_is_not_the_series_silence():
"""#5441: a series last sampled 18 minutes ago, while Steward has only been
back for 2 minutes, is not silent yet; once Steward has watched it for
three of its gaps with nothing arriving, it is, from then."""
last = NOW - timedelta(minutes=18)
watch = NOW - timedelta(days=1)
assert silent_since(last, 60, watch, NOW, awake_since=NOW - timedelta(minutes=2)) is None
awake = NOW - timedelta(minutes=6)
assert silent_since(last, 60, watch, NOW, awake_since=awake) == awake + timedelta(seconds=300)
# A sample since Steward came back is what counts, when there is one.
assert silent_since(NOW - timedelta(minutes=1), 60, watch, NOW, awake_since=awake) is None
class _Session:
def __init__(self):
self.added = []