fix(alerts): Steward's own outage no longer reads as every agent going silent (#5441)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m26s
CI / unit (push) Successful in 1m31s
CI / publish (push) Skipped

After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.

A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 21:45:47 -04:00
co-authored by Claude Opus 5.5
parent 5a44afe86e
commit a9bf4b7f16
7 changed files with 183 additions and 6 deletions
@@ -39,3 +39,14 @@ def test_a_clock_ahead_of_ours_reads_as_just_seen():
def test_empty_input():
assert seconds_since_seen([], NAMES, NOW) == []
def test_steward_s_own_outage_is_not_the_agent_s_silence():
"""#5441: agents that last reported 10 minutes ago, while Steward has only
been back for a minute, are a minute old; an agent heard since then keeps
its own age."""
back = NOW - timedelta(minutes=1)
regs = [FakeReg("h1", NOW - timedelta(minutes=10), NOW - timedelta(days=9)),
FakeReg("h2", NOW - timedelta(seconds=20), NOW - timedelta(days=9)),
FakeReg("h3", None, NOW - timedelta(days=9))]
assert seconds_since_seen(regs, NAMES, NOW, back) == [("nas", 60.0), ("web", 20.0), ("db", 60.0)]