fix(alerts): Steward's own outage no longer reads as every agent going silent (#5441)
After Steward or its database was down, every agent's age and every series' last sample were old, so all agent-offline rules fired and all rules went no-data at once, clearing a minute later. With per-account alert email that would have mailed everyone a dozen notices per restart. A periodic job now ticks a Watch (steward/core/watch.py) once it has reached the database; the Watch says since when runs have been unbroken (process start, or the end of a gap over 120 s). seconds_since_seen and the no-data sweep count silence from no earlier than that, so a source must stay quiet while Steward is watching before it counts as silent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -39,3 +39,14 @@ def test_a_clock_ahead_of_ours_reads_as_just_seen():
|
||||
|
||||
def test_empty_input():
|
||||
assert seconds_since_seen([], NAMES, NOW) == []
|
||||
|
||||
|
||||
def test_steward_s_own_outage_is_not_the_agent_s_silence():
|
||||
"""#5441: agents that last reported 10 minutes ago, while Steward has only
|
||||
been back for a minute, are a minute old; an agent heard since then keeps
|
||||
its own age."""
|
||||
back = NOW - timedelta(minutes=1)
|
||||
regs = [FakeReg("h1", NOW - timedelta(minutes=10), NOW - timedelta(days=9)),
|
||||
FakeReg("h2", NOW - timedelta(seconds=20), NOW - timedelta(days=9)),
|
||||
FakeReg("h3", None, NOW - timedelta(days=9))]
|
||||
assert seconds_since_seen(regs, NAMES, NOW, back) == [("nas", 60.0), ("web", 20.0), ("db", 60.0)]
|
||||
|
||||
Reference in New Issue
Block a user