fix(alerts): Steward's own outage no longer reads as every agent going silent (#5441)
After Steward or its database was down, every agent's age and every series' last sample were old, so all agent-offline rules fired and all rules went no-data at once, clearing a minute later. With per-account alert email that would have mailed everyone a dozen notices per restart. A periodic job now ticks a Watch (steward/core/watch.py) once it has reached the database; the Watch says since when runs have been unbroken (process start, or the end of a gap over 120 s). seconds_since_seen and the no-data sweep count silence from no earlier than that, so a source must stay quiet while Steward is watching before it counts as silent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -60,6 +60,19 @@ def test_silent_since(last_ago, interval, watch_ago, silent_after):
|
||||
assert got == (last or watch) + timedelta(seconds=silent_after)
|
||||
|
||||
|
||||
def test_steward_s_own_outage_is_not_the_series_silence():
|
||||
"""#5441: a series last sampled 18 minutes ago, while Steward has only been
|
||||
back for 2 minutes, is not silent yet; once Steward has watched it for
|
||||
three of its gaps with nothing arriving, it is, from then."""
|
||||
last = NOW - timedelta(minutes=18)
|
||||
watch = NOW - timedelta(days=1)
|
||||
assert silent_since(last, 60, watch, NOW, awake_since=NOW - timedelta(minutes=2)) is None
|
||||
awake = NOW - timedelta(minutes=6)
|
||||
assert silent_since(last, 60, watch, NOW, awake_since=awake) == awake + timedelta(seconds=300)
|
||||
# A sample since Steward came back is what counts, when there is one.
|
||||
assert silent_since(NOW - timedelta(minutes=1), 60, watch, NOW, awake_since=awake) is None
|
||||
|
||||
|
||||
class _Session:
|
||||
def __init__(self):
|
||||
self.added = []
|
||||
|
||||
Reference in New Issue
Block a user