fix(alerts): Steward's own outage no longer reads as every agent going silent (#5441)
After Steward or its database was down, every agent's age and every series' last sample were old, so all agent-offline rules fired and all rules went no-data at once, clearing a minute later. With per-account alert email that would have mailed everyone a dozen notices per restart. A periodic job now ticks a Watch (steward/core/watch.py) once it has reached the database; the Watch says since when runs have been unbroken (process start, or the end of a gap over 120 s). seconds_since_seen and the no-data sweep count silence from no earlier than that, so a source must stay quiet while Steward is watching before it counts as silent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,43 @@
|
||||
"""Since when Steward has been watching without a break (#5441).
|
||||
|
||||
Silence is measured by Steward: an agent's age is now minus its last report,
|
||||
a series is silent once no sample has arrived for a while. While Steward (or
|
||||
its database) is down nothing arrives, so on its return every agent and every
|
||||
series looks silent at once, and every agent-offline and no-data alert used to
|
||||
fire together for an outage that was Steward's own.
|
||||
|
||||
A periodic job ticks its Watch each run, once it has reached the database.
|
||||
The Watch answers "since when have I been running without a break": since
|
||||
the process started, or since the end of a gap longer than GAP_S between
|
||||
ticks (the database was down, the loop stalled). Silence is then counted from
|
||||
no earlier than that, so a source has to stay quiet while Steward is
|
||||
watching before it counts as silent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
# Two missed 60 s ticks. A gap this long means Steward was not watching; a
|
||||
# shorter one only delays a real alert by the length of the gap.
|
||||
GAP_S = 120.0
|
||||
|
||||
|
||||
class Watch:
|
||||
def __init__(self) -> None:
|
||||
self.since: datetime | None = None
|
||||
self.last: datetime | None = None
|
||||
|
||||
def tick(self, now: datetime) -> datetime:
|
||||
"""Record a run at `now`; return since when runs have been unbroken."""
|
||||
if self.since is None or self.last is None or (now - self.last).total_seconds() > GAP_S:
|
||||
self.since = now
|
||||
self.last = now
|
||||
return self.since
|
||||
|
||||
|
||||
def watch(app, name: str) -> Watch:
|
||||
"""The app's Watch for one periodic job, made on first use."""
|
||||
watches = getattr(app, "_watches", None)
|
||||
if watches is None:
|
||||
watches = app._watches = {}
|
||||
return watches.setdefault(name, Watch())
|
||||
Reference in New Issue
Block a user