bvandeusenandClaude Opus 5.5
a9bf4b7f16
After Steward or its database was down, every agent's age and every series' last sample were old, so all agent-offline rules fired and all rules went no-data at once, clearing a minute later. With per-account alert email that would have mailed everyone a dozen notices per restart. A periodic job now ticks a Watch (steward/core/watch.py) once it has reached the database; the Watch says since when runs have been unbroken (process start, or the end of a gap over 120 s). seconds_since_seen and the no-data sweep count silence from no earlier than that, so a source must stay quiet while Steward is watching before it counts as silent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
44 lines
1.7 KiB
Python
44 lines
1.7 KiB
Python
"""Since when Steward has been watching without a break (#5441).
|
|
|
|
Silence is measured by Steward: an agent's age is now minus its last report,
|
|
a series is silent once no sample has arrived for a while. While Steward (or
|
|
its database) is down nothing arrives, so on its return every agent and every
|
|
series looks silent at once, and every agent-offline and no-data alert used to
|
|
fire together for an outage that was Steward's own.
|
|
|
|
A periodic job ticks its Watch each run, once it has reached the database.
|
|
The Watch answers "since when have I been running without a break": since
|
|
the process started, or since the end of a gap longer than GAP_S between
|
|
ticks (the database was down, the loop stalled). Silence is then counted from
|
|
no earlier than that, so a source has to stay quiet while Steward is
|
|
watching before it counts as silent.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from datetime import datetime
|
|
|
|
# Two missed 60 s ticks. A gap this long means Steward was not watching; a
|
|
# shorter one only delays a real alert by the length of the gap.
|
|
GAP_S = 120.0
|
|
|
|
|
|
class Watch:
|
|
def __init__(self) -> None:
|
|
self.since: datetime | None = None
|
|
self.last: datetime | None = None
|
|
|
|
def tick(self, now: datetime) -> datetime:
|
|
"""Record a run at `now`; return since when runs have been unbroken."""
|
|
if self.since is None or self.last is None or (now - self.last).total_seconds() > GAP_S:
|
|
self.since = now
|
|
self.last = now
|
|
return self.since
|
|
|
|
|
|
def watch(app, name: str) -> Watch:
|
|
"""The app's Watch for one periodic job, made on first use."""
|
|
watches = getattr(app, "_watches", None)
|
|
if watches is None:
|
|
watches = app._watches = {}
|
|
return watches.setdefault(name, Watch())
|