fix(alerts): one silent agent is one alert and one finding (#5721)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m19s
CI / unit (push) Successful in 1m29s
CI / publish (push) Successful in 1m8s

VDNT-Docker03's silent agent read as three overview findings (agent offline,
agent stale, the disk rule's no-data) and every drop emailed the disk rule's
silence and return on top of agent offline.

- The host agent registers its heartbeat (seconds_since_seen) and the
  sources it carries (host_agent, docker). While a host's agent is past its
  offline window, its other rules' no_data is recorded "while its agent is
  offline" and not emailed, nor is their quiet return; a return past a
  threshold still fires.
- The overview folds those rules under the host's agent-offline finding
  ("also silent with it: ..."), and a stale agent whose agent-offline alert
  is active is that alert's finding, not a second one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:49:29 -04:00
co-authored by Claude Opus 5.5
parent 76d75961f8
commit d0c563abd0
6 changed files with 239 additions and 12 deletions
+6
View File
@@ -19,6 +19,12 @@ def setup(app: "Quart") -> None:
learn_normal("host_agent", "cpu_pct", floor_setting(app, "host_cpu_pct"), parts=False)
learn_normal("host_agent", "mem_used_pct", floor_setting(app, "host_mem_pct"), parts=False)
learn_normal("host_agent", "load_pct", floor_setting(app, "host_load_pct"), parts=False)
# "Agent offline" speaks for everything the agent carries: while it is
# firing, the host's other rules going silent isn't emailed (#5721).
from steward.core.alerts import agent_fed, agent_heartbeat
from steward.hosts.service import agent_stale_after_seconds
agent_heartbeat("host_agent", "seconds_since_seen", lambda: agent_stale_after_seconds(app.config))
agent_fed("host_agent", lambda resource: resource.split(":", 1)[0])
def get_scheduled_tasks() -> list: