fix(host_agent): a post that reached Steward counts as the agent seen, even if storing it failed (#5514)

At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s). Ingest commits stalled past the agent's 10 s timeout and were dropped, and last_seen_at, written in that same transaction, went with them. Agents posting every few seconds read as silent for 3 minutes, and "Agent offline" fired and emailed for all three hosts. Ingest now notes each agent's arrival in memory right after the (read-only) registration lookup, before any write; seconds_since_seen counts from the later of last_seen_at and that arrival. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:48:19 -04:00
parent 715ddfd72f
commit 400e91a0d8
4 changed files with 49 additions and 4 deletions
@@ -205,6 +205,8 @@ One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration`

 **Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.

+**As built:** the task records `host_agent/<host>/seconds_since_seen` each minute, and "Agent offline" is an ordinary alert rule on it (`> 180`). It counts from the later of `last_seen_at` and the agent's newest post to *arrive* (kept in memory, `scheduler.arrivals`). `last_seen_at` is written in the ingest's own transaction, so a post Steward received but couldn't store (its storage stalled, #5514) would otherwise read as the agent's silence. Nothing counts from before Steward last started watching without a break (#5441).
+
 ### `METRIC_CATALOG` registration

 Add to `steward/alerts/routes.py`: