fix(host_agent): a post that reached Steward counts as the agent seen, even if storing it failed (#5514)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m14s
CI / unit (push) Successful in 1m28s
CI / publish (push) Successful in 1m16s

At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.

Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 16:48:19 -04:00
co-authored by Claude Opus 5.5
parent 715ddfd72f
commit 400e91a0d8
4 changed files with 49 additions and 4 deletions
+3
View File
@@ -24,6 +24,7 @@ from steward.core.time_range import parse_range, RANGE_OPTIONS
from steward.models.hosts import Host
from steward.models.metrics import PluginMetric
from .models import HostAgentRegistration
from .scheduler import arrivals
# Query helpers live in a model-free module so integration tests can import them
# without the plugin-loader double-registration guard tripping (see metrics_query).
from .metrics_query import (
@@ -242,6 +243,8 @@ async def ingest():
)).scalar_one_or_none()
if host is None:
return _error(401, "invalid_token")
# The agent reached us: that holds even if storing its post fails (#5514).
arrivals(current_app)[reg.host_id] = received_at
accepted = 0
metric_rows: list[PluginMetric] = []