fix(host_agent): a post that reached Steward counts as the agent seen, even if storing it failed (#5514)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m14s
CI / unit (push) Successful in 1m28s
CI / publish (push) Successful in 1m16s

At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.

Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 16:48:19 -04:00
co-authored by Claude Opus 5.5
parent 715ddfd72f
commit 400e91a0d8
4 changed files with 49 additions and 4 deletions
@@ -50,3 +50,24 @@ def test_steward_s_own_outage_is_not_the_agent_s_silence():
FakeReg("h2", NOW - timedelta(seconds=20), NOW - timedelta(days=9)),
FakeReg("h3", None, NOW - timedelta(days=9))]
assert seconds_since_seen(regs, NAMES, NOW, back) == [("nas", 60.0), ("web", 20.0), ("db", 60.0)]
def test_a_post_that_arrived_but_was_not_stored_counts_as_seen():
"""#5514: Steward's storage stalled and dropped the agents' posts, so
last_seen_at (written with them) stayed old, and agents posting every few
seconds read as offline. Arriving is the agent's part; the later of the
two counts. An agent with no arrival keeps its stored age."""
regs = [FakeReg("h1", NOW - timedelta(minutes=4), NOW - timedelta(days=9)),
FakeReg("h2", NOW - timedelta(minutes=4), NOW - timedelta(days=9)),
FakeReg("h3", NOW - timedelta(seconds=10), NOW - timedelta(days=9))]
arrived = {"h1": NOW - timedelta(seconds=5), "h3": NOW - timedelta(minutes=2)}
assert seconds_since_seen(regs, NAMES, NOW, arrived=arrived) == [
("nas", 5.0), ("web", 240.0), ("db", 10.0)]
def test_arrivals_are_kept_per_app():
from types import SimpleNamespace
from plugins.host_agent.scheduler import arrivals
app, other = SimpleNamespace(), SimpleNamespace()
arrivals(app)["h1"] = NOW
assert arrivals(app) == {"h1": NOW} and arrivals(other) == {}