feat: readings kept through a storage stall; an agent's backlog arrives in order (#5995)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m39s
CI / integration (push) Successful in 1m49s
CI / publish (push) Successful in 1m49s

On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:

- The SNMP poll stored before it polled again, so while the database hung
  it stopped polling. It now collects on schedule and hands each round to a
  Holding (steward/core/holding.py): one writer stores what is held, oldest
  first, each reading at the time it was collected. Bounded at two hours of
  polls, with a 30-minute deadline on a store; the device list read falls
  back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
  Docker03 and never fired: the agent sent each post's backlog with its
  newest, healthy sample, and that sample ended every post's run. Agent
  1.22.0 queues the new sample behind the backlog, so Steward receives a
  host's samples in the order they were taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-17 09:36:47 -04:00
co-authored by Claude Opus 5.5
parent bd1c32d2a4
commit 41fc8073db
7 changed files with 387 additions and 77 deletions
@@ -120,11 +120,11 @@ def test_backoff_schedule_stays_inside_the_stale_window():
assert next_backoff(60) == 60 # capped below Steward's 180 s stale window (#5154)
def test_a_backlog_drains_a_few_samples_a_post_oldest_first():
def test_a_backlog_drains_a_few_samples_a_post_oldest_first_and_the_new_one_waits_its_turn():
from plugins.host_agent.agent import MAX_SAMPLES_PER_POST, split_backlog
buffered = list(range(20))
send, held = split_backlog(buffered)
assert send == [0, 1, 2, 3] and held == list(range(4, 20))
assert len(send) + 1 == MAX_SAMPLES_PER_POST # room for the new sample
assert split_backlog([7]) == ([7], [])
assert split_backlog([]) == ([], [])
queued = list(range(20)) + ["new"]
send, held = split_backlog(queued)
assert send == [0, 1, 2, 3, 4] and len(send) == MAX_SAMPLES_PER_POST
assert held == list(range(5, 20)) + ["new"] # never ahead of an older sample
assert split_backlog(["new"]) == (["new"], [])
assert split_backlog([7, "new"]) == ([7, "new"], [])