feat: readings kept through a storage stall; an agent's backlog arrives in order (#5995)
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things went wrong for the readings that would have explained it: - The SNMP poll stored before it polled again, so while the database hung it stopped polling. It now collects on schedule and hands each round to a Holding (steward/core/holding.py): one writer stores what is held, oldest first, each reading at the time it was collected. Bounded at two hours of polls, with a 30-minute deadline on a store; the device list read falls back to the last one read after 10 s. - "Mount not answering" (10 checks in a row) went pending on Docker02 and Docker03 and never fired: the agent sent each post's backlog with its newest, healthy sample, and that sample ended every post's run. Agent 1.22.0 queues the new sample behind the backlog, so Steward receives a host's samples in the order they were taken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -120,11 +120,11 @@ def test_backoff_schedule_stays_inside_the_stale_window():
|
||||
assert next_backoff(60) == 60 # capped below Steward's 180 s stale window (#5154)
|
||||
|
||||
|
||||
def test_a_backlog_drains_a_few_samples_a_post_oldest_first():
|
||||
def test_a_backlog_drains_a_few_samples_a_post_oldest_first_and_the_new_one_waits_its_turn():
|
||||
from plugins.host_agent.agent import MAX_SAMPLES_PER_POST, split_backlog
|
||||
buffered = list(range(20))
|
||||
send, held = split_backlog(buffered)
|
||||
assert send == [0, 1, 2, 3] and held == list(range(4, 20))
|
||||
assert len(send) + 1 == MAX_SAMPLES_PER_POST # room for the new sample
|
||||
assert split_backlog([7]) == ([7], [])
|
||||
assert split_backlog([]) == ([], [])
|
||||
queued = list(range(20)) + ["new"]
|
||||
send, held = split_backlog(queued)
|
||||
assert send == [0, 1, 2, 3, 4] and len(send) == MAX_SAMPLES_PER_POST
|
||||
assert held == list(range(5, 20)) + ["new"] # never ahead of an older sample
|
||||
assert split_backlog(["new"]) == (["new"], [])
|
||||
assert split_backlog([7, "new"]) == ([7, "new"], [])
|
||||
|
||||
Reference in New Issue
Block a user