feat: readings kept through a storage stall; an agent's backlog arrives in order (#5995)

On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:

- The SNMP poll stored before it polled again, so while the database hung
  it stopped polling. It now collects on schedule and hands each round to a
  Holding (steward/core/holding.py): one writer stores what is held, oldest
  first, each reading at the time it was collected. Bounded at two hours of
  polls, with a 30-minute deadline on a store; the device list read falls
  back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
  Docker03 and never fired: the agent sent each post's backlog with its
  newest, healthy sample, and that sample ended every post's run. Agent
  1.22.0 queues the new sample behind the backlog, so Steward receives a
  host's samples in the order they were taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

plugins/host_agent/agent.py

@@ -22,7 +22,7 @@ from collections import deque
 from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
 from datetime import datetime, timezone
 
-AGENT_VERSION = "1.21.0"
+AGENT_VERSION = "1.22.0"
 
 # Default path to the local Docker Engine socket. Overridable via the
 # `docker_socket` config key; collection is silently skipped if it's absent or
@@ -1956,10 +1956,16 @@ def _drop_logs(sample: dict) -> dict:
     return sample
 
 
-def split_backlog(buffered: list, limit: int = MAX_SAMPLES_PER_POST) -> tuple:
-    """(the buffered samples to send with this one, the ones held for later):
-    the oldest limit-1, leaving room for the new sample."""
-    return buffered[:limit - 1], buffered[limit - 1:]
+def split_backlog(queued: list, limit: int = MAX_SAMPLES_PER_POST) -> tuple:
+    """(the samples to send in this post, the ones held for later): the oldest
+    `limit` of the backlog with the new sample queued behind it.
+
+    The new sample waits its turn. Sent ahead of the backlog it reached Steward
+    before older samples did, and a rule counting checks in a row saw each
+    post's run of stalled checks ended by the newest, healthy one: after the
+    2026-10-17 NFS stall "Mount not answering" (10 in a row) never fired
+    (#5995)."""
+    return queued[:limit], queued[limit:]
 
 
 def next_backoff(current: int) -> int:
@@ -2074,10 +2080,10 @@ def main_loop(conf_path: str) -> int:
                               docker_logs_enabled=docker_logs_enabled,
                               docker_log_exclude=docker_log_exclude)
         buffered = buffer.drain()
-        send, held = split_backlog(buffered)
+        send, held = split_backlog(buffered + [sample])
         maint_events, maint_keys = maintenance_results(maint_seen)
         payload = build_payload(
-            samples=send + [sample],
+            samples=send,
             hostname=hostname,
             metadata=metadata,
             events=events + ([lost_contact_event(lost_since)] if lost_since else [])
@@ -2098,7 +2104,7 @@ def main_loop(conf_path: str) -> int:
                 rate_state["updates_due"] = True
             for s in held:
                 buffer.push(s)
-            if send:
+            if buffered:
                 _log("INFO", f"flushed {len(send)} buffered samples"
                              + (f", {len(held)} still held" if held else ""))
             backoff = 0