The same Holding as the SNMP poll, for the rest of Steward's own
collectors:
- Monitors: the list is read within 10 s or the last one read stands in;
a start the database doesn't take is kept in memory, so a monitor isn't
checked twice; each result is held and stored at the time it was
checked. A stall shows on the monitors as the checks it saw, not a gap.
- UniFi and Traefik: each poll's writes are held (hold_write) and run
oldest first in one transaction; every reading is stamped with its scrape.
- A held round the database refuses (as opposed to can't be reached) is
stored alone and let go, so it can't hold up every round after it.
The Prometheus scrape is left as it is: its one target, Fabled Archives,
keeps its own database on the same NFS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:
- The SNMP poll stored before it polled again, so while the database hung
it stopped polling. It now collects on schedule and hands each round to a
Holding (steward/core/holding.py): one writer stores what is held, oldest
first, each reading at the time it was collected. Bounded at two hours of
polls, with a 30-minute deadline on a store; the device list read falls
back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
Docker03 and never fired: the agent sent each post's backlog with its
newest, healthy sample, and that sample ended every post's run. Agent
1.22.0 queues the new sample behind the backlog, so Steward receives a
host's samples in the order they were taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>