4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 8e2f17d198 fix(maintenance): a late return, once back, stays back, and every reader agrees (#6175)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m31s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m26s
18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.

The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:44:38 -04:00
bvandeusenandClaude Opus 5.5 18032a4782 fix(maintenance): a host that came back late stops reading as failed once it is back (#6105)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m30s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m5s
Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.

Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:23:53 -04:00
bvandeusenandClaude Opus 5.5 72e754b580 fix: error-pages graded healthy, unwired sensors listed apart, Maintenance failed reads 0 (#5780, #5781, #5782)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m23s
CI / unit (push) Successful in 1m29s
CI / publish (push) Successful in 18s
- #5780 "The error-pages service is graded on other routers' errors". The
  Traefik scrape reads which services an errors middleware sends to, and
  grades them healthy: their 4xx and 5xx are the codes of the errors they
  cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
  a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
  page lists sensors that counts_as_hottest leaves out (unwired board
  inputs, 0 °C or below) on one muted line, not as heat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:15:44 -04:00
bvandeusenandClaude Opus 5.5 200384e52d feat(host_agent): a group's updates and reboots on a weekly cadence, one host at a time (#5768)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m20s
CI / unit (push) Successful in 1m34s
CI / publish (push) Skipped
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:59:07 -04:00