18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.
Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5777 "Maintenance would make a node the operator keeps drained active after
its reboot". A swarm node already set to drain or pause, such as a manager
kept free of work, is no longer drained before its reboot or made active
after it. Its job only updates and reboots it, and back-in-service needs it
ready but not active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing plugins.docker.models by path defines its tables a second time,
because the plugin manager loaded them as _steward_plugin_docker.models.
Inside the app that raises InvalidRequestError, not ImportError. The cadence
integration test hit it, and #5767's _swarm (used when a job is planned)
carried the same import. Both now go through maintenance.docker_models().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5