Each weekly run drains Docker02 then Docker03, and the swarm never moves a
task back by itself, so every run ended with nearly every service on one
node. On 2026-10-25 the operator had that spread out by hand.
- Agent 1.26.0 reports each swarm service's pinned digest, and every
repo@sha256 its host holds (from /system/df, every 10 min).
- Once a run's last host is back in service, Steward picks services on the
fullest node to move to the emptiest until they are within one task. It
takes services with several replicas on one node first. A service moves only
when the emptier node already holds its exact image. One the plan keeps in
place, one below its replicas, and one with no pinned digest stay put, and
the run says why.
- A swarm manager's helper carries it out through a new action, rebalance,
with the service names (validated, at most 48). It force-updates each in
turn with the spec unchanged and no registry lookup, skips any a
constraint places, and gives each 5 min to settle.
- The run ends done with what moved, or failed if a move didn't settle or
the manager didn't answer in 60 min. A docker-scoped window covers the
restarts, and the step goes on the timeline.
- Plans gain "Spread swarm services back out" (on by default) and "Keep in
place", on the create form and in an "After the run" form on each plan.
list_maintenance_plans shows both and the latest run's spread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.
A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.
get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.
Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5777 "Maintenance would make a node the operator keeps drained active after
its reboot". A swarm node already set to drain or pause, such as a manager
kept free of work, is no longer drained before its reboot or made active
after it. Its job only updates and reboots it, and back-in-service needs it
ready but not active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing plugins.docker.models by path defines its tables a second time,
because the plugin manager loaded them as _steward_plugin_docker.models.
Inside the app that raises InvalidRequestError, not ImportError. The cadence
integration test hit it, and #5767's _swarm (used when a job is planned)
carried the same import. Both now go through maintenance.docker_models().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5