feat(maintenance): a run spreads swarm services back out after its last host, moving only what the emptier node can start (#6398)
Each weekly run drains Docker02 then Docker03, and the swarm never moves a task back by itself, so every run ended with nearly every service on one node. On 2026-10-25 the operator had that spread out by hand. - Agent 1.26.0 reports each swarm service's pinned digest, and every repo@sha256 its host holds (from /system/df, every 10 min). - Once a run's last host is back in service, Steward picks services on the fullest node to move to the emptiest until they are within one task. It takes services with several replicas on one node first. A service moves only when the emptier node already holds its exact image. One the plan keeps in place, one below its replicas, and one with no pinned digest stay put, and the run says why. - A swarm manager's helper carries it out through a new action, rebalance, with the service names (validated, at most 48). It force-updates each in turn with the spec unchanged and no registry lookup, skips any a constraint places, and gives each 5 min to settle. - The run ends done with what moved, or failed if a move didn't settle or the manager didn't answer in 60 min. A docker-scoped window covers the restarts, and the step goes on the timeline. - Plans gain "Spread swarm services back out" (on by default) and "Keep in place", on the create form and in an "After the run" form on each plan. list_maintenance_plans shows both and the latest run's spread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -310,6 +310,40 @@ Maintenance** (`/plugins/host_agent/maintenance`).
|
||||
last week, then each plan with its schedule, next slot and last five runs.
|
||||
Each host's Agent panel shows its last maintenance job.
|
||||
|
||||
#### Spreading swarm services back out (1.26.0, `rebalance.py`)
|
||||
|
||||
A run drains each swarm node in turn, and the swarm never moves a task back
|
||||
by itself, so every run used to end with the services on the node drained
|
||||
first. A plan with **Spread swarm services back out** (on by default) ends a
|
||||
run that drained a node with a run-level step, once its last host is back in
|
||||
service:
|
||||
|
||||
- **What moves.** While the fullest active node has two or more tasks than
|
||||
the emptiest, a replicated service whose running tasks are all on the
|
||||
fullest moves; one with several replicas there goes first. A service is
|
||||
left where it is, and the run says why, when the plan keeps it in place
|
||||
(**Keep in place**, service names), when it is below its replicas, when its
|
||||
image isn't pinned to a digest, or when the emptier node doesn't hold that
|
||||
digest. Only what kept the nodes from evening out is reported.
|
||||
- **The image check.** Agent 1.26.0 reports each service's pinned digest and
|
||||
every `repo@sha256:…` its host holds (from `/system/df`, every 10 min; a
|
||||
list older than 30 min isn't trusted). A service moves only to a node that
|
||||
already holds its exact image, so a pruned image or a digest the registry
|
||||
has since deleted can't strand it.
|
||||
- **Who moves them.** A swarm manager's helper, through a fifth step action,
|
||||
`rebalance`, carrying the service names (each checked against Docker's name
|
||||
alphabet, at most 48). It force-updates each in turn (`docker service update
|
||||
--force --detach=false`, spec unchanged, no registry lookup), skipping one a
|
||||
constraint or preference places, gives each 5 min to settle and starts no
|
||||
move after 40 min. The swarm starts each on its emptiest node. A stolen
|
||||
manager's agent token can therefore also restart named services in place.
|
||||
- **How it ends.** The run stays `running` until the result arrives, then
|
||||
ends `done` with what moved, or `failed` if a move didn't settle or the
|
||||
manager didn't answer within 60 min. A window over the `docker` figures
|
||||
covers the restarts, and the step goes on the timeline
|
||||
(`maintenance.rebalanced`). With no manager on 1.26.0, or nothing to move,
|
||||
the run ends `done` and says so.
|
||||
|
||||
### The swarm's health, alertable (1.19.0, #5788)
|
||||
|
||||
Every manager's agent reports the whole swarm. From each report the docker plugin (`plugins/docker/swarm_health.py`) records three swarm-wide figures on resource `swarm`, source `docker`:
|
||||
|
||||
Reference in New Issue
Block a user