feat(maintenance): a run spreads swarm services back out after its last host, moving only what the emptier node can start (#6398)

Each weekly run drains Docker02 then Docker03, and the swarm never moves a
task back by itself, so every run ended with nearly every service on one
node. On 2026-10-25 the operator had that spread out by hand.

- Agent 1.26.0 reports each swarm service's pinned digest, and every
  repo@sha256 its host holds (from /system/df, every 10 min).
- Once a run's last host is back in service, Steward picks services on the
  fullest node to move to the emptiest until they are within one task. It
  takes services with several replicas on one node first. A service moves only
  when the emptier node already holds its exact image. One the plan keeps in
  place, one below its replicas, and one with no pinned digest stay put, and
  the run says why.
- A swarm manager's helper carries it out through a new action, rebalance,
  with the service names (validated, at most 48). It force-updates each in
  turn with the spec unchanged and no registry lookup, skips any a
  constraint places, and gives each 5 min to settle.
- The run ends done with what moved, or failed if a move didn't settle or
  the manager didn't answer in 60 min. A docker-scoped window covers the
  restarts, and the step goes on the timeline.
- Plans gain "Spread swarm services back out" (on by default) and "Keep in
  place", on the create form and in an "After the run" form on each plan.
  list_maintenance_plans shows both and the latest run's spread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plugins/host-agent-design.md

@@ -310,6 +310,40 @@ Maintenance** (`/plugins/host_agent/maintenance`).
   last week, then each plan with its schedule, next slot and last five runs.
   Each host's Agent panel shows its last maintenance job.
 
+#### Spreading swarm services back out (1.26.0, `rebalance.py`)
+
+A run drains each swarm node in turn, and the swarm never moves a task back
+by itself, so every run used to end with the services on the node drained
+first. A plan with **Spread swarm services back out** (on by default) ends a
+run that drained a node with a run-level step, once its last host is back in
+service:
+
+- **What moves.** While the fullest active node has two or more tasks than
+  the emptiest, a replicated service whose running tasks are all on the
+  fullest moves; one with several replicas there goes first. A service is
+  left where it is, and the run says why, when the plan keeps it in place
+  (**Keep in place**, service names), when it is below its replicas, when its
+  image isn't pinned to a digest, or when the emptier node doesn't hold that
+  digest. Only what kept the nodes from evening out is reported.
+- **The image check.** Agent 1.26.0 reports each service's pinned digest and
+  every `repo@sha256:…` its host holds (from `/system/df`, every 10 min; a
+  list older than 30 min isn't trusted). A service moves only to a node that
+  already holds its exact image, so a pruned image or a digest the registry
+  has since deleted can't strand it.
+- **Who moves them.** A swarm manager's helper, through a fifth step action,
+  `rebalance`, carrying the service names (each checked against Docker's name
+  alphabet, at most 48). It force-updates each in turn (`docker service update
+  --force --detach=false`, spec unchanged, no registry lookup), skipping one a
+  constraint or preference places, gives each 5 min to settle and starts no
+  move after 40 min. The swarm starts each on its emptiest node. A stolen
+  manager's agent token can therefore also restart named services in place.
+- **How it ends.** The run stays `running` until the result arrives, then
+  ends `done` with what moved, or `failed` if a move didn't settle or the
+  manager didn't answer within 60 min. A window over the `docker` figures
+  covers the restarts, and the step goes on the timeline
+  (`maintenance.rebalanced`). With no manager on 1.26.0, or nothing to move,
+  the run ends `done` and says so.
+
 ### The swarm's health, alertable (1.19.0, #5788)
 
 Every manager's agent reports the whole swarm. From each report the docker plugin (`plugins/docker/swarm_health.py`) records three swarm-wide figures on resource `swarm`, source `docker`:

plugins/docker/ingest.py

@@ -239,6 +239,8 @@ async def _persist_swarm(session, host, swarm: dict) -> None:
         row.desired = int(s.get("desired") or 0)
         row.running = int(s.get("running") or 0)
         row.image = s.get("image")
+        digest = s.get("image_digest")
+        row.image_digest = digest[:80] if isinstance(digest, str) else None
         row.placement_json = json.dumps(s.get("placement") or [])
         row.tasks_json = json.dumps(s.get("tasks") or [])
         row.updated_at = now
@@ -298,6 +300,9 @@ async def _persist_disk(session, host, disk: dict) -> None:
     summary.images_active = int(disk.get("images_active") or 0)
     summary.containers_count = int(disk.get("containers_count") or 0)
     summary.volumes_count = int(disk.get("volumes_count") or 0)
+    digests = disk.get("digests")
+    summary.digests_json = json.dumps(
+        [d[:512] for d in digests if isinstance(d, str)][:400] if isinstance(digests, list) else [])
     summary.updated_at = now
     # The totals as metrics too (#5304): a history of each host's growth, and
     # something an alert rule can watch. Resource: the host's name.

plugins/docker/migrations/versions/docker_014_image_digests.py

@@ -0,0 +1,30 @@
+"""Which image a swarm service is pinned to, and which images each host holds
+
+Adds docker_swarm_services.image_digest (the sha256 digest a stack deploy
+pinned) and docker_disk_usage.digests_json (every "repo@sha256:…" the host
+holds), both from agent 1.26.0. A maintenance run spreads services back out
+only onto a node holding the service's image. Older agents leave them empty.
+
+Revision ID: docker_014_image_digests
+Revises: docker_013_mem_used
+Create Date: 2026-10-25
+"""
+from typing import Sequence, Union
+from alembic import op
+import sqlalchemy as sa
+
+revision: str = "docker_014_image_digests"
+down_revision: Union[str, None] = "docker_013_mem_used"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("docker_swarm_services", sa.Column("image_digest", sa.String(80), nullable=True))
+    op.add_column("docker_disk_usage", sa.Column("digests_json", sa.Text, nullable=False,
+                                                 server_default="[]"))
+
+
+def downgrade() -> None:
+    op.drop_column("docker_disk_usage", "digests_json")
+    op.drop_column("docker_swarm_services", "image_digest")

plugins/docker/models.py

@@ -159,6 +159,9 @@ class DockerSwarmService(Base):
     desired: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
     running: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
     image: Mapped[str | None] = mapped_column(String(512), nullable=True)
+    # The digest a stack deploy pinned ("sha256:…"), agent ≥ 1.26.0; None when
+    # the spec names a tag only.
+    image_digest: Mapped[str | None] = mapped_column(String(80), nullable=True)
     # task→node placement of running replicas: [{"node_id", "running"}], JSON.
     # A manager sees every task, so this captures cross-node placement that the
     # local docker_containers rows (this host only) can't.
@@ -196,6 +199,10 @@ class DockerDiskUsage(Base):
     images_active: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
     containers_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
     volumes_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    # Every image the host holds by repository digest, ["repo@sha256:…"], JSON
+    # (agent ≥ 1.26.0): what a swarm service's pinned image is checked against
+    # before a maintenance run moves the service here.
+    digests_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
     updated_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False,
         default=lambda: datetime.now(timezone.utc),

plugins/host_agent/agent.py

@@ -22,7 +22,7 @@ from collections import deque
 from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
 from datetime import datetime, timezone
 
-AGENT_VERSION = "1.25.0"
+AGENT_VERSION = "1.26.0"
 
 # Default path to the local Docker Engine socket. Overridable via the
 # `docker_socket` config key; collection is silently skipped if it's absent or
@@ -585,6 +585,9 @@ DOCKER_DEADLINE_SECONDS = 20.0
 # /system/df walks every image, container and volume: asked this often, not
 # every sample. Steward keeps the last answer between reports.
 DOCKER_DF_INTERVAL_SECONDS = 600.0
+# How many of a host's image digests a report carries: what Steward checks a
+# swarm service's pinned image against before moving it to this node.
+DOCKER_DIGESTS_MAX = 400
 
 # Container-log collection (m79). Logs ride in the same push as metrics, so one
 # interval's batch is capped to keep a chatty container from bloating a POST (and
@@ -1251,11 +1254,14 @@ def _swarm_services(services: list, tasks: list) -> list:
             desired = (mode_obj.get("Replicated") or {}).get("Replicas", 0) or 0
         image = (((spec.get("TaskTemplate") or {}).get("ContainerSpec") or {})
                  .get("Image") or "")
-        image = image.split("@", 1)[0]  # drop the @sha256:… digest suffix
+        image, _, digest = image.partition("@")  # the @sha256:… digest apart
         out.append({
             "service_name": name, "mode": mode,
             "desired": desired, "running": running_by_svc.get(sid, 0),
             "image": image,
+            # The digest a stack deploy pinned, which a node must hold to start
+            # the service without the registry.
+            "image_digest": digest if digest.startswith("sha256:") else None,
             # task→node placement of the running replicas (cross-node — a manager
             # sees every task, even ones whose containers live on other hosts).
             "placement": [{"node_id": n, "running": cnt}
@@ -1383,6 +1389,10 @@ def collect_disk_usage(socket_path: str):
         "volumes_size": sum(_vol_size(v) for v in volumes if _vol_size(v) > 0),
         "build_cache_size": sum(int(b.get("Size") or 0) for b in build_cache),
         "images": _disk_images(images),
+        # Every image this host holds, by repository digest ("repo@sha256:…"),
+        # so Steward can tell whether a swarm service could start here.
+        "digests": sorted({d for im in images for d in (im.get("RepoDigests") or [])
+                           if isinstance(d, str) and "@sha256:" in d})[:DOCKER_DIGESTS_MAX],
     }
 
 
@@ -1667,13 +1677,13 @@ def collect_reboot(path: str = REBOOT_REQUIRED, modules: str = KERNEL_MODULES):
 # The agent can't update or reboot its host: it runs unprivileged under
 # NoNewPrivileges. It leaves a request in MAINT_REQUESTS, the one directory its
 # unit lets it write; steward-maint.path starts the root helper (maint.py),
-# which does only check / update / reboot / drain / undrain and writes each
+# which does only check / update / reboot / drain / undrain / rebalance and writes each
 # outcome to MAINT_RESULTS. The agent reads those back and reports each new state once.
 
 MAINT_REQUESTS = "/var/lib/steward-agent/requests"
 MAINT_RESULTS = "/var/lib/steward-maint/results"
 MAINT_PATH_UNIT = "/etc/systemd/system/steward-maint.path"
-MAINT_ACTIONS = ("check", "update", "reboot", "drain", "undrain", "drives")
+MAINT_ACTIONS = ("check", "update", "reboot", "drain", "undrain", "drives", "rebalance")
 MAINT_RESULT_MAX_BYTES = 8192
 # The helper keeps results a week; only this recent a one is reported, so a
 # job a reboot interrupted still reaches Steward from the agent's next start
@@ -1683,7 +1693,9 @@ _MAINT_ID = re.compile(r"^[A-Za-z0-9-]{1,64}$")
 _MAINT_FIELDS = ("action", "state", "ok", "packages", "reboot_required", "node",
                  "detail", "started_at", "finished_at",
                  # a check's answer (#6098)
-                 "pending", "docker_update", "docker_packages", "containers")
+                 "pending", "docker_update", "docker_packages", "containers",
+                 # a rebalance's answer
+                 "moved", "left")
 
 
 def maint_helper_installed(unit: str = MAINT_PATH_UNIT,
@@ -1693,14 +1705,17 @@ def maint_helper_installed(unit: str = MAINT_PATH_UNIT,
 
 
 def request_maintenance(job_id: str, action: str, node: str | None = None,
-                        requests_dir: str = MAINT_REQUESTS) -> None:
+                        requests_dir: str = MAINT_REQUESTS, services=None) -> None:
     """Leave the helper a request, whole: written to a dot-file the path unit
-    doesn't match, then renamed to `<id>.json`."""
+    doesn't match, then renamed to `<id>.json`. A rebalance names the swarm
+    services to move; the helper checks each name itself."""
     if action not in MAINT_ACTIONS or not _MAINT_ID.match(job_id or ""):
         raise ValueError(f"not a maintenance request: {action!r} {job_id!r}")
     req = {"id": job_id, "action": action}
     if node:
         req["node"] = node
+    if action == "rebalance":
+        req["services"] = [s for s in services if isinstance(s, str)] if isinstance(services, list) else []
     tmp = os.path.join(requests_dir, f".{job_id}.tmp")
     with open(tmp, "w") as fh:
         json.dump(req, fh)
@@ -1718,7 +1733,8 @@ def take_maintenance(steps, requested: set, requests_dir: str = MAINT_REQUESTS)
             continue
         try:
             request_maintenance(str(step.get("id") or ""), str(step.get("action") or ""),
-                                step.get("node"), requests_dir=requests_dir)
+                                step.get("node"), requests_dir=requests_dir,
+                                services=step.get("services"))
         except (ValueError, OSError) as exc:
             _log("WARN", f"maintenance step {step.get('id')!r} not taken: {exc}")
             continue

plugins/host_agent/cadence.py

@@ -10,7 +10,8 @@ active, every replicated service at its replicas. A job that fails, or a host
 that doesn't come back into service, stops the run, and the host's
 maintenance_failed figure goes to 1 for an alert rule to catch. No host is
 started after the window; the run then stops and says which hosts it didn't
-reach.
+reach. A run that drained swarm nodes ends by spreading the swarm's services
+back out (rebalance.py), unless its plan says not to.
 
 The decisions are model-free here (plans and runs are anything with the
 attributes); the database side is below them, and the plugin's minute tick
@@ -22,7 +23,7 @@ from __future__ import annotations
 import json
 from datetime import datetime, time, timedelta, timezone
 
-from . import maintenance, timeline
+from . import maintenance, rebalance, timeline
 
 WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
 # How long a host may take to be back in service after its job finished.
@@ -244,11 +245,14 @@ async def advance(session, run, plan, now: datetime) -> str:
                 await timeline.write(session, f"maintenance job {job.id}",
                                      timeline.job_entry(session, job, host.name if host else None))
             name = host.name if host else host_id
-            _end(run, "failed", now, f"{name}: {why}")
+            _end(run, "failed", f"{name}: {why}", now=now)
         elif move == "stop":
-            _end(run, "stopped", now, why)
+            _end(run, "stopped", why, now=now)
         elif move == "done":
-            _end(run, "done", now, None)
+            if await rebalance.begin(session, run, plan, now) or \
+                    (run.rebalance or {}).get("state") == "handed":
+                return run.state            # a manager is spreading services back out
+            _end(run, *rebalance.outcome(run), now=now)
         return run.state
     return run.state
 
@@ -262,7 +266,7 @@ def _skip(run, host_id: str, reason: str) -> None:
     run.host_ids = json.dumps(hosts)
 
 
-def _end(run, state: str, now: datetime, detail: str | None) -> None:
+def _end(run, state: str, detail: str | None = None, *, now: datetime) -> None:
     run.state, run.finished_at, run.settle_since = state, now, None
     if detail:
         run.detail = detail[:maintenance.DETAIL_MAX]
@@ -289,6 +293,8 @@ async def tick(session, now: datetime) -> list[str]:
         plan = await session.get(HostMaintenancePlan, run.plan_id)
         if plan is None:
             continue
+        if rebalance.check_deadline(run, now):
+            await rebalance.finish(session, run, now)
         before = run.state
         after = await advance(session, run, plan, now)
         if after != before:

plugins/host_agent/files/maint.py

@@ -8,7 +8,8 @@ REQUESTS, the one directory it may write. This helper reads each request
 without following links, refuses anything outside the list, deletes it, does
 the action, and writes the outcome to RESULTS (root's own directory, which the
 agent reads back and reports to Steward). A stolen agent token can therefore
-make its host update, reboot or drain, and nothing else.
+make its host update, reboot or drain, or on a swarm manager restart named
+services where the swarm puts them, and nothing else.
 
 One action only reads: `drives` (#5789) asks smartctl for each disk's health,
 which needs root, and writes a summary to DRIVES, never a result: it is not a
@@ -16,6 +17,12 @@ maintenance job. `check` (#6098) changes nothing but the package lists: it
 says what an update would upgrade, whether Docker is among it, and which
 containers run outside the swarm.
 
+`rebalance` spreads swarm services back out after a maintenance run left them
+on one node. Steward names the services, having checked that the emptier node
+holds each one's image; the helper force-updates them one at a time, spec
+unchanged and no image pulled, so the swarm places each on its emptiest node.
+A service a constraint or a preference places is left alone.
+
 Stdlib only; served by Steward at /plugins/host_agent/files/maint.py.
 """
 from __future__ import annotations
@@ -32,7 +39,7 @@ from datetime import datetime, timezone
 REQUESTS = "/var/lib/steward-agent/requests"
 RESULTS = "/var/lib/steward-maint/results"
 DRIVES = "/var/lib/steward-maint/drives.json"
-ACTIONS = ("check", "update", "reboot", "drain", "undrain", "drives")
+ACTIONS = ("check", "update", "reboot", "drain", "undrain", "drives", "rebalance")
 REQUEST_MAX_BYTES = 4096
 RESULTS_KEPT_SECONDS = 7 * 86400
 UPDATE_DEADLINE_SECONDS = 3000
@@ -42,6 +49,12 @@ OUTPUT_KEPT_CHARS = 2000
 LISTING_KEPT_CHARS = 1_000_000
 _ID = re.compile(r"^[A-Za-z0-9-]{1,64}$")
 _NODE = re.compile(r"^[a-z0-9]{25}$")
+# A swarm service's name as Docker allows it, never a flag.
+_SERVICE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$")
+SERVICES_MAX = 48
+MOVE_DEADLINE_SECONDS = 300
+# A rebalance starts no move after this long.
+REBALANCE_SECONDS = 2400
 
 
 def now_iso() -> str:
@@ -70,12 +83,19 @@ def read_request(path: str):
 
 
 def valid(req) -> bool:
-    """A request names a known action, has a plain id, and a node id only in
+    """A request names a known action, has a plain id, a node id only in the
-    the swarm's own form."""
+    swarm's own form, and, for a rebalance alone, a list of service names."""
     if not isinstance(req, dict) or req.get("action") not in ACTIONS:
         return False
     if not isinstance(req.get("id"), str) or not _ID.match(req["id"]):
         return False
+    services = req.get("services")
+    if req["action"] == "rebalance":
+        if not (isinstance(services, list) and 0 < len(services) <= SERVICES_MAX
+                and all(isinstance(s, str) and _SERVICE.match(s) for s in services)):
+            return False
+    elif services is not None:
+        return False
     node = req.get("node")
     return node is None or (isinstance(node, str) and bool(_NODE.match(node)))
 
@@ -344,6 +364,52 @@ def do_node(availability: str, node: str | None) -> dict:
     return {"ok": ok, "node": node, "detail": out.strip()[-500:]}
 
 
+def placed_by_rule(name: str) -> bool | None:
+    """Whether a constraint or a preference decides where the service runs
+    (a force-update would leave it where it is); None when it can't be read."""
+    ok, out = run(["docker", "service", "inspect", "--format",
+                   "{{json .Spec.TaskTemplate.Placement}}", name], deadline=30)
+    if not ok:
+        return None
+    try:
+        placement = json.loads(out.strip() or "null") or {}
+    except ValueError:
+        return None
+    return bool(placement.get("Constraints") or placement.get("Preferences"))
+
+
+def do_rebalance(services: list, clock=time.monotonic) -> dict:
+    """Force-update each named service in turn, waiting for it to settle, so
+    the swarm starts it again on its emptiest node. Fails when a move didn't
+    settle; what was left and why is in `left`."""
+    if not shutil.which("docker"):
+        return {"ok": False, "detail": "no docker on this host"}
+    began = clock()
+    moved, left, failed = [], [], []
+    for name in services:
+        if clock() - began > REBALANCE_SECONDS:
+            left.append({"service": name, "reason": "out of time"})
+            continue
+        rule = placed_by_rule(name)
+        if rule is None:
+            left.append({"service": name, "reason": "not found on this manager"})
+            continue
+        if rule:
+            left.append({"service": name, "reason": "placed by a constraint"})
+            continue
+        ok, out = run(["docker", "service", "update", "--force", "--detach=false", "--quiet", name],
+                      deadline=MOVE_DEADLINE_SECONDS)
+        if ok:
+            moved.append(name)
+        else:
+            failed.append(name)
+            left.append({"service": name, "reason": f"didn't settle: {out.strip()[-120:]}"})
+    detail = f"moved {len(moved)} of {len(services)}"
+    if failed:
+        detail += "; didn't settle: " + ", ".join(failed)
+    return {"ok": not failed, "moved": moved, "left": left, "detail": detail}
+
+
 # ── drive health (#5789) ─────────────────────────────────────────────────────
 # smartctl ≥ 7.0 answers in JSON, the same for SATA, SAS and NVMe drives on any
 # distribution; an older one is reported, never parsed as text. The helper
@@ -509,6 +575,8 @@ def handle(req: dict) -> dict:
         outcome = do_update()
     elif action in ("drain", "undrain"):
         outcome = do_node("drain" if action == "drain" else "active", req.get("node"))
+    elif action == "rebalance":
+        outcome = do_rebalance(req["services"])
     else:                                            # reboot
         result = {**base, "state": "rebooting", "ok": True, "finished_at": now_iso()}
         write_result(result)

plugins/host_agent/maintenance.py

@@ -409,8 +409,9 @@ async def apply_events(session, host_id: str, events, now: datetime) -> int:
     """Apply one post's `maintenance` results and `started` boots to the jobs
     they belong to; returns how many jobs changed. A result is matched by its
     step id, so a drain reported by a manager reaches the job of the node it
-    drained."""
+    drained. A result no job owns may be a run's rebalance (rebalance.py)."""
     from sqlalchemy import select
+    from . import rebalance
     from .downtime import _at
     from .models import HostMaintenanceJob
     if not isinstance(events, list):
@@ -425,6 +426,8 @@ async def apply_events(session, host_id: str, events, now: datetime) -> int:
                 HostMaintenanceJob.state.in_(ACTIVE)))).scalar_one_or_none()
             if job is not None and apply_result(job, event, now):
                 changed.add(job)
+            elif job is None:
+                await rebalance.apply_event(session, event, now)
         elif event.get("kind") == "started":
             boot_at = _at(event.get("boot_at"))
             for job in (await session.execute(select(HostMaintenanceJob).where(
@@ -441,8 +444,10 @@ async def handoffs(session, host_id: str, host_name: str, now: datetime,
                    agent_version: str | None = None) -> list[dict]:
     """The steps this host's agent should carry out now. A queued job for the
     host starts here, when its agent is known to be listening, with a check
-    when the agent is new enough to hand one over."""
+    when the agent is new enough to hand one over. A manager also carries out
+    a run's rebalance."""
     from sqlalchemy import or_, select
+    from . import rebalance
     from .models import HostMaintenanceJob
     jobs = (await session.execute(select(HostMaintenanceJob).where(
         HostMaintenanceJob.state.in_(ACTIVE),
@@ -458,7 +463,7 @@ async def handoffs(session, host_id: str, host_name: str, now: datetime,
             step = handoff(job)
             if step is not None:
                 out.append(step)
-    return out
+    return out + await rebalance.handoffs(session, host_id)
 
 
 async def _running(session, job) -> set:

plugins/host_agent/mcp_tools.py

@@ -15,7 +15,7 @@ from steward.mcp.tools.overview import finding
 from steward.models.hosts import Host
 from steward.models.users import UserRole
 
-from . import cadence, maintenance, senders
+from . import cadence, maintenance, rebalance, senders
 from .models import HostAgentRegistration, HostMaintenanceJob, HostMaintenancePlan, HostMaintenanceRun
 
 # A registration this old that has never checked in is a failed install, not one in progress.
@@ -143,6 +143,12 @@ async def list_maintenance_plans() -> dict:
     (`skipped`), and how it ended. `last_slot_at` is the last scheduled slot
     that fired, null until one has. get_maintenance_job shows one host's job.
     Plans are created and edited on the Maintenance page.
+
+    With `spread_after_run`, a run that drained swarm nodes ends by spreading
+    the swarm's services back out, leaving `keep_in_place` where they are;
+    `latest_run.spread` says what it moved and what it left and why (state:
+    handed, done, failed, timed_out, or nothing to move), null when the run
+    had none.
     """
     from steward.models.ansible_inventory import AnsibleGroup
     now = now_utc()
@@ -169,6 +175,8 @@ async def list_maintenance_plans() -> dict:
             "weekday": cadence.WEEKDAYS[plan.weekday], "at_time": plan.at_time,
             "time_zone": plan.time_zone, "window_minutes": plan.window_minutes,
             "reboot_policy": plan.reboot_policy, "enabled": plan.enabled,
+            "spread_after_run": plan.spread,
+            "keep_in_place": rebalance.keep_list(plan.keep_in_place),
             "next_slot_at": iso(nxt), "last_slot_at": iso(cadence.fired_slot(plan)),
             "latest_run": None,
         }
@@ -181,12 +189,23 @@ async def list_maintenance_plans() -> dict:
                 "skipped": [{"host": names.get(x["host_id"], x["host_id"]), "reason": x["reason"]}
                             for x in json.loads(run.skipped or "[]")],
                 "detail": run.detail,
+                "spread": _spread_row(run.rebalance),
                 "started_at": iso(run.started_at), "finished_at": iso(run.finished_at),
             }
         out.append(row)
     return {"plans": out, "total": len(out)}
 
 
+def _spread_row(rb: dict | None) -> dict | None:
+    """A run's rebalance as the tool shows it: no step ids or window ids."""
+    if not rb:
+        return None
+    return {"state": rb.get("state"), "planned": rb.get("planned") or [],
+            "moved": rb.get("moved") or [], "left": rb.get("left") or [],
+            "detail": rb.get("detail"), "started_at": rb.get("at"),
+            "finished_at": rb.get("finished_at")}
+
+
 async def get_maintenance_job(host: str = "", job: str = "", limit: int = 10) -> dict:
     """A host's maintenance jobs, newest first (its running one first), or one
     job by id.

plugins/host_agent/migrations/versions/host_agent_013_spread.py

@@ -0,0 +1,28 @@
+# plugins/host_agent/migrations/versions/host_agent_013_spread.py
+"""host_agent: a maintenance run spreads swarm services back out after its
+last host, and a plan can keep named services where they are
+
+Revision ID: host_agent_013_spread
+"""
+from typing import Sequence, Union
+from alembic import op
+import sqlalchemy as sa
+
+revision: str = "host_agent_013_spread"
+down_revision: Union[str, None] = "host_agent_012_ends_as"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("host_maintenance_plans", sa.Column("spread", sa.Boolean, nullable=False,
+                                                      server_default=sa.true()))
+    op.add_column("host_maintenance_plans", sa.Column("keep_in_place", sa.Text, nullable=False,
+                                                      server_default=""))
+    op.add_column("host_maintenance_runs", sa.Column("rebalance", sa.JSON, nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column("host_maintenance_runs", "rebalance")
+    op.drop_column("host_maintenance_plans", "keep_in_place")
+    op.drop_column("host_maintenance_plans", "spread")

plugins/host_agent/models.py

@@ -134,7 +134,9 @@ class HostMaintenancePlan(Base):
     `at_time` in `time_zone`, its hosts are updated one at a time, each
     rebooting per `reboot_policy`. No host is started after `window_minutes`
     from the run's start. last_slot_at is the last scheduled slot that fired;
-    Run now leaves it alone, so the schedule keeps its time."""
+    Run now leaves it alone, so the schedule keeps its time. With `spread`, a
+    run that drained swarm nodes ends by spreading services back out, leaving
+    the services named in `keep_in_place` (comma-separated) where they are."""
     __tablename__ = "host_maintenance_plans"
 
     id = Column(String(36), primary_key=True, default=_uuid)
@@ -147,6 +149,8 @@ class HostMaintenancePlan(Base):
     window_minutes = Column(Integer, nullable=False, default=240)
     reboot_policy = Column(String(16), nullable=False, default="if_required")
     enabled = Column(Boolean, nullable=False, default=True)
+    spread = Column(Boolean, nullable=False, default=True)
+    keep_in_place = Column(Text, nullable=False, default="")
     last_slot_at = Column(DateTime(timezone=True), nullable=True)
     created_by = Column(String(36), ForeignKey("users.id", ondelete="SET NULL"), nullable=True)
     created_at = Column(DateTime(timezone=True), nullable=False, default=_utcnow)
@@ -158,7 +162,10 @@ class HostMaintenanceRun(Base):
     host_ids is the order fixed at the start (JSON list); position is the host
     being worked on; skipped is [{"host_id", "reason"}] for hosts the run
     passed over. settle_since is when the current host's job finished and the
-    run began waiting for the host to be back in service."""
+    run began waiting for the host to be back in service. rebalance is the
+    spreading of swarm services back out once the last host is back
+    (rebalance.py): its state, the manager carrying it out, what it moved and
+    what it left and why; None while not reached or not wanted."""
     __tablename__ = "host_maintenance_runs"
 
     id = Column(String(36), primary_key=True, default=_uuid)
@@ -171,5 +178,6 @@ class HostMaintenanceRun(Base):
     settle_since = Column(DateTime(timezone=True), nullable=True)
     detail = Column(Text, nullable=True)
     manual = Column(Boolean, nullable=False, default=False)
+    rebalance = Column(JSON, nullable=True)
     started_at = Column(DateTime(timezone=True), nullable=False, default=_utcnow)
     finished_at = Column(DateTime(timezone=True), nullable=True)

plugins/host_agent/rebalance.py

@@ -0,0 +1,374 @@
+# plugins/host_agent/rebalance.py
+"""A maintenance run spreads swarm services back out after its last host.
+
+A run drains each swarm node in turn, and the swarm never moves a task back
+on its own, so every run used to end with the services on whichever node was
+drained first. Once the run's last host is back in service, Steward picks
+services to move from the fullest node to the emptiest and hands the list to
+a swarm manager's helper (maint.py `rebalance`), which force-updates each in
+turn: spec unchanged, no image pulled, and the swarm starts it again on its
+emptiest node.
+
+A service moves only when the emptier node already holds the exact image the
+service is pinned to (its agent reports every image digest it holds), so a
+pruned image or a digest the registry has since deleted never strands one.
+A service the plan keeps in place, one below its replicas, and one a
+constraint places are left where they are, and the run says why.
+
+The choice is model-free (`choose`); the database side is below it, driven by
+cadence.advance, the agent's posts (handoffs, results) and the minute tick.
+"""
+from __future__ import annotations
+
+import json
+from datetime import datetime, timedelta
+
+from . import maintenance
+
+# The first agent whose helper can rebalance, and that reports image digests.
+REBALANCE_AGENT = (1, 26, 0)
+# How many services one rebalance moves at most; the helper refuses more.
+SERVICES_MAX = 48
+# A node's list of images older than this isn't trusted (agents report it
+# every 10 minutes).
+DIGESTS_FRESH = timedelta(minutes=30)
+# The helper starts no move after 40 min and gives each 5 to settle.
+DEADLINE = timedelta(minutes=60)
+STATES = ("handed", "done", "failed", "timed_out", "nothing")
+WINDOW_NAME = "Maintenance: spreading swarm services back out"
+
+
+def keep_list(text: str | None) -> list[str]:
+    """A plan's keep_in_place as names, in the order written."""
+    out = []
+    for name in (text or "").replace("\n", ",").split(","):
+        name = name.strip()
+        if name and name not in out:
+            out.append(name)
+    return out
+
+
+def _repo(image: str) -> str:
+    """An image reference's repository, without tag or digest, in the short
+    form Docker reports (no docker.io/ or library/)."""
+    ref = image.split("@", 1)[0]
+    slash = ref.rfind("/")
+    colon = ref.rfind(":")
+    if colon > slash:
+        ref = ref[:colon]
+    for prefix in ("docker.io/", "index.docker.io/"):
+        if ref.startswith(prefix):
+            ref = ref[len(prefix):]
+    if ref.startswith("library/"):
+        ref = ref[len("library/"):]
+    return ref
+
+
+def pinned(image: str | None, digest: str | None) -> str | None:
+    """"repo@sha256:…" for a service's image, or None when it isn't pinned."""
+    if not image or not digest or not digest.startswith("sha256:"):
+        return None
+    return f"{_repo(image)}@{digest}"
+
+
+def held(digests) -> set:
+    """A node's reported digests, in the same form as pinned()."""
+    return {f"{_repo(d)}@{d.split('@', 1)[1]}" for d in digests or []
+            if isinstance(d, str) and "@sha256:" in d}
+
+
+def choose(services: list, nodes: dict, images: dict, keep=(),
+           limit: int = SERVICES_MAX) -> tuple[list, list]:
+    """(services to move, [{"service", "reason"}] left where they are).
+
+    services: {"name", "mode", "desired", "running", "image" ("repo@sha256:…"
+    or None), "placement": [{"node_id", "running"}]}; nodes: {node id: name}
+    for the nodes to balance across (ready and active); images: {node id: set
+    of "repo@sha256:…"}, a node missing when its list is unknown.
+
+    While the fullest node has two or more tasks than the emptiest, a service
+    whose tasks all run on the fullest moves; one with several replicas there
+    first, as one node holding them all defeats the replicas. A move counts
+    as its share of the replicas going to the emptiest. What is reported left
+    is what kept the nodes from evening out: the services that would have
+    moved next but can't."""
+    if len(nodes) < 2:
+        return [], []
+    counts = {n: 0 for n in nodes}
+    for s in services:
+        for p in s.get("placement") or []:
+            if p.get("node_id") in counts:
+                counts[p["node_id"]] += int(p.get("running") or 0)
+    keep = set(keep)
+    order = sorted(services, key=lambda s: (-int(s.get("running") or 0), s["name"]))
+    moves: list = []
+    left: dict = {}
+    while len(moves) < limit:
+        src = max(counts, key=lambda n: (counts[n], n))
+        dst = min(counts, key=lambda n: (counts[n], n))
+        gap = counts[src] - counts[dst]
+        if gap < 2:
+            break
+        pick, blocked = None, {}
+        for s in order:
+            name, running = s["name"], int(s.get("running") or 0)
+            on = {p.get("node_id"): int(p.get("running") or 0) for p in s.get("placement") or []
+                  if int(p.get("running") or 0) > 0}
+            if name in moves or s.get("mode") != "replicated" or set(on) != {src}:
+                continue
+            share = max(1, running // len(nodes))
+            if 2 * share > gap:
+                continue                    # it would only tip the balance the other way
+            if name in keep:
+                blocked[name] = "kept in place by the plan"
+            elif running < int(s.get("desired") or 0):
+                blocked[name] = "below its replicas"
+            elif not s.get("image"):
+                blocked[name] = "its image isn't pinned to a digest"
+            elif dst not in images:
+                blocked[name] = f"no image list from {nodes[dst]} (its agent needs 1.26.0)"
+            elif s["image"] not in images[dst]:
+                blocked[name] = f"its image isn't on {nodes[dst]}"
+            else:
+                pick = s
+                break
+        if pick is None:
+            left = blocked                  # what kept the nodes from evening out
+            break
+        share = max(1, int(pick.get("running") or 0) // len(nodes))
+        moves.append(pick["name"])
+        counts[src] -= share
+        counts[dst] += share
+    return moves, [{"service": n, "reason": r} for n, r in sorted(left.items())]
+
+
+def can_rebalance(agent_version) -> bool:
+    try:
+        return tuple(int(x) for x in str(agent_version).split(".")[:3]) >= REBALANCE_AGENT
+    except ValueError:
+        return False
+
+
+def step_id(run) -> str:
+    return f"{run.id}-rebalance"
+
+
+def _set(run, **changes) -> None:
+    """Change the run's rebalance record (a new dict, so the JSON column
+    sees the change)."""
+    run.rebalance = {**(run.rebalance or {}), **changes}
+
+
+def handoff(run) -> dict | None:
+    """The step to hand the manager's agent, until its result arrives."""
+    rb = run.rebalance or {}
+    if rb.get("state") != "handed" or rb.get("seen"):
+        return None
+    return {"id": rb["step_id"], "action": "rebalance", "services": rb.get("planned") or []}
+
+
+def apply_result(run, event: dict, now: datetime) -> bool:
+    """A `maintenance` event for the run's rebalance; True when it changed."""
+    rb = run.rebalance or {}
+    if rb.get("state") != "handed" or event.get("id") != rb.get("step_id"):
+        return False
+    state = event.get("state")
+    if state == "running":
+        if rb.get("seen"):
+            return False
+        _set(run, seen=True)
+        return True
+    if state not in ("done", "failed"):
+        return False
+    moved = [str(m)[:64] for m in event.get("moved") or [] if isinstance(m, str)][:SERVICES_MAX]
+    gone = [{"service": str(x.get("service"))[:64], "reason": str(x.get("reason"))[:160]}
+            for x in event.get("left") or [] if isinstance(x, dict)][:SERVICES_MAX]
+    ok = state == "done" and event.get("ok") is not False
+    _set(run, state="done" if ok else "failed", moved=moved, left=(rb.get("left") or []) + gone,
+         detail=str(event.get("detail") or "")[:maintenance.DETAIL_MAX] or None,
+         finished_at=now.isoformat())
+    return True
+
+
+def check_deadline(run, now: datetime) -> bool:
+    rb = run.rebalance or {}
+    if rb.get("state") != "handed" or now - datetime.fromisoformat(rb["at"]) <= DEADLINE:
+        return False
+    _set(run, state="timed_out", finished_at=now.isoformat(),
+         detail=f"the manager didn't finish within {int(DEADLINE.total_seconds() // 60)} min")
+    return True
+
+
+def outcome(run) -> tuple[str, str | None]:
+    """How a run whose hosts are all done ends: ("done" | "failed", detail)."""
+    rb = run.rebalance or {}
+    state = rb.get("state")
+    left = rb.get("left") or []
+    tail = f"; {len(left)} left where they are" if left else ""
+    if state == "done":
+        n = len(rb.get("moved") or [])
+        return "done", f"spread {n} swarm service{'s' if n != 1 else ''} back out{tail}"
+    if state in ("failed", "timed_out"):
+        return "failed", f"spreading swarm services back out: {rb.get('detail') or state}"
+    if state == "nothing" and rb.get("detail"):
+        return "done", f"swarm services not spread back out: {rb['detail']}"
+    if state == "nothing" and left:
+        return "done", f"no swarm service could be spread back out{tail}"
+    return "done", None
+
+
+def label(run) -> tuple[str, bool, bool] | None:
+    """(text, bad, busy) for the run's swarm row on the page, or None when
+    the run has no rebalance."""
+    rb = run.rebalance or {}
+    state = rb.get("state")
+    if state is None:
+        return None
+    if state == "handed":
+        n = len(rb.get("planned") or [])
+        return f"Spreading {n} service{'s' if n != 1 else ''} back out", False, True
+    text = outcome(run)[1] or "Nothing to spread back out"
+    return text[:1].upper() + text[1:], state in ("failed", "timed_out"), False
+
+
+# ── applied to the database ──────────────────────────────────────────────────
+
+async def _freshest(session, model, key: str) -> list:
+    """The newest row per `key` (several managers may report the swarm)."""
+    from sqlalchemy import select
+    rows: dict = {}
+    for r in (await session.execute(select(model))).scalars():
+        k = getattr(r, key)
+        if k not in rows or r.updated_at > rows[k].updated_at:
+            rows[k] = r
+    return list(rows.values())
+
+
+async def _swarm_view(session, now: datetime) -> tuple[list, dict, dict] | None:
+    """(services, nodes, images) as choose() takes them, from what the Docker
+    plugin last stored; None without it."""
+    from sqlalchemy import select
+    from steward.models.hosts import Host
+    models = maintenance.docker_models()
+    if models is None:
+        return None
+    services = [{"name": s.service_name, "mode": s.mode, "desired": s.desired,
+                 "running": s.running, "image": pinned(s.image, s.image_digest),
+                 "placement": json.loads(s.placement_json or "[]")}
+                for s in await _freshest(session, models.DockerSwarmService, "service_name")]
+    nodes = {n.node_id: n.hostname or n.node_id
+             for n in await _freshest(session, models.DockerSwarmNode, "node_id")
+             if n.status == "ready" and n.availability == "active"}
+    hosts = dict((await session.execute(select(Host.name, Host.id))).all())
+    by_name = {name.lower(): hid for name, hid in hosts.items()}
+    images = {}
+    for node_id, name in nodes.items():
+        host_id = by_name.get(name.lower())
+        disk = await session.get(models.DockerDiskUsage, host_id) if host_id else None
+        if disk is not None and now - disk.updated_at <= DIGESTS_FRESH:
+            got = held(json.loads(disk.digests_json or "[]"))
+            if got:
+                images[node_id] = got
+    return services, nodes, images
+
+
+async def _manager(session, jobs: list) -> tuple[str | None, str | None]:
+    """(host id, None) of a swarm manager whose helper can rebalance, the one
+    that drained the run's nodes first; (None, why) when there is none."""
+    from sqlalchemy import select
+    from .models import HostAgentRegistration
+    _, managers, _ = await maintenance._swarm(session)
+    regs = {r.host_id: r for r in (await session.execute(select(HostAgentRegistration).where(
+        HostAgentRegistration.host_id.in_(list(managers)),
+        HostAgentRegistration.maint_helper.is_(True)))).scalars()} if managers else {}
+    drained_by = [j.drain_host_id for j in jobs if j.drain_host_id]
+    for host_id in drained_by + sorted(regs):
+        reg = regs.get(host_id)
+        if reg is not None and can_rebalance(reg.agent_version):
+            return host_id, None
+    if regs:
+        return None, "no swarm manager's agent can do it yet: run Update agent on a manager"
+    return None, "no swarm manager has the maintenance helper"
+
+
+async def begin(session, run, plan, now: datetime) -> bool:
+    """The run's last host is back in service: hand a manager the services to
+    move, or record why there are none. True while a rebalance is handed out."""
+    from sqlalchemy import select
+    from .models import HostMaintenanceJob
+    if not plan.spread or run.rebalance is not None:
+        return False
+    jobs = (await session.execute(select(HostMaintenanceJob).where(
+        HostMaintenanceJob.run_id == run.id))).scalars().all()
+    if not any(maintenance.drains(j) for j in jobs):
+        return False                        # nothing was drained: nothing to put back
+    view = await _swarm_view(session, now)
+    if view is None:
+        return False
+    services, nodes, images = view
+    moves, left = choose(services, nodes, images, keep_list(plan.keep_in_place))
+    if not moves:
+        run.rebalance = {"state": "nothing", "left": left, "at": now.isoformat()}
+        return False
+    host_id, why = await _manager(session, jobs)
+    if host_id is None:
+        run.rebalance = {"state": "nothing", "left": left, "detail": why, "at": now.isoformat()}
+        return False
+    run.rebalance = {"state": "handed", "step_id": step_id(run), "host_id": host_id,
+                     "at": now.isoformat(), "planned": moves, "left": left, "seen": False,
+                     "window_id": await _window(session, plan, now)}
+    return True
+
+
+async def _window(session, plan, now: datetime) -> str | None:
+    """A window over the Docker figures while services restart, so their
+    restarts raise no alert; ended when the rebalance does."""
+    from steward.models.alerts import MaintenanceWindow
+    if plan.created_by is None:
+        return None
+    w = MaintenanceWindow(name=WINDOW_NAME, start_at=now, end_at=now + DEADLINE,
+                          scope_source="docker", scope_resource=None, created_by=plan.created_by)
+    session.add(w)
+    await session.flush()
+    return w.id
+
+
+async def finish(session, run, now: datetime) -> None:
+    """The rebalance ended: its window ends, and it goes on the timeline."""
+    from steward.models.alerts import MaintenanceWindow
+    from . import timeline
+    rb = run.rebalance or {}
+    if rb.get("window_id"):
+        w = await session.get(MaintenanceWindow, rb["window_id"])
+        if w is not None:
+            w.end_at = max(w.start_at, now)
+    if rb.get("state") in ("done", "failed", "timed_out"):
+        await timeline.write(session, f"maintenance run {run.id}", timeline.run_entry(session, run))
+
+
+async def running_runs(session) -> list:
+    from sqlalchemy import select
+    from .models import HostMaintenanceRun
+    return (await session.execute(select(HostMaintenanceRun).where(
+        HostMaintenanceRun.state == "running"))).scalars().all()
+
+
+async def handoffs(session, host_id: str) -> list[dict]:
+    """The rebalance steps this host's agent should carry out now."""
+    out = []
+    for run in await running_runs(session):
+        if (run.rebalance or {}).get("host_id") == host_id:
+            step = handoff(run)
+            if step is not None:
+                out.append(step)
+    return out
+
+
+async def apply_event(session, event: dict, now: datetime) -> bool:
+    """Apply a `maintenance` event to the running run whose rebalance it is."""
+    for run in await running_runs(session):
+        if apply_result(run, event, now):
+            if (run.rebalance or {}).get("state") != "handed":
+                await finish(session, run, now)
+            return True
+    return False

plugins/host_agent/routes.py

@@ -1442,8 +1442,10 @@ async def update_via_ansible():
 # ── Maintenance cadence (#5768) ───────────────────────────────────────────────
 
 def _run_rows(run, jobs: dict, names: dict) -> list[dict]:
-    """A run's hosts in order, each with where it stands, then those passed over."""
+    """A run's hosts in order, each with where it stands, then those passed
+    over, then the spreading of its swarm services back out."""
     import json as _json
+    from . import rebalance
     rows = []
     for i, host_id in enumerate(_json.loads(run.host_ids)):
         job = jobs.get((run.id, host_id))
@@ -1462,6 +1464,13 @@ def _run_rows(run, jobs: dict, names: dict) -> list[dict]:
     for s in _json.loads(run.skipped):
         rows.append({"host_id": s["host_id"], "name": names.get(s["host_id"], s["host_id"]),
                      "text": f"Skipped: {s['reason']}", "bad": False, "busy": False, "detail": None})
+    swarm = rebalance.label(run)
+    if swarm is not None:
+        rb = run.rebalance or {}
+        said = [f"moved: {', '.join(rb['moved'])}"] if rb.get("moved") else []
+        said += [f"{x['service']}: {x['reason']}" for x in rb.get("left") or []]
+        rows.append({"host_id": None, "name": "Swarm services", "text": swarm[0], "bad": swarm[1],
+                     "busy": swarm[2], "detail": "; ".join(said) or None})
     return rows
 
 
@@ -1532,6 +1541,7 @@ async def maintenance_plan_create():
         return _error(400, "invalid", "weekday, time (HH:MM) and window must be numbers")
     policy = form.get("reboot_policy") or "if_required"
     tz = (form.get("time_zone") or "UTC").strip()
+    spread, keep = _spread_form(form)
     if not name or not 0 <= weekday <= 6 or not 30 <= window <= 24 * 60 \
             or policy not in maintenance.REBOOT_POLICIES:
         return _error(400, "invalid", "name, a weekday, a window of 30 min to 24 h and a reboot policy are needed")
@@ -1543,6 +1553,7 @@ async def maintenance_plan_create():
         plan = HostMaintenancePlan(name=name, group_id=form["group_id"], weekday=weekday,
                                    at_time=f"{cadence.parse_at(at_time):%H:%M}", time_zone=tz,
                                    window_minutes=window, reboot_policy=policy, enabled=True,
+                                   spread=spread, keep_in_place=keep,
                                    created_by=session.get("user_id"),
                                    # the first slot is the next one, not one already past
                                    last_slot_at=datetime.now(timezone.utc))
@@ -1553,6 +1564,13 @@ async def maintenance_plan_create():
     return redirect(url_for("host_agent.maintenance_page"))
 
 
+def _spread_form(form) -> tuple[bool, str]:
+    """(spread, keep_in_place) from a plan form: the box, and the services
+    named to stay where they are, tidied to one comma-separated list."""
+    from .rebalance import keep_list
+    return form.get("spread") == "on", ", ".join(keep_list(form.get("keep_in_place")))[:2000]
+
+
 async def _plan_action(plan_id: str, act):
     from .models import HostMaintenancePlan
     async with current_app.db_sessionmaker() as db:
@@ -1578,6 +1596,22 @@ async def maintenance_plan_toggle(plan_id: str):
     return resp
 
 
+@host_agent_bp.post("/maintenance/plans/<plan_id>/spread")
+@require_role(UserRole.admin)
+async def maintenance_plan_spread(plan_id: str):
+    """Whether the plan's runs end by spreading swarm services back out, and
+    which services stay where they are."""
+    form = await request.form
+    spread, keep = _spread_form(form)
+
+    async def act(db, plan):
+        plan.spread, plan.keep_in_place = spread, keep
+    resp = await _plan_action(plan_id, act)
+    await audit_request("host_agent.maintenance_plan_spread", "maintenance_plan", plan_id,
+                        {"spread": spread, "keep_in_place": keep})
+    return resp
+
+
 @host_agent_bp.post("/maintenance/plans/<plan_id>/delete")
 @require_role(UserRole.admin)
 async def maintenance_plan_delete(plan_id: str):

plugins/host_agent/templates/maintenance.html

@@ -10,7 +10,7 @@
 <div style="display:flex;flex-direction:column;gap:0.15rem;font-size:0.82rem;">
   {% for r in rows %}
   <div style="display:flex;gap:0.6rem;align-items:baseline;" {% if r.detail %}title="{{ r.detail }}"{% endif %}>
-    <a href="/hosts/{{ r.host_id }}" style="min-width:10rem;">{{ r.name }}</a>
+    {% if r.host_id %}<a href="/hosts/{{ r.host_id }}" style="min-width:10rem;">{{ r.name }}</a>{% else %}<span style="min-width:10rem;">{{ r.name }}</span>{% endif %}
     <span class="{{ 'tone-warn' if r.bad else '' }}" style="{{ '' if r.bad or r.busy else 'color:var(--text-muted);' }}">{{ r.text }}</span>
   </div>
   {% endfor %}
@@ -24,7 +24,9 @@
   a reboot, or before the update when it upgrades Docker, given 5 minutes for its tasks to move,
   and made active again after. Containers running outside the swarm must be running again
   within 10 minutes. The next host starts once the last is back in service. A failure stops the
-  run, and the host's <em>Maintenance failed</em> figure can drive an alert rule.
+  run, and the host's <em>Maintenance failed</em> figure can drive an alert rule. Once every host
+  is back, a run that drained swarm nodes spreads the services back out: only those whose image
+  the emptier node already holds, never one the plan keeps in place.
 </p>
 
 {% if live %}
@@ -68,8 +70,26 @@
     Group <a href="/hosts/?group={{ p.group|urlencode }}">{{ p.group }}</a> ({{ p.hosts }} host{{ 's' if p.hosts != 1 }})
     · {{ p.when }}
     · reboot {{ {'if_required': 'only when required', 'always': 'always', 'never': 'never'}[p.plan.reboot_policy] }}
+    · {% if p.plan.spread %}spreads swarm services back out{% if p.plan.keep_in_place %}, keeping {{ p.plan.keep_in_place }} in place{% endif %}{% else %}leaves swarm services where the run put them{% endif %}
     {% if p.next %}· next {{ p.next|when }}{% endif %}
   </div>
+  {% if session.user_role == 'admin' %}
+  <details style="margin-bottom:0.5rem;">
+    <summary style="cursor:pointer;font-size:0.82rem;color:var(--text-muted);">After the run</summary>
+    <form method="post" action="/plugins/host_agent/maintenance/plans/{{ p.plan.id }}/spread"
+          style="display:flex;gap:0.75rem;align-items:flex-end;flex-wrap:wrap;margin:0.4rem 0 0 1rem;">{{ csrf_field() }}
+      <label style="display:flex;gap:0.4rem;align-items:center;font-size:0.84rem;">
+        <input type="checkbox" name="spread" {% if p.plan.spread %}checked{% endif %}> Spread swarm services back out
+      </label>
+      <div class="form-group" style="margin-bottom:0;">
+        <label>Keep in place</label>
+        <input type="text" name="keep_in_place" value="{{ p.plan.keep_in_place }}" maxlength="2000"
+               placeholder="web_proxy, metrics_app" style="width:22rem;" title="Swarm service names, comma-separated">
+      </div>
+      <button type="submit" class="btn btn-sm">Save</button>
+    </form>
+  </details>
+  {% endif %}
   {% for r in p.runs %}
   <details {% if loop.first %}open{% endif %} style="margin-top:0.3rem;">
     <summary style="cursor:pointer;font-size:0.82rem;">
@@ -129,6 +149,14 @@
         <option value="never">Never</option>
       </select>
     </div>
+    <label style="display:flex;gap:0.4rem;align-items:center;font-size:0.84rem;margin-bottom:0.45rem;">
+      <input type="checkbox" name="spread" checked> Spread swarm services back out after
+    </label>
+    <div class="form-group" style="margin-bottom:0;">
+      <label>Keep in place</label>
+      <input type="text" name="keep_in_place" maxlength="2000" placeholder="web_proxy, metrics_app"
+             style="width:14rem;" title="Swarm service names, comma-separated">
+    </div>
     <button type="submit" class="btn">Add plan</button>
   </form>
   <p style="font-size:0.78rem;color:var(--text-dim);margin:0.6rem 0 0;">

plugins/host_agent/timeline.py

@@ -1,6 +1,7 @@
 # plugins/host_agent/timeline.py
 """What the host agent puts on the timeline: each maintenance job once it
-ends, and each stretch a host was away (a reboot, a crash, lost contact, the
+ends, the swarm services a maintenance run spread back out after its last
+host, and each stretch a host was away (a reboot, a crash, lost contact, the
 agent restarting). One entry per job and per downtime row, kept up to date
 as the row changes: a downtime that ends gets its length, a failed job
 whose host came back late reads as back.
@@ -8,6 +9,7 @@ whose host came back late reads as back.
 from __future__ import annotations
 
 import logging
+from datetime import datetime
 
 from sqlalchemy import select
 
@@ -23,6 +25,8 @@ TYPES = (
      "A host's agent updated it, and rebooted it when the update needed one."),
     ("maintenance.failed", "Maintenance failed", "automation", "host_os", True,
      "A maintenance job failed or timed out."),
+    ("maintenance.rebalanced", "Swarm services spread back out", "automation", "service", True,
+     "After a maintenance run, its swarm services were moved back onto the nodes it had drained."),
     ("host.rebooted", "Host rebooted", "observed", "host_os", True,
      "A host went down and came back, announced or not (a crash, a power cut)."),
     ("host.lost_contact", "Agent lost contact", "observed", "network", True,
@@ -35,8 +39,9 @@ TYPES = (
 def register() -> None:
     for t in TYPES:
         catalog.register_type(*t)
-    from .models import HostDowntime, HostMaintenanceJob
+    from .models import HostDowntime, HostMaintenanceJob, HostMaintenanceRun
     catalog.register_ref("maintenance_job", HostMaintenanceJob)
+    catalog.register_ref("maintenance_run", HostMaintenanceRun)
     catalog.register_ref("host_downtime", HostDowntime)
 
 
@@ -62,6 +67,26 @@ async def job_entry(session, job, host_name: str | None = None):
         actor=await username(session, job.created_by) or "Steward")
 
 
+async def run_entry(session, run):
+    """The entry for a run's spreading of swarm services back out, once it ended."""
+    from . import rebalance
+    from .models import HostMaintenancePlan
+    rb = run.rebalance or {}
+    if rb.get("state") not in ("done", "failed", "timed_out"):
+        return None
+    plan = await session.get(HostMaintenancePlan, run.plan_id)
+    summary = rebalance.outcome(run)[1] or "swarm services spread back out"
+    return await record_or_update(
+        session, ("maintenance_run", run.id), type="maintenance.rebalanced",
+        at=datetime.fromisoformat(rb["at"]),
+        until=datetime.fromisoformat(rb["finished_at"]) if rb.get("finished_at") else None,
+        summary=f"{plan.name if plan else 'Maintenance'}: {summary}",
+        hosts=[], subject=None,
+        detail={"state": rb["state"], "moved": rb.get("moved") or [], "left": rb.get("left") or [],
+                "note": rb.get("detail"), "url": "/plugins/host_agent/maintenance"},
+        actor=await username(session, plan.created_by if plan else None) or "Steward")
+
+
 _DOWNTIME_TYPES = {"reboot": "host.rebooted", "unclean": "host.rebooted",
                    "lost_contact": "host.lost_contact", "agent_stop": "host.agent_restarted"}
 

tests/integration/test_mcp_maintenance.py

@@ -110,7 +110,9 @@ def test_maintenance_plans_jobs_and_starting_one_through_mcp(app):
         plan = plans[ids["plan"]]
         assert plan["group"] == f"mm-{tag}" and plan["schedule"].startswith("Sundays at 03:00")
         assert plan["next_slot_at"] and plan["enabled"] is True
+        assert plan["spread_after_run"] is True and plan["keep_in_place"] == []
         run = plan["latest_run"]
+        assert run["spread"] is None            # it failed before its last host
         assert run["state"] == "failed" and run["hosts_done"] == 1
         assert run["hosts"] == [f"mm-failed-{tag}", f"mm-busy-{tag}"]
         assert run["skipped"] == [{"host": f"mm-free-{tag}", "reason": "no agent"}]

tests/plugins/host_agent/test_agent_docker.py

@@ -286,10 +286,12 @@ def test_swarm_services_rolls_up_replicas():
     assert len(out["agent"].pop("tasks")) == 2
     assert out["web"] == {"service_name": "web", "mode": "replicated",
                           "desired": 3, "running": 2, "image": "nginx:1",
+                          "image_digest": "sha256:deadbeef",
                           "placement": [{"node_id": "n1", "running": 2}]}
     # Global mode has no fixed replica count → desired derived from desired tasks.
     assert out["agent"] == {"service_name": "agent", "mode": "global",
                             "desired": 2, "running": 2, "image": "agent:latest",
+                            "image_digest": None,
                             "placement": [{"node_id": "n1", "running": 1},
                                           {"node_id": "n2", "running": 1}]}
 
@@ -476,8 +478,9 @@ def test_collect_disk_usage_computes_reclaimable(monkeypatch):
     df = {
         "LayersSize": 1000,
         "Images": [
-            {"Id": "sha256:a", "RepoTags": ["app:1"], "Size": 300, "Containers": 1},
+            {"Id": "sha256:a", "RepoTags": ["app:1"], "Size": 300, "Containers": 1,
-            {"Id": "sha256:b", "RepoTags": None, "Size": 700, "Containers": 0},
+             "RepoDigests": ["registry.example:5000/app@sha256:aaa", "app@sha256:aaa"]},
+            {"Id": "sha256:b", "RepoTags": None, "Size": 700, "Containers": 0, "RepoDigests": None},
         ],
         "Containers": [{"Id": "c1", "SizeRw": 25}, {"Id": "c2", "SizeRw": 75}],
         "Volumes": [{"Name": "v1", "UsageData": {"Size": 40}},
@@ -495,6 +498,8 @@ def test_collect_disk_usage_computes_reclaimable(monkeypatch):
     assert out["volumes_count"] == 2 and out["volumes_size"] == 40  # negative skipped
     assert out["build_cache_size"] == 15
     assert len(out["images"]) == 2
+    # Every image the host holds, by digest: what a swarm service is checked against.
+    assert out["digests"] == ["app@sha256:aaa", "registry.example:5000/app@sha256:aaa"]
 
 
 def test_collect_disk_usage_silent_skip_when_no_socket():

tests/plugins/host_agent/test_maint_helper.py

@@ -103,6 +103,46 @@ def test_a_reboot_says_so_before_it_goes(maint, tmp_path, monkeypatch):
     assert called == [["systemctl", "reboot"]]
 
 
+def test_a_rebalance_names_services_and_nothing_else_may(maint, tmp_path):
+    req = {"id": "run-1-rebalance", "action": "rebalance", "services": ["web_app", "db.1"]}
+    assert maint.read_request(_write(tmp_path / "r.json", req))["services"] == ["web_app", "db.1"]
+    for bad in ([], ["--detach"], ["web app"], ["web;rm"], "web_app", ["x"] * (maint.SERVICES_MAX + 1)):
+        assert maint.read_request(_write(tmp_path / "r.json", {**req, "services": bad})) is None
+    # Only a rebalance carries services.
+    assert maint.read_request(_write(tmp_path / "r.json",
+                                     {"id": "job-1", "action": "update", "services": ["web_app"]})) is None
+
+
+def test_a_rebalance_force_updates_each_service_and_leaves_one_a_constraint_places(maint, monkeypatch):
+    monkeypatch.setattr(maint.shutil, "which", lambda _n: "/usr/bin/docker")
+    calls = []
+    placement = {"web": "null", "pinned": '{"Constraints":["node.hostname==a"]}', "spread": "{}"}
+
+    def run(cmd, deadline=0, env=None):
+        calls.append(cmd)
+        name = cmd[-1]
+        if cmd[2] == "inspect":
+            return (name in placement, placement.get(name, "no such service"))
+        return (name != "spread", "update paused")
+    monkeypatch.setattr(maint, "run", run)
+    out = maint.do_rebalance(["web", "pinned", "gone", "spread"])
+    assert out["moved"] == ["web"] and out["ok"] is False
+    assert out["left"] == [{"service": "pinned", "reason": "placed by a constraint"},
+                           {"service": "gone", "reason": "not found on this manager"},
+                           {"service": "spread", "reason": "didn't settle: update paused"}]
+    assert ["docker", "service", "update", "--force", "--detach=false", "--quiet", "web"] in calls
+    assert not any(c[2] == "update" and c[-1] in ("pinned", "gone") for c in calls)
+
+
+def test_a_rebalance_starts_no_move_once_out_of_time(maint, monkeypatch):
+    monkeypatch.setattr(maint.shutil, "which", lambda _n: "/usr/bin/docker")
+    monkeypatch.setattr(maint, "run", lambda cmd, deadline=0, env=None: (True, "null"))
+    ticks = iter([0, 0, maint.REBALANCE_SECONDS + 1])
+    out = maint.do_rebalance(["a", "b"], clock=lambda: next(ticks))
+    assert out["moved"] == ["a"] and out["left"] == [{"service": "b", "reason": "out of time"}]
+    assert out["ok"] is True
+
+
 # ── the agent's side ─────────────────────────────────────────────────────────
 
 def test_the_agent_leaves_a_whole_request_the_helper_accepts(maint, tmp_path):
@@ -111,6 +151,13 @@ def test_the_agent_leaves_a_whole_request_the_helper_accepts(maint, tmp_path):
     assert maint.read_request(str(tmp_path / "job-3.json"))["action"] == "drain"
 
 
+def test_the_agent_passes_a_rebalance_its_services(maint, tmp_path):
+    a.take_maintenance([{"id": "run-1-rebalance", "action": "rebalance", "services": ["web", "db"]}],
+                       set(), requests_dir=str(tmp_path))
+    req = maint.read_request(str(tmp_path / "run-1-rebalance.json"))
+    assert req["action"] == "rebalance" and req["services"] == ["web", "db"]
+
+
 def test_the_agent_refuses_to_ask_for_anything_else(tmp_path):
     with pytest.raises(ValueError):
         a.request_maintenance("job-3", "shell", requests_dir=str(tmp_path))

tests/plugins/host_agent/test_rebalance.py

@@ -0,0 +1,151 @@
+"""A maintenance run spreads swarm services back out after its last host:
+what it moves, what it leaves and why, and how the step reaches the
+manager and comes back."""
+from datetime import datetime, timedelta, timezone
+from types import SimpleNamespace as NS
+
+from plugins.host_agent import rebalance as rb
+
+NOW = datetime(2026, 10, 25, 14, 0, tzinfo=timezone.utc)
+NODES = {"n1": "node-1", "n2": "node-2"}
+
+
+def _svc(name, node="n1", running=1, desired=None, mode="replicated", image="auto"):
+    return {"name": name, "mode": mode, "desired": running if desired is None else desired,
+            "running": running,
+            "image": f"registry.example/{name}@sha256:{name}" if image == "auto" else image,
+            "placement": [{"node_id": node, "running": running}] if running else []}
+
+
+def _held(*names):
+    return {f"registry.example/{n}@sha256:{n}" for n in names}
+
+
+def test_services_on_the_fuller_node_move_until_the_nodes_are_even():
+    services = [_svc(n) for n in "abcdef"]
+    moves, left = rb.choose(services, NODES, {"n2": _held(*"abcdef")})
+    assert moves == ["a", "b", "c"] and left == []
+
+
+def test_nothing_moves_when_the_nodes_are_already_even():
+    services = [_svc("a"), _svc("b", "n2"), _svc("c")]
+    assert rb.choose(services, NODES, {"n1": _held("b"), "n2": _held("a", "c")}) == ([], [])
+
+
+def test_a_service_whose_image_the_emptier_node_lacks_stays_and_says_why():
+    services = [_svc("a"), _svc("b"), _svc("c"), _svc("d")]
+    moves, left = rb.choose(services, NODES, {"n2": _held("b")})
+    assert moves == ["b"]
+    assert left == [{"service": "a", "reason": "its image isn't on node-2"},
+                    {"service": "c", "reason": "its image isn't on node-2"},
+                    {"service": "d", "reason": "its image isn't on node-2"}]
+
+
+def test_an_unknown_image_list_moves_nothing_there():
+    moves, left = rb.choose([_svc("a"), _svc("b")], NODES, {})
+    assert moves == []
+    assert left[0]["reason"] == "no image list from node-2 (its agent needs 1.26.0)"
+
+
+def test_kept_unpinned_short_and_global_services_stay():
+    services = [_svc("keep"), _svc("tag", image=None), _svc("short", running=1, desired=2),
+                _svc("agent", mode="global"), _svc("move")]
+    moves, left = rb.choose(services, NODES, {"n2": _held("keep", "tag", "short", "agent", "move")},
+                            keep=["keep"])
+    assert moves == ["move"]
+    reasons = {x["service"]: x["reason"] for x in left}
+    # Even after one move (4 on n1 against 1): what else could have moved, and why it didn't.
+    assert reasons == {"keep": "kept in place by the plan", "short": "below its replicas",
+                       "tag": "its image isn't pinned to a digest"}
+
+
+def test_a_service_with_all_its_replicas_on_one_node_goes_first():
+    services = [_svc("a"), _svc("b"), _svc("c"), _svc("workers", running=2)]
+    moves, _ = rb.choose(services, NODES, {"n2": _held("a", "b", "c", "workers")})
+    assert moves[0] == "workers" and len(moves) == 2      # 5 on n1: the workers' one, then a
+
+
+def test_a_service_already_spread_or_on_the_emptier_node_is_not_moved():
+    split = {"name": "split", "mode": "replicated", "desired": 2, "running": 2,
+             "image": "registry.example/split@sha256:split",
+             "placement": [{"node_id": "n1", "running": 1}, {"node_id": "n2", "running": 1}]}
+    services = [split, _svc("a"), _svc("b"), _svc("d"), _svc("c", "n2")]   # 4 on n1, 2 on n2
+    moves, _ = rb.choose(services, NODES, {"n2": _held("split", "a", "b", "d")})
+    assert "split" not in moves and moves == ["a"]
+
+
+def test_one_node_has_nothing_to_spread_to():
+    assert rb.choose([_svc("a"), _svc("b")], {"n1": "node-1"}, {}) == ([], [])
+
+
+def test_a_pinned_image_reads_the_way_docker_reports_it():
+    assert rb.pinned("traefik:v3.1", "sha256:abc") == "traefik@sha256:abc"
+    assert rb.pinned("docker.io/library/redis:7", "sha256:abc") == "redis@sha256:abc"
+    assert rb.pinned("registry.example:5000/team/app:dev", "sha256:abc") == \
+        "registry.example:5000/team/app@sha256:abc"
+    assert rb.pinned("app:dev", None) is None
+    assert rb.held(["traefik@sha256:abc", "docker.io/library/redis@sha256:def", "junk"]) == \
+        {"traefik@sha256:abc", "redis@sha256:def"}
+
+
+def test_the_plan_names_the_services_it_keeps_once_each():
+    assert rb.keep_list(" web_proxy, metrics_app,\nweb_proxy ,, ") == ["web_proxy", "metrics_app"]
+    assert rb.keep_list(None) == []
+
+
+def test_only_a_new_enough_agent_rebalances():
+    assert rb.can_rebalance("1.26.0") and rb.can_rebalance("2.0.0")
+    assert not rb.can_rebalance("1.25.3") and not rb.can_rebalance(None)
+
+
+def _run(**rebalance):
+    return NS(id="run-1", rebalance=rebalance or None)
+
+
+def test_the_step_is_handed_until_the_manager_says_it_started():
+    run = _run(state="handed", step_id="run-1-rebalance", planned=["a", "b"], seen=False,
+               at=NOW.isoformat(), left=[])
+    assert rb.handoff(run) == {"id": "run-1-rebalance", "action": "rebalance", "services": ["a", "b"]}
+    assert rb.apply_result(run, {"id": "run-1-rebalance", "state": "running"}, NOW)
+    assert rb.handoff(run) is None
+    assert not rb.apply_result(run, {"id": "other", "state": "done"}, NOW)
+
+
+def test_the_result_says_what_moved_and_the_run_ends_on_it():
+    run = _run(state="handed", step_id="run-1-rebalance", planned=["a", "b"], seen=True,
+               at=NOW.isoformat(), left=[{"service": "c", "reason": "its image isn't on node-2"}])
+    assert rb.apply_result(run, {"id": "run-1-rebalance", "state": "done", "ok": True,
+                                 "moved": ["a", "b"], "left": [], "detail": "moved 2 of 2"}, NOW)
+    assert run.rebalance["state"] == "done" and run.rebalance["moved"] == ["a", "b"]
+    assert rb.outcome(run) == ("done", "spread 2 swarm services back out; 1 left where they are")
+    assert rb.label(run) == ("Spread 2 swarm services back out; 1 left where they are", False, False)
+    # A result already applied changes nothing.
+    assert not rb.apply_result(run, {"id": "run-1-rebalance", "state": "done"}, NOW)
+
+
+def test_a_move_that_didnt_settle_fails_the_run():
+    run = _run(state="handed", step_id="run-1-rebalance", planned=["a"], seen=True,
+               at=NOW.isoformat(), left=[])
+    rb.apply_result(run, {"id": "run-1-rebalance", "state": "failed", "ok": False, "moved": [],
+                          "left": [{"service": "a", "reason": "didn't settle: paused"}],
+                          "detail": "moved 0 of 1; didn't settle: a"}, NOW)
+    assert rb.outcome(run) == ("failed", "spreading swarm services back out: moved 0 of 1; didn't settle: a")
+    assert rb.label(run)[1] is True
+
+
+def test_a_manager_that_never_answers_times_out():
+    run = _run(state="handed", step_id="run-1-rebalance", planned=["a"], seen=True,
+               at=NOW.isoformat(), left=[])
+    assert not rb.check_deadline(run, NOW + rb.DEADLINE)
+    assert rb.check_deadline(run, NOW + rb.DEADLINE + timedelta(seconds=1))
+    assert run.rebalance["state"] == "timed_out" and rb.outcome(run)[0] == "failed"
+
+
+def test_a_run_with_nothing_to_move_ends_done_and_says_why():
+    assert rb.outcome(_run()) == ("done", None) and rb.label(_run()) is None
+    nothing = _run(state="nothing", left=[], detail="no swarm manager has the maintenance helper")
+    assert rb.outcome(nothing) == (
+        "done", "swarm services not spread back out: no swarm manager has the maintenance helper")
+    even = _run(state="nothing", left=[])
+    assert rb.outcome(even) == ("done", None)
+    assert rb.label(even) == ("Nothing to spread back out", False, False)