feat(maintenance): a run spreads swarm services back out after its last host, moving only what the emptier node can start (#6398)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m36s
CI / integration (push) Successful in 1m47s
CI / publish (push) Skipped

Each weekly run drains Docker02 then Docker03, and the swarm never moves a
task back by itself, so every run ended with nearly every service on one
node. On 2026-10-25 the operator had that spread out by hand.

- Agent 1.26.0 reports each swarm service's pinned digest, and every
  repo@sha256 its host holds (from /system/df, every 10 min).
- Once a run's last host is back in service, Steward picks services on the
  fullest node to move to the emptiest until they are within one task. It
  takes services with several replicas on one node first. A service moves only
  when the emptier node already holds its exact image. One the plan keeps in
  place, one below its replicas, and one with no pinned digest stay put, and
  the run says why.
- A swarm manager's helper carries it out through a new action, rebalance,
  with the service names (validated, at most 48). It force-updates each in
  turn with the spec unchanged and no registry lookup, skips any a
  constraint places, and gives each 5 min to settle.
- The run ends done with what moved, or failed if a move didn't settle or
  the manager didn't answer in 60 min. A docker-scoped window covers the
  restarts, and the step goes on the timeline.
- Plans gain "Spread swarm services back out" (on by default) and "Keep in
  place", on the create form and in an "After the run" form on each plan.
  list_maintenance_plans shows both and the latest run's spread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-25 10:42:26 -04:00
co-authored by Claude Opus 5.5
parent d102fc3333
commit 715b58a09f
19 changed files with 923 additions and 31 deletions
+20 -1
View File
@@ -15,7 +15,7 @@ from steward.mcp.tools.overview import finding
from steward.models.hosts import Host
from steward.models.users import UserRole
from . import cadence, maintenance, senders
from . import cadence, maintenance, rebalance, senders
from .models import HostAgentRegistration, HostMaintenanceJob, HostMaintenancePlan, HostMaintenanceRun
# A registration this old that has never checked in is a failed install, not one in progress.
@@ -143,6 +143,12 @@ async def list_maintenance_plans() -> dict:
(`skipped`), and how it ended. `last_slot_at` is the last scheduled slot
that fired, null until one has. get_maintenance_job shows one host's job.
Plans are created and edited on the Maintenance page.
With `spread_after_run`, a run that drained swarm nodes ends by spreading
the swarm's services back out, leaving `keep_in_place` where they are;
`latest_run.spread` says what it moved and what it left and why (state:
handed, done, failed, timed_out, or nothing to move), null when the run
had none.
"""
from steward.models.ansible_inventory import AnsibleGroup
now = now_utc()
@@ -169,6 +175,8 @@ async def list_maintenance_plans() -> dict:
"weekday": cadence.WEEKDAYS[plan.weekday], "at_time": plan.at_time,
"time_zone": plan.time_zone, "window_minutes": plan.window_minutes,
"reboot_policy": plan.reboot_policy, "enabled": plan.enabled,
"spread_after_run": plan.spread,
"keep_in_place": rebalance.keep_list(plan.keep_in_place),
"next_slot_at": iso(nxt), "last_slot_at": iso(cadence.fired_slot(plan)),
"latest_run": None,
}
@@ -181,12 +189,23 @@ async def list_maintenance_plans() -> dict:
"skipped": [{"host": names.get(x["host_id"], x["host_id"]), "reason": x["reason"]}
for x in json.loads(run.skipped or "[]")],
"detail": run.detail,
"spread": _spread_row(run.rebalance),
"started_at": iso(run.started_at), "finished_at": iso(run.finished_at),
}
out.append(row)
return {"plans": out, "total": len(out)}
def _spread_row(rb: dict | None) -> dict | None:
"""A run's rebalance as the tool shows it: no step ids or window ids."""
if not rb:
return None
return {"state": rb.get("state"), "planned": rb.get("planned") or [],
"moved": rb.get("moved") or [], "left": rb.get("left") or [],
"detail": rb.get("detail"), "started_at": rb.get("at"),
"finished_at": rb.get("finished_at")}
async def get_maintenance_job(host: str = "", job: str = "", limit: int = 10) -> dict:
"""A host's maintenance jobs, newest first (its running one first), or one
job by id.