feat(maintenance): a run spreads swarm services back out after its last host, moving only what the emptier node can start (#6398)
Each weekly run drains Docker02 then Docker03, and the swarm never moves a task back by itself, so every run ended with nearly every service on one node. On 2026-10-25 the operator had that spread out by hand. - Agent 1.26.0 reports each swarm service's pinned digest, and every repo@sha256 its host holds (from /system/df, every 10 min). - Once a run's last host is back in service, Steward picks services on the fullest node to move to the emptiest until they are within one task. It takes services with several replicas on one node first. A service moves only when the emptier node already holds its exact image. One the plan keeps in place, one below its replicas, and one with no pinned digest stay put, and the run says why. - A swarm manager's helper carries it out through a new action, rebalance, with the service names (validated, at most 48). It force-updates each in turn with the spec unchanged and no registry lookup, skips any a constraint places, and gives each 5 min to settle. - The run ends done with what moved, or failed if a move didn't settle or the manager didn't answer in 60 min. A docker-scoped window covers the restarts, and the step goes on the timeline. - Plans gain "Spread swarm services back out" (on by default) and "Keep in place", on the create form and in an "After the run" form on each plan. list_maintenance_plans shows both and the latest run's spread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -286,10 +286,12 @@ def test_swarm_services_rolls_up_replicas():
|
||||
assert len(out["agent"].pop("tasks")) == 2
|
||||
assert out["web"] == {"service_name": "web", "mode": "replicated",
|
||||
"desired": 3, "running": 2, "image": "nginx:1",
|
||||
"image_digest": "sha256:deadbeef",
|
||||
"placement": [{"node_id": "n1", "running": 2}]}
|
||||
# Global mode has no fixed replica count → desired derived from desired tasks.
|
||||
assert out["agent"] == {"service_name": "agent", "mode": "global",
|
||||
"desired": 2, "running": 2, "image": "agent:latest",
|
||||
"image_digest": None,
|
||||
"placement": [{"node_id": "n1", "running": 1},
|
||||
{"node_id": "n2", "running": 1}]}
|
||||
|
||||
@@ -476,8 +478,9 @@ def test_collect_disk_usage_computes_reclaimable(monkeypatch):
|
||||
df = {
|
||||
"LayersSize": 1000,
|
||||
"Images": [
|
||||
{"Id": "sha256:a", "RepoTags": ["app:1"], "Size": 300, "Containers": 1},
|
||||
{"Id": "sha256:b", "RepoTags": None, "Size": 700, "Containers": 0},
|
||||
{"Id": "sha256:a", "RepoTags": ["app:1"], "Size": 300, "Containers": 1,
|
||||
"RepoDigests": ["registry.example:5000/app@sha256:aaa", "app@sha256:aaa"]},
|
||||
{"Id": "sha256:b", "RepoTags": None, "Size": 700, "Containers": 0, "RepoDigests": None},
|
||||
],
|
||||
"Containers": [{"Id": "c1", "SizeRw": 25}, {"Id": "c2", "SizeRw": 75}],
|
||||
"Volumes": [{"Name": "v1", "UsageData": {"Size": 40}},
|
||||
@@ -495,6 +498,8 @@ def test_collect_disk_usage_computes_reclaimable(monkeypatch):
|
||||
assert out["volumes_count"] == 2 and out["volumes_size"] == 40 # negative skipped
|
||||
assert out["build_cache_size"] == 15
|
||||
assert len(out["images"]) == 2
|
||||
# Every image the host holds, by digest: what a swarm service is checked against.
|
||||
assert out["digests"] == ["app@sha256:aaa", "registry.example:5000/app@sha256:aaa"]
|
||||
|
||||
|
||||
def test_collect_disk_usage_silent_skip_when_no_socket():
|
||||
|
||||
Reference in New Issue
Block a user