feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s

Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 09:46:07 -04:00
co-authored by Claude Opus 5.5
parent 3474cdff46
commit bac2859455
17 changed files with 680 additions and 79 deletions
@@ -226,6 +226,8 @@ def test_swarm_services_rolls_up_replicas():
"Status": {"State": "running"}},
]
out = {s["service_name"]: s for s in a._swarm_services(services, tasks)}
assert [t["state"] for t in out["web"].pop("tasks")] == ["running", "running", "failed"]
assert len(out["agent"].pop("tasks")) == 2
assert out["web"] == {"service_name": "web", "mode": "replicated",
"desired": 3, "running": 2, "image": "nginx:1",
"placement": [{"node_id": "n1", "running": 2}]}
@@ -236,6 +238,33 @@ def test_swarm_services_rolls_up_replicas():
{"node_id": "n2", "running": 1}]}
def test_swarm_task_keeps_why_it_stopped():
t = a._swarm_task({
"ID": "t1", "Slot": 1, "NodeID": "n2", "DesiredState": "Shutdown",
"Status": {"State": "failed", "Timestamp": "2026-10-06T13:19:51.5Z",
"Message": "started", "Err": "task: non-zero exit (1)" + "x" * 400,
"ContainerStatus": {"ContainerID": "c1", "ExitCode": 1}}})
assert t["task_id"] == "t1" and t["slot"] == 1 and t["node_id"] == "n2"
assert t["desired_state"] == "shutdown" and t["state"] == "failed" and t["exit_code"] == 1
assert t["at"] == "2026-10-06T13:19:51.5Z" and t["message"] == "started"
assert t["error"].startswith("task: non-zero exit (1)") and len(t["error"]) == 300
# A task that never got a container has no exit code.
bare = a._swarm_task({"ID": "t2", "DesiredState": "running",
"Status": {"State": "rejected", "Err": "No such image: x"}})
assert bare["exit_code"] is None and bare["slot"] is None and bare["error"] == "No such image: x"
def test_swarm_services_keeps_the_newest_tasks():
services = [{"ID": "s", "Spec": {"Name": "db", "Mode": {"Replicated": {"Replicas": 1}}}}]
tasks = [{"ID": f"t{i}", "ServiceID": "s", "Slot": 1, "DesiredState": "shutdown",
"Status": {"State": "failed", "Timestamp": f"2026-10-06T13:{i:02d}:00Z"}}
for i in range(a.SWARM_TASKS_PER_SERVICE + 5)]
(svc,) = a._swarm_services(services, tasks)
ids = [t["task_id"] for t in svc["tasks"]]
newest = a.SWARM_TASKS_PER_SERVICE + 4
assert len(ids) == a.SWARM_TASKS_PER_SERVICE and ids[0] == f"t{newest}"
def test_swarm_nodes_normalises_fields():
nodes = [
{"ID": "n1", "Description": {"Hostname": "mgr"},