feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -226,6 +226,8 @@ def test_swarm_services_rolls_up_replicas():
|
||||
"Status": {"State": "running"}},
|
||||
]
|
||||
out = {s["service_name"]: s for s in a._swarm_services(services, tasks)}
|
||||
assert [t["state"] for t in out["web"].pop("tasks")] == ["running", "running", "failed"]
|
||||
assert len(out["agent"].pop("tasks")) == 2
|
||||
assert out["web"] == {"service_name": "web", "mode": "replicated",
|
||||
"desired": 3, "running": 2, "image": "nginx:1",
|
||||
"placement": [{"node_id": "n1", "running": 2}]}
|
||||
@@ -236,6 +238,33 @@ def test_swarm_services_rolls_up_replicas():
|
||||
{"node_id": "n2", "running": 1}]}
|
||||
|
||||
|
||||
def test_swarm_task_keeps_why_it_stopped():
|
||||
t = a._swarm_task({
|
||||
"ID": "t1", "Slot": 1, "NodeID": "n2", "DesiredState": "Shutdown",
|
||||
"Status": {"State": "failed", "Timestamp": "2026-10-06T13:19:51.5Z",
|
||||
"Message": "started", "Err": "task: non-zero exit (1)" + "x" * 400,
|
||||
"ContainerStatus": {"ContainerID": "c1", "ExitCode": 1}}})
|
||||
assert t["task_id"] == "t1" and t["slot"] == 1 and t["node_id"] == "n2"
|
||||
assert t["desired_state"] == "shutdown" and t["state"] == "failed" and t["exit_code"] == 1
|
||||
assert t["at"] == "2026-10-06T13:19:51.5Z" and t["message"] == "started"
|
||||
assert t["error"].startswith("task: non-zero exit (1)") and len(t["error"]) == 300
|
||||
# A task that never got a container has no exit code.
|
||||
bare = a._swarm_task({"ID": "t2", "DesiredState": "running",
|
||||
"Status": {"State": "rejected", "Err": "No such image: x"}})
|
||||
assert bare["exit_code"] is None and bare["slot"] is None and bare["error"] == "No such image: x"
|
||||
|
||||
|
||||
def test_swarm_services_keeps_the_newest_tasks():
|
||||
services = [{"ID": "s", "Spec": {"Name": "db", "Mode": {"Replicated": {"Replicas": 1}}}}]
|
||||
tasks = [{"ID": f"t{i}", "ServiceID": "s", "Slot": 1, "DesiredState": "shutdown",
|
||||
"Status": {"State": "failed", "Timestamp": f"2026-10-06T13:{i:02d}:00Z"}}
|
||||
for i in range(a.SWARM_TASKS_PER_SERVICE + 5)]
|
||||
(svc,) = a._swarm_services(services, tasks)
|
||||
ids = [t["task_id"] for t in svc["tasks"]]
|
||||
newest = a.SWARM_TASKS_PER_SERVICE + 4
|
||||
assert len(ids) == a.SWARM_TASKS_PER_SERVICE and ids[0] == f"t{newest}"
|
||||
|
||||
|
||||
def test_swarm_nodes_normalises_fields():
|
||||
nodes = [
|
||||
{"ID": "n1", "Description": {"Hostname": "mgr"},
|
||||
|
||||
Reference in New Issue
Block a user