feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ containers on different hosts no longer collide. Requires STEWARD_DATABASE_URL.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
@@ -391,7 +392,11 @@ def test_swarm_topology_persisted(app):
|
||||
swarm = {
|
||||
"services": [{"service_name": "web", "mode": "replicated",
|
||||
"desired": 3, "running": 2, "image": "nginx",
|
||||
"placement": [{"node_id": "n1", "running": 2}]}],
|
||||
"placement": [{"node_id": "n1", "running": 2}],
|
||||
"tasks": [{"task_id": "t1", "slot": 3, "node_id": "n1",
|
||||
"desired_state": "shutdown", "state": "failed",
|
||||
"message": "started", "error": "task: non-zero exit (1)",
|
||||
"exit_code": 1, "at": "2026-10-06T13:19:51Z"}]}],
|
||||
"nodes": [{"node_id": "n1", "hostname": "mgr", "role": "manager",
|
||||
"availability": "active", "status": "ready", "leader": True}],
|
||||
}
|
||||
@@ -408,7 +413,7 @@ def test_swarm_topology_persisted(app):
|
||||
await persist(s, h, [], swarm)
|
||||
hid = h.id
|
||||
svc = (await s.execute(text(
|
||||
"SELECT mode, desired, running, image, placement_json "
|
||||
"SELECT mode, desired, running, image, placement_json, tasks_json "
|
||||
"FROM docker_swarm_services WHERE host_id = :h"), {"h": hid})).first()
|
||||
node = (await s.execute(text(
|
||||
"SELECT role, availability, status, leader "
|
||||
@@ -418,6 +423,8 @@ def test_swarm_topology_persisted(app):
|
||||
svc, node = asyncio.run(_go())
|
||||
assert svc[0] == "replicated" and svc[1] == 3 and svc[2] == 2 and svc[3] == "nginx"
|
||||
assert "n1" in svc[4] # placement JSON carries the node id
|
||||
(task,) = json.loads(svc[5]) # docker_010: the manager's task records survive
|
||||
assert task["error"] == "task: non-zero exit (1)" and task["slot"] == 3
|
||||
assert node[0] == "manager" and node[2] == "ready" and node[3] is True
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user