feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -73,6 +73,40 @@ def test_service_findings_by_state():
|
||||
assert got == {"b": "warning", "c": "critical"}
|
||||
|
||||
|
||||
def test_service_findings_name_the_crash_loop_and_swarms_error():
|
||||
err = {"error": "task: non-zero exit (1)", "message": "started", "state": "failed",
|
||||
"exit_code": 1, "node": "docker03"}
|
||||
svcs = [{"name": "db", "state": "down", "running": 0, "desired": 1, "crash_looping": True,
|
||||
"failures_1h": 4, "failures_24h": 5, "last_error": err},
|
||||
{"name": "app", "state": "down", "running": 0, "desired": 1, "crash_looping": False,
|
||||
"failures_1h": 0, "failures_24h": 0,
|
||||
"last_error": {"error": "No such image: app:dev", "state": "rejected",
|
||||
"exit_code": None, "node": None, "message": None}},
|
||||
{"name": "web", "state": "healthy", "running": 2, "desired": 2, "crash_looping": False,
|
||||
"failures_1h": 0, "failures_24h": 1, "last_error": err},
|
||||
{"name": "ok", "state": "healthy", "running": 1, "desired": 1, "crash_looping": False,
|
||||
"failures_1h": 0, "failures_24h": 0, "last_error": None}]
|
||||
got = {f["subject"]: f for f in t.service_findings(svcs)}
|
||||
assert set(got) == {"db", "app", "web"}
|
||||
db = got["db"]
|
||||
assert db["kind"] == "swarm_service_crash_looping" and db["severity"] == "critical"
|
||||
assert "4 tasks failed in the last hour" in db["summary"]
|
||||
assert "last task error: task: non-zero exit (1), exit 1 on docker03" in db["summary"]
|
||||
# Down without a loop still names why: here Swarm could not even start it.
|
||||
assert got["app"]["kind"] == "swarm_service_down"
|
||||
assert got["app"]["summary"].endswith("last task error: No such image: app:dev")
|
||||
# Recovered, but a task died in the last day: worth knowing, not critical.
|
||||
assert got["web"]["kind"] == "swarm_task_failed" and got["web"]["severity"] == "warning"
|
||||
assert "had 1 failed task in the last 24h" in got["web"]["summary"]
|
||||
|
||||
|
||||
def test_a_replaced_swarm_task_is_marked_and_judged_on_its_service():
|
||||
task = _c(name="app.1.old", status="exited", exit_code=137, service_name="app")
|
||||
row = t.container_row(task, {}, [], NOW, replaced=True)
|
||||
assert row["swarm_replaced"] is True and row["problems"] == []
|
||||
assert t.container_row(_c(), {}, [], NOW)["swarm_replaced"] is None # not a swarm task
|
||||
|
||||
|
||||
def test_plugin_exposes_the_mcp_hooks():
|
||||
reg = MagicMock()
|
||||
docker_plugin.register_mcp_tools(reg)
|
||||
|
||||
Reference in New Issue
Block a user