feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s

Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 09:46:07 -04:00
co-authored by Claude Opus 5.5
parent 3474cdff46
commit bac2859455
17 changed files with 680 additions and 79 deletions
+36 -2
View File
@@ -20,7 +20,7 @@ from collections import deque
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
AGENT_VERSION = "1.7.0"
AGENT_VERSION = "1.8.0"
# Default path to the local Docker Engine socket. Overridable via the
# `docker_socket` config key; collection is silently skipped if it's absent or
@@ -874,14 +874,46 @@ def _swarm_is_manager(info: dict) -> bool:
return bool((info or {}).get("Swarm", {}).get("ControlAvailable"))
# Newest tasks kept per service. Swarm retains about 5 per slot by default, so
# this covers a replicated service's recent history and a crash loop's deaths
# without shipping a global service's whole fleet every sample.
SWARM_TASKS_PER_SERVICE = 10
_TASK_TEXT_CHARS = 300
def _swarm_task(t: dict) -> dict:
"""One task as Steward stores it: where it ran, what Swarm wants of it,
what it is doing, and why it stopped (Err/Message, exit code)."""
status = t.get("Status") or {}
cstatus = status.get("ContainerStatus") or {}
exit_code = cstatus.get("ExitCode")
return {
"task_id": t.get("ID", "") or "",
"slot": t.get("Slot"),
"node_id": t.get("NodeID", "") or "",
"desired_state": (t.get("DesiredState") or "").lower(),
"state": (status.get("State") or "").lower(),
"message": (status.get("Message") or "")[:_TASK_TEXT_CHARS],
"error": (status.get("Err") or "")[:_TASK_TEXT_CHARS],
"exit_code": exit_code if isinstance(exit_code, int) else None,
"at": status.get("Timestamp") or t.get("UpdatedAt") or "",
}
def _swarm_services(services: list, tasks: list) -> list:
"""Roll desired-vs-running replicas per service up from the task list.
Replica health isn't on the service object — it's derived by counting tasks:
`running` = tasks currently in state running; `desired` = the service's
configured replica count (replicated) or the count of tasks the scheduler
wants running (global, which has no fixed number).
wants running (global, which has no fixed number). Each service also
carries its newest tasks (`tasks`), so Steward can tell a task Swarm
replaced from one that crashed, and say why the last one failed.
"""
tasks_by_svc: dict = {}
for t in (tasks or []):
if t.get("ServiceID"):
tasks_by_svc.setdefault(t["ServiceID"], []).append(_swarm_task(t))
running_by_svc: dict = {}
desired_by_svc: dict = {} # only used for global-mode services
placement: dict = {} # svc_id -> {node_id: running_count}
@@ -921,6 +953,8 @@ def _swarm_services(services: list, tasks: list) -> list:
# sees every task, even ones whose containers live on other hosts).
"placement": [{"node_id": n, "running": cnt}
for n, cnt in sorted(placement.get(sid, {}).items())],
"tasks": sorted(tasks_by_svc.get(sid, []), key=lambda t: t["at"],
reverse=True)[:SWARM_TASKS_PER_SERVICE],
})
return out