feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,7 @@ from collections import deque
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
|
||||
AGENT_VERSION = "1.7.0"
|
||||
AGENT_VERSION = "1.8.0"
|
||||
|
||||
# Default path to the local Docker Engine socket. Overridable via the
|
||||
# `docker_socket` config key; collection is silently skipped if it's absent or
|
||||
@@ -874,14 +874,46 @@ def _swarm_is_manager(info: dict) -> bool:
|
||||
return bool((info or {}).get("Swarm", {}).get("ControlAvailable"))
|
||||
|
||||
|
||||
# Newest tasks kept per service. Swarm retains about 5 per slot by default, so
|
||||
# this covers a replicated service's recent history and a crash loop's deaths
|
||||
# without shipping a global service's whole fleet every sample.
|
||||
SWARM_TASKS_PER_SERVICE = 10
|
||||
_TASK_TEXT_CHARS = 300
|
||||
|
||||
|
||||
def _swarm_task(t: dict) -> dict:
|
||||
"""One task as Steward stores it: where it ran, what Swarm wants of it,
|
||||
what it is doing, and why it stopped (Err/Message, exit code)."""
|
||||
status = t.get("Status") or {}
|
||||
cstatus = status.get("ContainerStatus") or {}
|
||||
exit_code = cstatus.get("ExitCode")
|
||||
return {
|
||||
"task_id": t.get("ID", "") or "",
|
||||
"slot": t.get("Slot"),
|
||||
"node_id": t.get("NodeID", "") or "",
|
||||
"desired_state": (t.get("DesiredState") or "").lower(),
|
||||
"state": (status.get("State") or "").lower(),
|
||||
"message": (status.get("Message") or "")[:_TASK_TEXT_CHARS],
|
||||
"error": (status.get("Err") or "")[:_TASK_TEXT_CHARS],
|
||||
"exit_code": exit_code if isinstance(exit_code, int) else None,
|
||||
"at": status.get("Timestamp") or t.get("UpdatedAt") or "",
|
||||
}
|
||||
|
||||
|
||||
def _swarm_services(services: list, tasks: list) -> list:
|
||||
"""Roll desired-vs-running replicas per service up from the task list.
|
||||
|
||||
Replica health isn't on the service object — it's derived by counting tasks:
|
||||
`running` = tasks currently in state running; `desired` = the service's
|
||||
configured replica count (replicated) or the count of tasks the scheduler
|
||||
wants running (global, which has no fixed number).
|
||||
wants running (global, which has no fixed number). Each service also
|
||||
carries its newest tasks (`tasks`), so Steward can tell a task Swarm
|
||||
replaced from one that crashed, and say why the last one failed.
|
||||
"""
|
||||
tasks_by_svc: dict = {}
|
||||
for t in (tasks or []):
|
||||
if t.get("ServiceID"):
|
||||
tasks_by_svc.setdefault(t["ServiceID"], []).append(_swarm_task(t))
|
||||
running_by_svc: dict = {}
|
||||
desired_by_svc: dict = {} # only used for global-mode services
|
||||
placement: dict = {} # svc_id -> {node_id: running_count}
|
||||
@@ -921,6 +953,8 @@ def _swarm_services(services: list, tasks: list) -> list:
|
||||
# sees every task, even ones whose containers live on other hosts).
|
||||
"placement": [{"node_id": n, "running": cnt}
|
||||
for n, cnt in sorted(placement.get(sid, {}).items())],
|
||||
"tasks": sorted(tasks_by_svc.get(sid, []), key=lambda t: t["at"],
|
||||
reverse=True)[:SWARM_TASKS_PER_SERVICE],
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
Reference in New Issue
Block a user