feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s

Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 09:46:07 -04:00
co-authored by Claude Opus 5.5
parent 3474cdff46
commit bac2859455
17 changed files with 680 additions and 79 deletions
+31
View File
@@ -14,6 +14,37 @@ from steward.models.hosts import Host
from .models import DockerContainer, DockerEvent, DockerLog, DockerMetric, DockerMetricHourly
CRASH_LOOP_DEATHS = 3 # default; Settings → Thresholds sets it
def crash_loop_deaths(config) -> int:
"""Deaths within an hour that make a container or a swarm service a crash
loop, from Settings → Thresholds (app.config["THRESHOLDS"])."""
th = (config.get("THRESHOLDS") or {}).get("crash_loop") or {}
return int(th.get("warn") or CRASH_LOOP_DEATHS)
def failed_24h(dead_names, swarm_services) -> int:
"""Failures in the last 24h for the dashboard: distinct plain containers
that died or were OOM-killed, plus each swarm service's failed tasks.
A swarm task's container dies whenever Swarm replaces it (a rolling update
kills it with 137), so its die event is not a failure. Services with
Swarm's task records count their failed tasks instead; a service without
them (an agent older than 1.8.0) falls back to its tasks' die events.
"""
by_prefix = {s["name"] + ".": s for s in swarm_services}
plain, fallback = 0, set()
for name in dead_names:
svc = next((s for p, s in by_prefix.items() if name.startswith(p)), None)
if svc is None:
plain += 1
elif not svc["tasks_known"]:
fallback.add(name)
tasks = sum(s["failures_24h"] or 0 for s in swarm_services if s["tasks_known"])
return plain + len(fallback) + tasks
def bucket(values: list[float], target: int = 40) -> list[float]:
"""Bucket-average a series down to ~target points (DB-agnostic, in Python).