feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,37 @@ from steward.models.hosts import Host
|
||||
from .models import DockerContainer, DockerEvent, DockerLog, DockerMetric, DockerMetricHourly
|
||||
|
||||
|
||||
CRASH_LOOP_DEATHS = 3 # default; Settings → Thresholds sets it
|
||||
|
||||
|
||||
def crash_loop_deaths(config) -> int:
|
||||
"""Deaths within an hour that make a container or a swarm service a crash
|
||||
loop, from Settings → Thresholds (app.config["THRESHOLDS"])."""
|
||||
th = (config.get("THRESHOLDS") or {}).get("crash_loop") or {}
|
||||
return int(th.get("warn") or CRASH_LOOP_DEATHS)
|
||||
|
||||
|
||||
def failed_24h(dead_names, swarm_services) -> int:
|
||||
"""Failures in the last 24h for the dashboard: distinct plain containers
|
||||
that died or were OOM-killed, plus each swarm service's failed tasks.
|
||||
|
||||
A swarm task's container dies whenever Swarm replaces it (a rolling update
|
||||
kills it with 137), so its die event is not a failure. Services with
|
||||
Swarm's task records count their failed tasks instead; a service without
|
||||
them (an agent older than 1.8.0) falls back to its tasks' die events.
|
||||
"""
|
||||
by_prefix = {s["name"] + ".": s for s in swarm_services}
|
||||
plain, fallback = 0, set()
|
||||
for name in dead_names:
|
||||
svc = next((s for p, s in by_prefix.items() if name.startswith(p)), None)
|
||||
if svc is None:
|
||||
plain += 1
|
||||
elif not svc["tasks_known"]:
|
||||
fallback.add(name)
|
||||
tasks = sum(s["failures_24h"] or 0 for s in swarm_services if s["tasks_known"])
|
||||
return plain + len(fallback) + tasks
|
||||
|
||||
|
||||
def bucket(values: list[float], target: int = 40) -> list[float]:
|
||||
"""Bucket-average a series down to ~target points (DB-agnostic, in Python).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user