feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s

Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 09:46:07 -04:00
co-authored by Claude Opus 5.5
parent 3474cdff46
commit bac2859455
17 changed files with 680 additions and 79 deletions
@@ -0,0 +1,31 @@
{#- One line on a swarm service's task failures, from Swarm's own task records
(agent ≥ 1.8.0): a crash loop or the failures of the last 24h, and how the
newest troubled task ended, in Swarm's words. Expects `s` with state,
crash_looping, failures_1h, failures_24h and last_error; `style` is
optional extra CSS for the wrapper. Renders nothing for a healthy service
with no recent failures. -#}
{%- macro ago(sec) -%}
{%- if sec < 90 -%}{{ sec }}s ago
{%- elif sec < 5400 -%}{{ sec // 60 }}m ago
{%- elif sec < 172800 -%}{{ sec // 3600 }}h ago
{%- else -%}{{ sec // 86400 }}d ago{%- endif -%}
{%- endmacro -%}
{%- set e = s.last_error -%}
{%- if s.crash_looping or s.failures_24h or (e and s.state != 'healthy') %}
<div style="display:flex;flex-wrap:wrap;align-items:baseline;gap:0.2rem 0.6rem;font-size:0.78rem;color:var(--text-muted);{{ style or '' }}">
{% if s.crash_looping %}
<span style="color:var(--red);font-weight:600;">Crash-looping · {{ s.failures_1h }} tasks failed in the last hour</span>
{% elif s.failures_24h %}
<span style="color:var(--orange);">{{ s.failures_24h }} task{{ 's' if s.failures_24h != 1 }} failed in the last 24h</span>
{% endif %}
{% if e %}
<span title="How Swarm recorded the end of this service's newest failed task">
Last task {{ e.state }}:
<span style="color:var(--text);font-family:ui-monospace,monospace;overflow-wrap:anywhere;">{{ e.error or e.message or e.state }}</span>
{%- if e.exit_code %} · exit {{ e.exit_code }}{% endif %}
{%- if e.node %} · {{ e.node }}{% endif %}
{%- if e.age_s is not none %} · {{ ago(e.age_s) }}{% endif %}
</span>
{% endif %}
</div>
{%- endif %}
+5 -3
View File
@@ -26,9 +26,10 @@
</div>
</div>
{# ── Swarm services: collapsed per-service, each replica labelled with its host.
Ghost replicas (tasks on nodes with no Steward agent) come from the managers'
placement data so the picture is cluster-complete. ──────────────────────── #}
{# ── Swarm services: collapsed per-service, each current replica labelled with
its host (tasks Swarm replaced are left out). Ghost replicas (tasks on nodes
with no Steward agent) come from the managers' placement data so the picture
is cluster-complete. A failing service names Swarm's last task error. ───── #}
{% if swarm_services %}
<div class="card-flush" style="margin-bottom:1.5rem;">
<div style="display:flex;align-items:baseline;gap:0.6rem;padding:0.5rem 0.75rem;border-bottom:1px solid var(--border);">
@@ -46,6 +47,7 @@
{{ s.running }}/{{ s.desired }} running
</span>
</div>
{% with style="padding-left:1.1rem;margin-bottom:0.35rem;" %}{% include "docker/_swarm_failures.html" %}{% endwith %}
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(210px,1fr));gap:0.4rem;padding-left:1.1rem;">
{% for r in s.replicas %}
{% if r.ghost %}
+7 -2
View File
@@ -22,7 +22,7 @@
<table class="table">
<thead>
<tr>
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th>
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th><th>Task failures</th>
</tr>
</thead>
<tbody>
@@ -39,9 +39,14 @@
<td style="font-size:0.78rem;color:var(--text-muted);">
{% for p in s.placement %}{{ p.node }} ({{ p.running }}){% if not loop.last %}, {% endif %}{% else %}—{% endfor %}
</td>
<td style="max-width:360px;">
{% include "docker/_swarm_failures.html" %}
{%- if not s.tasks_known %}<span style="font-size:0.78rem;color:var(--text-dim);" title="The reporting manager's agent predates task reporting (1.8.0); update it to see task failures.">—</span>
{%- elif not (s.crash_looping or s.failures_24h or (s.last_error and s.state != 'healthy')) %}<span style="font-size:0.78rem;color:var(--text-dim);">none in 24h</span>{% endif %}
</td>
</tr>
{% else %}
<tr><td colspan="5" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
<tr><td colspan="6" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
{% endfor %}
</tbody>
</table>
+2 -1
View File
@@ -8,7 +8,7 @@
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:var(--green);">{{ running_count }}</span>
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">running</span>
</div>
<div title="Distinct containers with a die or OOM event in the last 24h">
<div title="Containers that died or were OOM-killed in the last 24h, plus swarm tasks that failed (a task Swarm replaced is not a failure)">
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:{% if failed_24h %}var(--red){% else %}var(--text-muted){% endif %};">{{ failed_24h }}</span>
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">failed (24h)</span>
</div>
@@ -24,6 +24,7 @@
<div style="display:flex;align-items:center;gap:0.4rem;font-size:0.82rem;">
<span class="dot {% if s.state == 'healthy' %}dot-up{% elif s.state == 'degraded' %}dot-warn{% else %}dot-down{% endif %}"></span>
<span style="font-weight:500;min-width:0;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;">{{ s.name }}</span>
{% if s.crash_looping %}<span style="color:var(--red);font-size:0.7rem;font-weight:600;flex-shrink:0;" title="{{ s.failures_1h }} tasks failed in the last hour{% if s.last_error %}: {{ s.last_error.error or s.last_error.message or s.last_error.state }}{% endif %}">crash-looping</span>{% endif %}
<span style="color:var(--text-muted);font-family:ui-monospace,monospace;font-size:0.74rem;margin-left:auto;flex-shrink:0;">{{ s.running }}/{{ s.desired }}</span>
</div>
<div style="display:flex;flex-wrap:wrap;gap:0.25rem;margin:0.2rem 0 0 1rem;">