feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
{#- One line on a swarm service's task failures, from Swarm's own task records
|
||||
(agent ≥ 1.8.0): a crash loop or the failures of the last 24h, and how the
|
||||
newest troubled task ended, in Swarm's words. Expects `s` with state,
|
||||
crash_looping, failures_1h, failures_24h and last_error; `style` is
|
||||
optional extra CSS for the wrapper. Renders nothing for a healthy service
|
||||
with no recent failures. -#}
|
||||
{%- macro ago(sec) -%}
|
||||
{%- if sec < 90 -%}{{ sec }}s ago
|
||||
{%- elif sec < 5400 -%}{{ sec // 60 }}m ago
|
||||
{%- elif sec < 172800 -%}{{ sec // 3600 }}h ago
|
||||
{%- else -%}{{ sec // 86400 }}d ago{%- endif -%}
|
||||
{%- endmacro -%}
|
||||
{%- set e = s.last_error -%}
|
||||
{%- if s.crash_looping or s.failures_24h or (e and s.state != 'healthy') %}
|
||||
<div style="display:flex;flex-wrap:wrap;align-items:baseline;gap:0.2rem 0.6rem;font-size:0.78rem;color:var(--text-muted);{{ style or '' }}">
|
||||
{% if s.crash_looping %}
|
||||
<span style="color:var(--red);font-weight:600;">Crash-looping · {{ s.failures_1h }} tasks failed in the last hour</span>
|
||||
{% elif s.failures_24h %}
|
||||
<span style="color:var(--orange);">{{ s.failures_24h }} task{{ 's' if s.failures_24h != 1 }} failed in the last 24h</span>
|
||||
{% endif %}
|
||||
{% if e %}
|
||||
<span title="How Swarm recorded the end of this service's newest failed task">
|
||||
Last task {{ e.state }}:
|
||||
<span style="color:var(--text);font-family:ui-monospace,monospace;overflow-wrap:anywhere;">{{ e.error or e.message or e.state }}</span>
|
||||
{%- if e.exit_code %} · exit {{ e.exit_code }}{% endif %}
|
||||
{%- if e.node %} · {{ e.node }}{% endif %}
|
||||
{%- if e.age_s is not none %} · {{ ago(e.age_s) }}{% endif %}
|
||||
</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{%- endif %}
|
||||
@@ -26,9 +26,10 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Swarm services: collapsed per-service, each replica labelled with its host.
|
||||
Ghost replicas (tasks on nodes with no Steward agent) come from the managers'
|
||||
placement data so the picture is cluster-complete. ──────────────────────── #}
|
||||
{# ── Swarm services: collapsed per-service, each current replica labelled with
|
||||
its host (tasks Swarm replaced are left out). Ghost replicas (tasks on nodes
|
||||
with no Steward agent) come from the managers' placement data so the picture
|
||||
is cluster-complete. A failing service names Swarm's last task error. ───── #}
|
||||
{% if swarm_services %}
|
||||
<div class="card-flush" style="margin-bottom:1.5rem;">
|
||||
<div style="display:flex;align-items:baseline;gap:0.6rem;padding:0.5rem 0.75rem;border-bottom:1px solid var(--border);">
|
||||
@@ -46,6 +47,7 @@
|
||||
{{ s.running }}/{{ s.desired }} running
|
||||
</span>
|
||||
</div>
|
||||
{% with style="padding-left:1.1rem;margin-bottom:0.35rem;" %}{% include "docker/_swarm_failures.html" %}{% endwith %}
|
||||
<div style="display:grid;grid-template-columns:repeat(auto-fill,minmax(210px,1fr));gap:0.4rem;padding-left:1.1rem;">
|
||||
{% for r in s.replicas %}
|
||||
{% if r.ghost %}
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
<table class="table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th>
|
||||
<th>Service</th><th>Mode</th><th>Replicas</th><th>Image</th><th>Placement</th><th>Task failures</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@@ -39,9 +39,14 @@
|
||||
<td style="font-size:0.78rem;color:var(--text-muted);">
|
||||
{% for p in s.placement %}{{ p.node }} ({{ p.running }}){% if not loop.last %}, {% endif %}{% else %}—{% endfor %}
|
||||
</td>
|
||||
<td style="max-width:360px;">
|
||||
{% include "docker/_swarm_failures.html" %}
|
||||
{%- if not s.tasks_known %}<span style="font-size:0.78rem;color:var(--text-dim);" title="The reporting manager's agent predates task reporting (1.8.0); update it to see task failures.">—</span>
|
||||
{%- elif not (s.crash_looping or s.failures_24h or (s.last_error and s.state != 'healthy')) %}<span style="font-size:0.78rem;color:var(--text-dim);">none in 24h</span>{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% else %}
|
||||
<tr><td colspan="5" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
|
||||
<tr><td colspan="6" style="color:var(--text-muted);font-size:0.85rem;text-align:center;padding:1.5rem;">No services in this swarm.</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:var(--green);">{{ running_count }}</span>
|
||||
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">running</span>
|
||||
</div>
|
||||
<div title="Distinct containers with a die or OOM event in the last 24h">
|
||||
<div title="Containers that died or were OOM-killed in the last 24h, plus swarm tasks that failed (a task Swarm replaced is not a failure)">
|
||||
<span style="font-size:1.4rem;font-weight:700;line-height:1;color:{% if failed_24h %}var(--red){% else %}var(--text-muted){% endif %};">{{ failed_24h }}</span>
|
||||
<span style="font-size:0.78rem;color:var(--text-muted);margin-left:0.25rem;">failed (24h)</span>
|
||||
</div>
|
||||
@@ -24,6 +24,7 @@
|
||||
<div style="display:flex;align-items:center;gap:0.4rem;font-size:0.82rem;">
|
||||
<span class="dot {% if s.state == 'healthy' %}dot-up{% elif s.state == 'degraded' %}dot-warn{% else %}dot-down{% endif %}"></span>
|
||||
<span style="font-weight:500;min-width:0;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;">{{ s.name }}</span>
|
||||
{% if s.crash_looping %}<span style="color:var(--red);font-size:0.7rem;font-weight:600;flex-shrink:0;" title="{{ s.failures_1h }} tasks failed in the last hour{% if s.last_error %}: {{ s.last_error.error or s.last_error.message or s.last_error.state }}{% endif %}">crash-looping</span>{% endif %}
|
||||
<span style="color:var(--text-muted);font-family:ui-monospace,monospace;font-size:0.74rem;margin-left:auto;flex-shrink:0;">{{ s.running }}/{{ s.desired }}</span>
|
||||
</div>
|
||||
<div style="display:flex;flex-wrap:wrap;gap:0.25rem;margin:0.2rem 0 0 1rem;">
|
||||
|
||||
Reference in New Issue
Block a user