feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s

Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 09:46:07 -04:00
co-authored by Claude Opus 5.5
parent 3474cdff46
commit bac2859455
17 changed files with 680 additions and 79 deletions
@@ -0,0 +1,31 @@
{#- One line on a swarm service's task failures, from Swarm's own task records
(agent ≥ 1.8.0): a crash loop or the failures of the last 24h, and how the
newest troubled task ended, in Swarm's words. Expects `s` with state,
crash_looping, failures_1h, failures_24h and last_error; `style` is
optional extra CSS for the wrapper. Renders nothing for a healthy service
with no recent failures. -#}
{%- macro ago(sec) -%}
{%- if sec < 90 -%}{{ sec }}s ago
{%- elif sec < 5400 -%}{{ sec // 60 }}m ago
{%- elif sec < 172800 -%}{{ sec // 3600 }}h ago
{%- else -%}{{ sec // 86400 }}d ago{%- endif -%}
{%- endmacro -%}
{%- set e = s.last_error -%}
{%- if s.crash_looping or s.failures_24h or (e and s.state != 'healthy') %}
<div style="display:flex;flex-wrap:wrap;align-items:baseline;gap:0.2rem 0.6rem;font-size:0.78rem;color:var(--text-muted);{{ style or '' }}">
{% if s.crash_looping %}
<span style="color:var(--red);font-weight:600;">Crash-looping · {{ s.failures_1h }} tasks failed in the last hour</span>
{% elif s.failures_24h %}
<span style="color:var(--orange);">{{ s.failures_24h }} task{{ 's' if s.failures_24h != 1 }} failed in the last 24h</span>
{% endif %}
{% if e %}
<span title="How Swarm recorded the end of this service's newest failed task">
Last task {{ e.state }}:
<span style="color:var(--text);font-family:ui-monospace,monospace;overflow-wrap:anywhere;">{{ e.error or e.message or e.state }}</span>
{%- if e.exit_code %} · exit {{ e.exit_code }}{% endif %}
{%- if e.node %} · {{ e.node }}{% endif %}
{%- if e.age_s is not none %} · {{ ago(e.age_s) }}{% endif %}
</span>
{% endif %}
</div>
{%- endif %}