On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six seconds in, before the drain that came only with the reboot, and every container on the node stopped. Docker's own packages restart the daemon from their install scripts; needrestart's list-only mode doesn't stop that. - The helper gains a `check` action (agent 1.24.0): a dry run of the update for each package manager, whether Docker is among it, and the running containers outside the swarm with their restart policies. - A swarm node is drained before an update that upgrades Docker, else only before a reboot, and at most once a job. - After a drain the job settles 5 minutes before the update or reboot. - Once the work is over, the containers that ran outside the swarm must be seen running again within 10 minutes, or the job fails naming them. - An older agent gets no check and its job runs as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
146 lines
7.3 KiB
HTML
146 lines
7.3 KiB
HTML
{% extends "base.html" %}
|
|
{% from "_macros.html" import crumbs, empty_state, page_header %}
|
|
{% block title %}Maintenance — Steward{% endblock %}
|
|
{% block breadcrumb %}{{ crumbs([("Hosts", "/hosts/"), ("Maintenance", "")]) }}{% endblock %}
|
|
{% block content %}
|
|
{# A group's updates and reboots on a weekly cadence, one host at a time
|
|
(#5768). What needs attention first: a run going now, a run that failed;
|
|
then the plans with their recent runs. #}
|
|
{% macro run_hosts(rows) -%}
|
|
<div style="display:flex;flex-direction:column;gap:0.15rem;font-size:0.82rem;">
|
|
{% for r in rows %}
|
|
<div style="display:flex;gap:0.6rem;align-items:baseline;" {% if r.detail %}title="{{ r.detail }}"{% endif %}>
|
|
<a href="/hosts/{{ r.host_id }}" style="min-width:10rem;">{{ r.name }}</a>
|
|
<span class="{{ 'tone-warn' if r.bad else '' }}" style="{{ '' if r.bad or r.busy else 'color:var(--text-muted);' }}">{{ r.text }}</span>
|
|
</div>
|
|
{% endfor %}
|
|
</div>
|
|
{%- endmacro %}
|
|
|
|
{{ page_header("Maintenance") }}
|
|
<p style="color:var(--text-muted);font-size:0.84rem;margin:-0.4rem 0 1.25rem;">
|
|
Each plan updates an inventory group's hosts once a week, one host at a time. A host reboots
|
|
only when its update needs it, unless the plan says otherwise. A swarm node is drained before
|
|
a reboot, or before the update when it upgrades Docker, given 5 minutes for its tasks to move,
|
|
and made active again after. Containers running outside the swarm must be running again
|
|
within 10 minutes. The next host starts once the last is back in service. A failure stops the
|
|
run, and the host's <em>Maintenance failed</em> figure can drive an alert rule.
|
|
</p>
|
|
|
|
{% if live %}
|
|
{% for l in live %}
|
|
<div class="card">
|
|
<h3 style="margin-bottom:0.5rem;">Running: {{ l.plan }} <span style="font-size:0.78rem;color:var(--text-muted);font-weight:normal;">· started {{ l.run.started_at|ago }}{% if l.run.manual %} · run now{% endif %}</span></h3>
|
|
{{ run_hosts(l.rows) }}
|
|
</div>
|
|
{% endfor %}
|
|
{% endif %}
|
|
|
|
{% for f in failed %}
|
|
<div class="card" style="border-color:color-mix(in srgb,var(--yellow) 40%,var(--border));">
|
|
<span class="tone-warn">Failed: {{ f.plan }}</span>
|
|
<span style="font-size:0.82rem;color:var(--text-muted);">· {{ f.run.finished_at|ago }} · {{ f.run.detail or "no detail" }}</span>
|
|
</div>
|
|
{% endfor %}
|
|
|
|
{% if plans %}
|
|
{% for p in plans %}
|
|
<div class="card">
|
|
<div style="display:flex;align-items:baseline;justify-content:space-between;gap:1rem;flex-wrap:wrap;">
|
|
<h3 style="margin-bottom:0.2rem;">{{ p.plan.name }}{% if not p.plan.enabled %} <span style="font-size:0.78rem;color:var(--text-muted);font-weight:normal;">· paused</span>{% endif %}</h3>
|
|
{% if session.user_role == 'admin' %}
|
|
<div style="display:flex;gap:0.4rem;">
|
|
<form method="post" action="/plugins/host_agent/maintenance/plans/{{ p.plan.id }}/run" style="margin:0;"
|
|
data-confirm="Run {{ p.plan.name }} now? Its {{ p.hosts }} host{{ 's' if p.hosts != 1 }} are updated one at a time; the schedule keeps its time.">{{ csrf_field() }}
|
|
<button type="submit" class="btn btn-sm">Run now</button>
|
|
</form>
|
|
<form method="post" action="/plugins/host_agent/maintenance/plans/{{ p.plan.id }}/toggle" style="margin:0;">{{ csrf_field() }}
|
|
<button type="submit" class="btn btn-sm btn-ghost">{{ 'Pause' if p.plan.enabled else 'Resume' }}</button>
|
|
</form>
|
|
<form method="post" action="/plugins/host_agent/maintenance/plans/{{ p.plan.id }}/delete" style="margin:0;"
|
|
data-confirm="Delete {{ p.plan.name }} and its run history? Hosts' own job history stays.">{{ csrf_field() }}
|
|
<button type="submit" class="btn btn-sm btn-danger">Delete</button>
|
|
</form>
|
|
</div>
|
|
{% endif %}
|
|
</div>
|
|
<div style="font-size:0.82rem;color:var(--text-muted);margin-bottom:0.6rem;">
|
|
Group <a href="/hosts/?group={{ p.group|urlencode }}">{{ p.group }}</a> ({{ p.hosts }} host{{ 's' if p.hosts != 1 }})
|
|
· {{ p.when }}
|
|
· reboot {{ {'if_required': 'only when required', 'always': 'always', 'never': 'never'}[p.plan.reboot_policy] }}
|
|
{% if p.next %}· next {{ p.next|when }}{% endif %}
|
|
</div>
|
|
{% for r in p.runs %}
|
|
<details {% if loop.first %}open{% endif %} style="margin-top:0.3rem;">
|
|
<summary style="cursor:pointer;font-size:0.82rem;">
|
|
<span class="{{ 'tone-warn' if r.run.state == 'failed' else '' }}">{{ {'running': 'Running', 'done': 'Done', 'failed': 'Failed', 'stopped': 'Stopped'}[r.run.state] }}</span>
|
|
<span style="color:var(--text-muted);">· {{ r.run.started_at|when }}{% if r.run.manual %} · run now{% endif %}{% if r.run.detail %} · {{ r.run.detail|truncate(100) }}{% endif %}</span>
|
|
</summary>
|
|
<div style="margin:0.3rem 0 0.4rem 1rem;">{{ run_hosts(r.rows) }}</div>
|
|
</details>
|
|
{% else %}
|
|
<p style="font-size:0.82rem;color:var(--text-dim);margin:0;">No runs yet.</p>
|
|
{% endfor %}
|
|
</div>
|
|
{% endfor %}
|
|
{% else %}
|
|
<div class="card">{{ empty_state("No maintenance plans yet." ~ (" Add one below." if session.user_role == 'admin' else "")) }}</div>
|
|
{% endif %}
|
|
|
|
{% if session.user_role == 'admin' %}
|
|
<div class="card">
|
|
<h3 style="margin-bottom:0.6rem;">Add a plan</h3>
|
|
{% if groups %}
|
|
<form method="post" action="/plugins/host_agent/maintenance/plans"
|
|
style="display:flex;gap:0.75rem;align-items:flex-end;flex-wrap:wrap;">{{ csrf_field() }}
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Name</label>
|
|
<input type="text" name="name" required maxlength="128" placeholder="Docker hosts, weekly" style="width:13rem;">
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Group</label>
|
|
<select name="group_id" required>
|
|
{% for g in groups %}<option value="{{ g.id }}">{{ g.name }}</option>{% endfor %}
|
|
</select>
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Day</label>
|
|
<select name="weekday">
|
|
{% for d in weekdays %}<option value="{{ loop.index0 }}" {% if loop.index0 == 6 %}selected{% endif %}>{{ d }}</option>{% endfor %}
|
|
</select>
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Time</label>
|
|
<input type="time" name="at_time" value="03:00" required style="width:7rem;">
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Time zone</label>
|
|
<input type="text" name="time_zone" value="{{ default_zone }}" required style="width:11rem;">
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Window (min)</label>
|
|
<input type="number" name="window_minutes" value="240" min="30" max="1440" style="width:6rem;">
|
|
</div>
|
|
<div class="form-group" style="margin-bottom:0;">
|
|
<label>Reboot</label>
|
|
<select name="reboot_policy">
|
|
<option value="if_required" selected>Only when required</option>
|
|
<option value="always">Always</option>
|
|
<option value="never">Never</option>
|
|
</select>
|
|
</div>
|
|
<button type="submit" class="btn">Add plan</button>
|
|
</form>
|
|
<p style="font-size:0.78rem;color:var(--text-dim);margin:0.6rem 0 0;">
|
|
Hosts whose agent has no maintenance helper are skipped and listed; run Update agent on them.
|
|
No host is started once the window has passed.
|
|
</p>
|
|
{% else %}
|
|
<p style="font-size:0.85rem;color:var(--text-dim);margin:0;">
|
|
Plans work on inventory groups. Add one under <a href="/ansible/inventory/">Ansible → Inventory</a>.
|
|
</p>
|
|
{% endif %}
|
|
</div>
|
|
{% endif %}
|
|
{% endblock %}
|