feat(host_agent): Steward hands a host a maintenance job and follows it through the reboot (#5767)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m28s
CI / unit (push) Failing after 1m31s
CI / publish (push) Skipped

#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.

A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.

The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.

Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.

The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 18:42:12 -04:00
co-authored by Claude Opus 5.5
parent 6a68bd75d9
commit 22b0be2b6c
12 changed files with 954 additions and 8 deletions
+23 -1
View File
@@ -21,8 +21,16 @@
{# What the host has waiting, from the agent's hourly check (#5765): a
needed reboot first, then the updates. #}
{% if reg.updates_checked_at or reg.reboot_required %}
{% if reg.updates_checked_at or reg.reboot_required or maint_job %}
<div style="margin-top:0.8rem;font-size:0.82rem;display:flex;flex-direction:column;gap:0.2rem;">
{# Steward's own maintenance job on this host (#5767): a running one first. #}
{% if maint_job %}
{% set running = maint_job.state not in ('done', 'failed', 'timed_out') %}
<div class="{{ 'tone-warn' if running or maint_job.state != 'done' else '' }}" title="{{ maint_job.detail or '' }}">
Maintenance: {{ maint_summary }}
<span style="color:var(--text-muted);">· {% if running %}since {{ maint_job.state_at|ago }}{% else %}{{ maint_job.finished_at|ago }}{% endif %}{% if maint_job.state in ('failed', 'timed_out') and maint_job.detail %} · {{ maint_job.detail|truncate(90) }}{% endif %}</span>
</div>
{% endif %}
{% if reg.reboot_required %}
<div class="tone-warn" title="{{ reg.reboot_packages or '' }}">
Reboot required{% if reg.reboot_required_since %} since {{ reg.reboot_required_since|ago }}{% endif %}{% if reg.reboot_packages %}
@@ -78,6 +86,20 @@
Link an Ansible target (below) to update the agent from here.
</p>
{% endif %}
{# Update now (#5767): the agent's helper updates the host and reboots it
only if the update needs it, draining its swarm node first. #}
{% if maint_blocker %}
<p style="font-size:0.8rem;margin:0.6rem 0 0;color:var(--text-dim);">Can't update from here: {{ maint_blocker }}.</p>
{% else %}
<form method="post" action="/plugins/host_agent/fleet/{{ host.id }}/maintain"
style="display:flex;gap:0.6rem;align-items:center;flex-wrap:wrap;margin:0.6rem 0 0;"
data-confirm="Update {{ host.name }} now? It reboots only if the update needs it, after its swarm node is drained.">{{ csrf_field() }}
<button type="submit" class="btn btn-sm">Update now</button>
<label style="font-size:0.8rem;color:var(--text-muted);display:flex;gap:0.3rem;align-items:center;">
<input type="checkbox" name="reboot_policy" value="never"> never reboot
</label>
</form>
{% endif %}
{# The root helper that carries out scheduled updates and reboots (#5766). #}
<p style="font-size:0.8rem;margin:0.6rem 0 0;color:var(--text-dim);">
{% if reg.maint_helper %}Maintenance helper installed: Steward can update and reboot this host.