feat(host_agent): Steward hands a host a maintenance job and follows it through the reboot (#5767)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m28s
CI / unit (push) Failing after 1m31s
CI / publish (push) Skipped

#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.

A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.

The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.

Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.

The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 18:42:12 -04:00
co-authored by Claude Opus 5.5
parent 6a68bd75d9
commit 22b0be2b6c
12 changed files with 954 additions and 8 deletions
+46 -1
View File
@@ -174,7 +174,52 @@ host that has an older agent.
So a stolen agent token can make its host update, reboot or drain, and nothing
else. Steward handing a host a job and following it through the reboot is
#5767; the cadence that does it one host at a time is #5768.
#5767 (below); the cadence that does it one host at a time is #5768.
### A maintenance job, followed through the reboot (1.18.0, #5767)
Steward keeps one `host_maintenance_jobs` row per job and moves it on only from
what the agents post, so a restart of Steward, or a reboot of the host Steward
runs on, loses nothing. `plugins/host_agent/maintenance.py` has the states and
transitions (model-free, unit-tested) and the database side.
```
queued → updating → [draining →] rebooting → [undraining →] done
↘ done (no reboot needed, or the job may never reboot)
any state → failed (a step's result said so) | timed_out (its deadline passed)
```
- **Hand-off.** Steward's answer to an ingest post carries
`"maintenance": [{"id", "action", ["node"]}]`, the step that host's agent
should carry out now. It is repeated on every post until a result for that
step id arrives; the agent leaves each id for the helper once
(`take_maintenance`), and the helper refuses an id it already has a result
for, so a repeat, or a re-hand after the agent restarts, never runs twice.
A step id is `<job id>-<action>`.
- **Results** are the `maintenance` events of #5766, matched to the job by step
id, so a drain reported by a manager reaches the job of the node it drained.
- **The reboot completes** when the host's agent sends `started` with a
`boot_at` after the reboot was handed out (60 s of clock slack). An agent
restart on a host that stayed up has an old `boot_at` and changes nothing.
- **Reboot only when required**: the update's result says whether a reboot is
needed; a job made with "never reboot" stops after the update.
- **Swarm nodes** are drained first and made active after. The node is found
by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
worker is drained by a manager whose agent has the helper. With no such
manager the job is refused rather than rebooting an undrained node.
- **Deadlines** (`DEADLINES`): queued 1 h, updating 90 min, draining 10 min,
rebooting 30 min, undraining 10 min. A minute tick (`host_agent_maintenance`)
times out a job past its deadline.
- **A maintenance window** scoped to the host's agent metrics
(`host_agent/<host>`) opens when the job starts, lasts as long as the rest of
the job could take, and ends when it does.
- Ingest applies all of this in a savepoint of its own: a job going wrong is
logged and never costs the host its samples.
On the host's Agent panel: a running job leads what the host has waiting, with
since-when; a finished one says what it did ("Done: 12 packages updated,
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
job, or the panel says why one can't start.
### What happened to the host (1.14.0, #5756)