feat(host_agent): Steward hands a host a maintenance job and follows it through the reboot (#5767)
#5767 "Steward hands a host a maintenance job and follows it to the end, through its own restarts", milestone 514. A host_maintenance_jobs row per job: queued → updating → [draining →] rebooting → [undraining →] done, or failed / timed_out. Steward's answer to an ingest post carries the step that host's agent should carry out now; the agent (1.18.0) leaves each step id for the root helper once, and the helper refuses an id it already ran, so repeats are harmless. Results come back as the #5766 maintenance events, matched by step id; the reboot completes when the host's `started` event has a boot_at after the reboot was handed out. The job reboots only when the update says one is needed (or never, if asked). A swarm node is drained first by itself if it's a manager or by a manager with the helper, and made active after; with no manager to drain it the job is refused. A minute tick times out a job past its state's deadline. A maintenance window over host_agent/<host> lasts the rest of the job. Everything is on the row, so a Steward restart, or the reboot of the host Steward runs on, resumes from the next post. Ingest runs it in its own savepoint so a job going wrong never costs the samples. The Agent panel leads with a running job and says what a finished one did; Manage agent has Update now (with "never reboot"), or says why a job can't start. Integration test: hand off, Steward restarts, the post-boot events close the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -174,7 +174,52 @@ host that has an older agent.
|
||||
|
||||
So a stolen agent token can make its host update, reboot or drain, and nothing
|
||||
else. Steward handing a host a job and following it through the reboot is
|
||||
#5767; the cadence that does it one host at a time is #5768.
|
||||
#5767 (below); the cadence that does it one host at a time is #5768.
|
||||
|
||||
### A maintenance job, followed through the reboot (1.18.0, #5767)
|
||||
|
||||
Steward keeps one `host_maintenance_jobs` row per job and moves it on only from
|
||||
what the agents post, so a restart of Steward, or a reboot of the host Steward
|
||||
runs on, loses nothing. `plugins/host_agent/maintenance.py` has the states and
|
||||
transitions (model-free, unit-tested) and the database side.
|
||||
|
||||
```
|
||||
queued → updating → [draining →] rebooting → [undraining →] done
|
||||
↘ done (no reboot needed, or the job may never reboot)
|
||||
any state → failed (a step's result said so) | timed_out (its deadline passed)
|
||||
```
|
||||
|
||||
- **Hand-off.** Steward's answer to an ingest post carries
|
||||
`"maintenance": [{"id", "action", ["node"]}]`, the step that host's agent
|
||||
should carry out now. It is repeated on every post until a result for that
|
||||
step id arrives; the agent leaves each id for the helper once
|
||||
(`take_maintenance`), and the helper refuses an id it already has a result
|
||||
for, so a repeat, or a re-hand after the agent restarts, never runs twice.
|
||||
A step id is `<job id>-<action>`.
|
||||
- **Results** are the `maintenance` events of #5766, matched to the job by step
|
||||
id, so a drain reported by a manager reaches the job of the node it drained.
|
||||
- **The reboot completes** when the host's agent sends `started` with a
|
||||
`boot_at` after the reboot was handed out (60 s of clock slack). An agent
|
||||
restart on a host that stayed up has an old `boot_at` and changes nothing.
|
||||
- **Reboot only when required**: the update's result says whether a reboot is
|
||||
needed; a job made with "never reboot" stops after the update.
|
||||
- **Swarm nodes** are drained first and made active after. The node is found
|
||||
by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
|
||||
worker is drained by a manager whose agent has the helper. With no such
|
||||
manager the job is refused rather than rebooting an undrained node.
|
||||
- **Deadlines** (`DEADLINES`): queued 1 h, updating 90 min, draining 10 min,
|
||||
rebooting 30 min, undraining 10 min. A minute tick (`host_agent_maintenance`)
|
||||
times out a job past its deadline.
|
||||
- **A maintenance window** scoped to the host's agent metrics
|
||||
(`host_agent/<host>`) opens when the job starts, lasts as long as the rest of
|
||||
the job could take, and ends when it does.
|
||||
- Ingest applies all of this in a savepoint of its own: a job going wrong is
|
||||
logged and never costs the host its samples.
|
||||
|
||||
On the host's Agent panel: a running job leads what the host has waiting, with
|
||||
since-when; a finished one says what it did ("Done: 12 packages updated,
|
||||
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
|
||||
job, or the panel says why one can't start.
|
||||
|
||||
### What happened to the host (1.14.0, #5756)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user