feat(host_agent): a group's updates and reboots on a weekly cadence, one host at a time (#5768)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m20s
CI / unit (push) Successful in 1m34s
CI / publish (push) Skipped

#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 18:59:07 -04:00
co-authored by Claude Opus 5.5
parent 78581c843e
commit 200384e52d
17 changed files with 1077 additions and 15 deletions
+31
View File
@@ -221,6 +221,37 @@ since-when; a finished one says what it did ("Done: 12 packages updated,
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
job, or the panel says why one can't start.
### A maintenance cadence: a group, one host at a time (#5768)
`plugins/host_agent/cadence.py`, driven by the same minute tick as the jobs.
A **plan** (`host_maintenance_plans`) names an inventory group, a weekday, a
time in a time zone (the creator's, by default), a window and a reboot policy:
`if_required` (the default), `always` or `never`. The page is **Hosts →
Maintenance** (`/plugins/host_agent/maintenance`).
- **When.** A plan's slot is the latest weekday-and-time at or before now, in
its zone. A slot fires once (`last_slot_at`) and only while its window is
open, so a Steward that was down over the slot starts late, not a day late.
A new or resumed plan starts from the next slot. **Run now** starts a run
without touching the schedule (the Ansible ruling on Run now).
- **A run** (`host_maintenance_runs`) fixes its hosts at the start, by name:
the group's hosts through their linked Ansible targets. A host with no agent,
or no maintenance helper, is skipped and listed.
- **One at a time.** Each host gets a #5767 job (`run_id` on the job). The next
host starts only when the last is **back in service**: its agent posting in
the last 3 min and, for a swarm node, the node `ready`/`active` and every
replicated service at its replicas, judged only from swarm reports made
after the job finished. A host not back within 30 min fails the run, and
its job is marked failed with the reason.
- **Stops.** A failed or timed-out job fails the run. No host is started after
the window; the run then stops and says how many weren't reached.
- **Alerting** is an ordinary alert rule on `host_agent/<host>/maintenance_failed`:
1 when the host's latest job (in 30 days) failed, timed out or didn't come
back into service, 0 once a later one succeeds.
- The page leads with a running run and its hosts, then failed runs from the
last week, then each plan with its schedule, next slot and last five runs.
Each host's Agent panel shows its last maintenance job.
### What happened to the host (1.14.0, #5756)
A post may carry `events`, said once each, with the next post that gets through: