feat(host_agent): a group's updates and reboots on a weekly cadence, one host at a time (#5768)
#5768 "A maintenance cadence: a group's updates and reboots, one host at a time, on a schedule". A plan names an inventory group, a weekday and time in a time zone, a window and a reboot policy (only when required by default). At its slot a run takes the group's hosts in order and gives each a #5767 maintenance job. The next host starts only when the last is back in service: its agent is posting and, for a swarm node, the node is active and every replicated service is at its replicas. A failure stops the run, and no host is started once the window has passed. A host's latest failure shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule. The page is Hosts → Maintenance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -221,6 +221,37 @@ since-when; a finished one says what it did ("Done: 12 packages updated,
|
||||
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
|
||||
job, or the panel says why one can't start.
|
||||
|
||||
### A maintenance cadence: a group, one host at a time (#5768)
|
||||
|
||||
`plugins/host_agent/cadence.py`, driven by the same minute tick as the jobs.
|
||||
A **plan** (`host_maintenance_plans`) names an inventory group, a weekday, a
|
||||
time in a time zone (the creator's, by default), a window and a reboot policy:
|
||||
`if_required` (the default), `always` or `never`. The page is **Hosts →
|
||||
Maintenance** (`/plugins/host_agent/maintenance`).
|
||||
|
||||
- **When.** A plan's slot is the latest weekday-and-time at or before now, in
|
||||
its zone. A slot fires once (`last_slot_at`) and only while its window is
|
||||
open, so a Steward that was down over the slot starts late, not a day late.
|
||||
A new or resumed plan starts from the next slot. **Run now** starts a run
|
||||
without touching the schedule (the Ansible ruling on Run now).
|
||||
- **A run** (`host_maintenance_runs`) fixes its hosts at the start, by name:
|
||||
the group's hosts through their linked Ansible targets. A host with no agent,
|
||||
or no maintenance helper, is skipped and listed.
|
||||
- **One at a time.** Each host gets a #5767 job (`run_id` on the job). The next
|
||||
host starts only when the last is **back in service**: its agent posting in
|
||||
the last 3 min and, for a swarm node, the node `ready`/`active` and every
|
||||
replicated service at its replicas, judged only from swarm reports made
|
||||
after the job finished. A host not back within 30 min fails the run, and
|
||||
its job is marked failed with the reason.
|
||||
- **Stops.** A failed or timed-out job fails the run. No host is started after
|
||||
the window; the run then stops and says how many weren't reached.
|
||||
- **Alerting** is an ordinary alert rule on `host_agent/<host>/maintenance_failed`:
|
||||
1 when the host's latest job (in 30 days) failed, timed out or didn't come
|
||||
back into service, 0 once a later one succeeds.
|
||||
- The page leads with a running run and its hosts, then failed runs from the
|
||||
last week, then each plan with its schedule, next slot and last five runs.
|
||||
Each host's Agent panel shows its last maintenance job.
|
||||
|
||||
### What happened to the host (1.14.0, #5756)
|
||||
|
||||
A post may carry `events`, said once each, with the next post that gets through:
|
||||
|
||||
Reference in New Issue
Block a user