Commit Graph
2 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 1ba4784fa6 fix(maintenance): check first, drain before a Docker upgrade, wait 5 min after a drain, see containers back (#6098)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m29s
CI / integration (push) Successful in 2m7s
CI / publish (push) Skipped
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.

- The helper gains a `check` action (agent 1.24.0): a dry run of the update
  for each package manager, whether Docker is among it, and the running
  containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
  before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
  seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 10:44:45 -04:00
bvandeusenandClaude Opus 5.5 200384e52d feat(host_agent): a group's updates and reboots on a weekly cadence, one host at a time (#5768)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m20s
CI / unit (push) Successful in 1m34s
CI / publish (push) Skipped
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:59:07 -04:00