fix(maintenance): check first, drain before a Docker upgrade, wait 5 min after a drain, see containers back (#6098)
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six seconds in, before the drain that came only with the reboot, and every container on the node stopped. Docker's own packages restart the daemon from their install scripts; needrestart's list-only mode doesn't stop that. - The helper gains a `check` action (agent 1.24.0): a dry run of the update for each package manager, whether Docker is among it, and the running containers outside the swarm with their restart policies. - A swarm node is drained before an update that upgrades Docker, else only before a reboot, and at most once a job. - After a drain the job settles 5 minutes before the update or reboot. - Once the work is over, the containers that ran outside the swarm must be seen running again within 10 minutes, or the job fails naming them. - An older agent gets no check and its job runs as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -118,6 +118,9 @@ def _job_row(job, names: dict, now) -> dict:
|
||||
"rebooted_at": iso(job.back_at),
|
||||
"swarm_node": job.node_id,
|
||||
"drained_by": names.get(job.drain_host_id, job.drain_host_id),
|
||||
"drained_at": iso(job.drained_at),
|
||||
"docker_update": job.docker_update,
|
||||
"checked": job.checked,
|
||||
"detail": job.detail,
|
||||
"state_since": iso(job.state_at), "state_age_s": age_s(job.state_at, now),
|
||||
"gives_up_at": iso(started + deadline) if deadline and started else None,
|
||||
@@ -192,11 +195,19 @@ async def get_maintenance_job(host: str = "", job: str = "", limit: int = 10) ->
|
||||
limit: how many of the host's jobs (default 10, max 50).
|
||||
|
||||
A job is one host's update and, only when the update needs it, a reboot,
|
||||
with its swarm node drained first and made active again after. `state`
|
||||
moves queued → updating → [draining →] rebooting → [undraining →] done,
|
||||
or failed / timed_out; `detail` says why a job failed. `summary` is the
|
||||
host page's line ("Done: 12 packages updated, rebooted"). `gives_up_at`
|
||||
is when an active job's current state times out.
|
||||
with its swarm node drained first and made active again after. It first
|
||||
checks what the update would change: `docker_update` is whether it
|
||||
upgrades Docker (whose packages restart the daemon), and `checked` holds
|
||||
{pending, docker_packages, containers}, the containers then running
|
||||
outside the swarm. A swarm node is drained before an update that upgrades
|
||||
Docker, else before a reboot; `drained_at` is when, and its tasks get 5
|
||||
min to move (settling). `state` moves queued → [checking →] [draining →
|
||||
settling →] updating → [draining → settling →] [rebooting →]
|
||||
[undraining →] [recovering →] done, or failed / timed_out. recovering
|
||||
waits up to 10 min for those containers to run again; `detail` says why
|
||||
a job failed, naming any that didn't. `summary` is the host page's line
|
||||
("Done: 12 packages updated, rebooted"). `gives_up_at` is when an active
|
||||
job's current state times out.
|
||||
"""
|
||||
if bool(host.strip()) == bool(job.strip()):
|
||||
raise ValueError("pass either `host` (a name or id) or `job` (a job id).")
|
||||
|
||||
Reference in New Issue
Block a user