fix(maintenance): check first, drain before a Docker upgrade, wait 5 min after a drain, see containers back (#6098)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m29s
CI / integration (push) Successful in 2m7s
CI / publish (push) Skipped

On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.

- The helper gains a `check` action (agent 1.24.0): a dry run of the update
  for each package manager, whether Docker is among it, and the running
  containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
  before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
  seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-19 10:44:45 -04:00
co-authored by Claude Opus 5.5
parent da81010a75
commit 1ba4784fa6
15 changed files with 671 additions and 67 deletions
+7
View File
@@ -114,6 +114,13 @@ class HostMaintenanceJob(Base):
# The cadence run this job belongs to (#5768); None for Update now.
run_id = Column(String(36), ForeignKey("host_maintenance_runs.id", ondelete="SET NULL"),
nullable=True, index=True)
# What the check said the update would change (#6098): whether it
# upgrades Docker, and {pending, docker_packages, containers} with the
# containers then running outside the swarm. drained_at is when the job's
# drain finished, before the update or before the reboot.
docker_update = Column(Boolean, nullable=True)
checked = Column(JSON, nullable=True)
drained_at = Column(DateTime(timezone=True), nullable=True)
class HostMaintenancePlan(Base):