fix(maintenance): check first, drain before a Docker upgrade, wait 5 min after a drain, see containers back (#6098)

On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.

- The helper gains a `check` action (agent 1.24.0): a dry run of the update
  for each package manager, whether Docker is among it, and the running
  containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
  before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
  seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plugins/host-agent-design.md

@@ -160,14 +160,24 @@ systemd when the agent leaves it a request:
 - `steward-maint.path` (`PathExistsGlob=…/requests/*.json`) starts
   `steward-maint.service`, a oneshot running `maint.py` as root.
 - The helper opens each request with `O_NOFOLLOW`, wants a regular file under
-  4 KiB, and accepts only `update`, `reboot`, `drain` or `undrain`, an id of
+  4 KiB, and accepts only `check`, `update`, `reboot`, `drain` or `undrain`, an id of
   letters, digits and dashes, and a node id in the swarm's own 25-character
   form. It deletes every request, valid or not, and writes the outcome to
   `/var/lib/steward-maint/results/<id>.json` (root's; kept a week). A request
   whose id already has a result is not run again.
+- **check** (1.24.0, #6098): what the update would upgrade, changing nothing
+  but the package lists: `apt-get update` then `apt-get -s dist-upgrade`;
+  `dnf`/`yum check-update`; `zypper list-updates` (`dup --dry-run` on
+  Tumbleweed); `pacman -Sy` then `-Qu`. It says whether Docker is among it
+  (`docker`, `docker-ce`, `docker.io`, `docker-engine`, `moby-engine`,
+  `containerd`, `containerd.io`: the packages whose install scripts restart
+  the daemon), and lists the running containers that aren't swarm tasks,
+  each with its restart policy (at most 40).
 - **update**: `apt-get update` then `dist-upgrade` keeping existing config
   files, with `NEEDRESTART_MODE=l` so needrestart only lists services rather
-  than restarting the Docker daemon under its containers;
+  than restarting them. A package's own install script still restarts its
+  service: Docker's does, stopping every container on the host, which is
+  what the check is for;
   `dnf -y upgrade --refresh` (or yum's); `zypper refresh` then
   `zypper update` (`zypper dup` on Tumbleweed), counting zypper's 102/103
   "reboot / restart needed" exits as success; or `pacman -Syu`
@@ -199,11 +209,27 @@ runs on, loses nothing. `plugins/host_agent/maintenance.py` has the states and
 transitions (model-free, unit-tested) and the database side.
 
 ```
-queued → updating → [draining →] rebooting → [undraining →] done
-                  ↘ done (no reboot needed, or the job may never reboot)
+queued → [checking →] [draining → settling →] updating
+       → [draining → settling →] [rebooting →] [undraining →] [recovering →] done
 any state → failed (a step's result said so) | timed_out (its deadline passed)
 ```
 
+**Docker upgraded, drained first (1.24.0, #6098).** On 2026-10-18 an update
+on VDNT-Docker02 restarted the Docker daemon six seconds in, before the drain
+that came only with the reboot: every container on the node stopped, and
+portal_app was down until it was redeployed by hand. A job for an agent of
+1.24.0 or later now starts with a **check**. When the update upgrades Docker,
+a swarm node is drained before it; otherwise it is drained only before a
+reboot, and a node is drained at most once a job. After a drain, **settling**
+gives the node's tasks 5 minutes (`DRAIN_SETTLE`) to move before the update
+or reboot goes ahead. The containers the check found running outside the
+swarm are kept on the job (`checked`); once the work is over, **recovering**
+waits up to 10 minutes (`RECOVER_LIMIT`) for each to be seen running again
+in the Docker plugin's stored containers, and fails the job naming any that
+aren't, marking those with no restart policy. settling and recovering are
+Steward's own waits, moved on by the minute tick. An older agent gets no
+check, and its job runs as before.
+
 - **Hand-off.** Steward's answer to an ingest post carries
   `"maintenance": [{"id", "action", ["node"]}]`, the step that host's agent
   should carry out now. It is repeated on every post until a result for that
@@ -218,13 +244,15 @@ any state → failed (a step's result said so) | timed_out (its deadline passed)
   restart on a host that stayed up has an old `boot_at` and changes nothing.
 - **Reboot only when required**: the update's result says whether a reboot is
   needed; a job made with "never reboot" stops after the update.
-- **Swarm nodes** are drained first and made active after. The node is found
+- **Swarm nodes** are drained first and made active after (see above for
+  when). The node is found
   by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
   worker is drained by a manager whose agent has the helper. With no such
   manager the job is refused rather than rebooting an undrained node.
-- **Deadlines** (`DEADLINES`): queued 1 h, updating 90 min, draining 10 min,
-  rebooting 30 min, undraining 10 min. A minute tick (`host_agent_maintenance`)
-  times out a job past its deadline.
+- **Deadlines** (`DEADLINES`): queued 1 h, checking 30 min, draining 10 min,
+  settling 15 min, updating 90 min, rebooting 30 min, undraining 10 min,
+  recovering 15 min. A minute tick (`host_agent_maintenance`) times out a job
+  past its deadline.
 - **A maintenance window** scoped to the host's agent metrics
   (`host_agent/<host>`) opens when the job starts, lasts as long as the rest of
   the job could take, and ends when it does.