fix(maintenance): a job given up after draining its node makes the node active again (#6387)
On 2026-10-25 the 09:00 job on VDNT-Docker02 drained the node, which ran Traefik and Steward. The agent's result reaches Steward through Traefik, and Traefik took 11.5 minutes to start on Docker03 (its image had to be pulled). The job gave up at the 10-minute drain deadline and finished timed_out where it stood, so the node stayed drained for hours. - A job that fails or times out while its node may be drained (the drain was handed out, or went through and wasn't undone) goes to undraining first. The outcome is kept in the new column ends_as, and the job finishes as that once the node is active again. If the undrain goes wrong too, detail says both. - The drain deadline is 30 minutes instead of 10, so a result held up while the proxy moves still lands. - Migration host_agent_012_ends_as. get_maintenance_job returns ends_as. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -214,8 +214,21 @@ transitions (model-free, unit-tested) and the database side.
|
||||
queued → [checking →] [draining → settling →] updating
|
||||
→ [draining → settling →] [rebooting →] [undraining →] [recovering →] done
|
||||
any state → failed (a step's result said so) | timed_out (its deadline passed)
|
||||
draining / settling / updating / rebooting, once drained → undraining → failed | timed_out
|
||||
```
|
||||
|
||||
**A job never ends leaving its drain behind (#6387).** On 2026-10-25 a job
|
||||
drained VDNT-Docker02, which ran Traefik and Steward. Its agent's result
|
||||
reaches Steward through Traefik, and Traefik took 11.5 minutes to start on
|
||||
Docker03 (its image had to be pulled). The job gave up at the 10-minute drain
|
||||
deadline and finished `timed_out` where it stood, leaving the node drained
|
||||
for hours. Now a job that fails or times out while its node may be drained
|
||||
(the drain was handed out, or went through and wasn't undone) goes to
|
||||
**undraining** first, with the outcome kept in `ends_as`, and finishes as that
|
||||
once the node is active again. If the undrain goes wrong too, `detail` says
|
||||
both. The drain deadline is 30 minutes, so a result held up by the proxy
|
||||
moving still lands.
|
||||
|
||||
**Docker upgraded, drained first (1.24.0, #6098).** On 2026-10-18 an update
|
||||
on VDNT-Docker02 restarted the Docker daemon six seconds in, before the drain
|
||||
that came only with the reboot: every container on the node stopped, and
|
||||
@@ -251,7 +264,7 @@ check, and its job runs as before.
|
||||
by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
|
||||
worker is drained by a manager whose agent has the helper. With no such
|
||||
manager the job is refused rather than rebooting an undrained node.
|
||||
- **Deadlines** (`DEADLINES`): queued 1 h, checking 30 min, draining 10 min,
|
||||
- **Deadlines** (`DEADLINES`): queued 1 h, checking 30 min, draining 30 min,
|
||||
settling 15 min, updating 90 min, rebooting 30 min, undraining 10 min,
|
||||
recovering 15 min. A minute tick (`host_agent_maintenance`) times out a job
|
||||
past its deadline.
|
||||
|
||||
Reference in New Issue
Block a user