fix(maintenance): a job given up after draining its node makes the node active again (#6387)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m48s
CI / publish (push) Successful in 20s

On 2026-10-25 the 09:00 job on VDNT-Docker02 drained the node, which ran
Traefik and Steward. The agent's result reaches Steward through Traefik,
and Traefik took 11.5 minutes to start on Docker03 (its image had to be
pulled). The job gave up at the 10-minute drain deadline and finished
timed_out where it stood, so the node stayed drained for hours.

- A job that fails or times out while its node may be drained (the drain
  was handed out, or went through and wasn't undone) goes to undraining
  first. The outcome is kept in the new column ends_as, and the job
  finishes as that once the node is active again. If the undrain goes
  wrong too, detail says both.
- The drain deadline is 30 minutes instead of 10, so a result held up while
  the proxy moves still lands.
- Migration host_agent_012_ends_as. get_maintenance_job returns ends_as.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-25 09:05:57 -04:00
co-authored by Claude Opus 5.5
parent 2a561210c3
commit d102fc3333
7 changed files with 157 additions and 9 deletions
+4 -1
View File
@@ -119,6 +119,7 @@ def _job_row(job, names: dict, now) -> dict:
"drained_by": names.get(job.drain_host_id, job.drain_host_id),
"drained_at": iso(job.drained_at),
"came_back_at": iso(job.came_back_at),
"ends_as": job.ends_as,
"docker_update": job.docker_update,
"checked": job.checked,
"detail": job.detail,
@@ -203,7 +204,9 @@ async def get_maintenance_job(host: str = "", job: str = "", limit: int = 10) ->
Docker, else before a reboot; `drained_at` is when, and its tasks get 5
min to move (settling). `state` moves queued → [checking →] [draining →
settling →] updating → [draining → settling →] [rebooting →]
[undraining →] [recovering →] done, or failed / timed_out. recovering
[undraining →] [recovering →] done, or failed / timed_out; a job given
up while its node may be drained goes through undraining first, and
`ends_as` is what it then finishes as. recovering
waits up to 10 min for those containers to run again; `detail` says why
a job failed, naming any that didn't. A job failed only for not coming
back into service in time has `came_back_at`, when its host was first