fix(maintenance): a late return, once back, stays back, and every reader agrees (#6175)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m31s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m26s

18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.

The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 07:44:38 -04:00
co-authored by Claude Opus 5.5
parent b75b883326
commit 8e2f17d198
11 changed files with 99 additions and 18 deletions
+2 -1
View File
@@ -1099,6 +1099,7 @@ async def host_panel(host_id: str):
host=host, reg=reg, reporting=reporting, stale=stale, target=target, downtime=downtime,
maint_job=maint_job, maint_blocker=maint_blocker,
maint_summary=maintenance.summary(maint_job) if maint_job else None,
maint_failed=maint_job is not None and maintenance.still_failed(maint_job),
ansible_available=has_capability("ansible.run_playbook"),
managed_key_set=bool((ansible_cfg.get("ssh_public_key") or "").strip()),
)
@@ -1448,7 +1449,7 @@ def _run_rows(run, jobs: dict, names: dict) -> list[dict]:
job = jobs.get((run.id, host_id))
if job is not None:
text = maintenance.summary(job)
bad = job.state in ("failed", "timed_out")
bad = maintenance.still_failed(job)
busy = job.state in maintenance.ACTIVE
elif run.state == "running" and i == run.position:
text, bad, busy = "Starting", False, True