fix(maintenance): a late return, once back, stays back, and every reader agrees (#6175)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m31s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m26s

18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.

The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 07:44:38 -04:00
co-authored by Claude Opus 5.5
parent b75b883326
commit 8e2f17d198
11 changed files with 99 additions and 18 deletions
+1 -1
View File
@@ -358,7 +358,7 @@ METRICS: dict[str, tuple[str, str]] = {
"updates_pending": ("Updates waiting", "packages the host's package manager would upgrade, checked hourly"),
"updates_security": ("Security updates waiting", "of the updates waiting, those from a security source"),
"reboot_required": ("Reboot required", "1 when the host needs a reboot to finish an update, 0 when not"),
"maintenance_failed": ("Maintenance failed", "1 when the host's latest scheduled or manual update failed or timed out, or the host didn't come back into service and still isn't"),
"maintenance_failed": ("Maintenance failed", "1 when the host's latest scheduled or manual update failed or timed out, or the host didn't come back into service in time and hasn't been back since"),
"temp_c_max": ("Hottest sensor", "the host's hottest real thermometer; inputs with nothing wired to them are left out"),
"temp_c": ("Sensor temperature", "one temperature sensor (resource host:temp:label)"),
"net_rx_bps": ("Network in", "bytes per second received, by the whole host, one interface or one container"),