fix(maintenance): a host that came back late stops reading as failed once it is back (#6105)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m30s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m5s

Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.

Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-19 14:23:53 -04:00
co-authored by Claude Opus 5.5
parent fcd58224de
commit 18032a4782
4 changed files with 64 additions and 11 deletions
+1 -1
View File
@@ -356,7 +356,7 @@ METRICS: dict[str, tuple[str, str]] = {
"updates_pending": ("Updates waiting", "packages the host's package manager would upgrade, checked hourly"),
"updates_security": ("Security updates waiting", "of the updates waiting, those from a security source"),
"reboot_required": ("Reboot required", "1 when the host needs a reboot to finish an update, 0 when not"),
"maintenance_failed": ("Maintenance failed", "1 when the host's latest scheduled or manual update failed, timed out or didn't come back into service"),
"maintenance_failed": ("Maintenance failed", "1 when the host's latest scheduled or manual update failed or timed out, or the host didn't come back into service and still isn't"),
"temp_c_max": ("Hottest sensor", "the host's hottest real thermometer; inputs with nothing wired to them are left out"),
"temp_c": ("Sensor temperature", "one temperature sensor (resource host:temp:label)"),
"net_rx_bps": ("Network in", "bytes per second received, by the whole host or one interface"),