fix(maintenance): a late return, once back, stays back, and every reader agrees (#6175)
18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -34,7 +34,7 @@ def test_a_job_row_says_what_it_did_and_when_it_gives_up():
|
||||
reboot_policy="if_required", packages=12, reboot_needed=True, back_at=None,
|
||||
node_id="n1", drain_host_id="h2", detail=None, created_by=None, run_id="r1",
|
||||
created_at=NOW - timedelta(minutes=20), finished_at=None,
|
||||
drained_at=NOW - timedelta(minutes=11), docker_update=False,
|
||||
drained_at=NOW - timedelta(minutes=11), docker_update=False, came_back_at=None,
|
||||
checked={"pending": 12, "docker_packages": [], "containers": []})
|
||||
row = _job_row(job, {"h1": "web-1", "h2": "mgr-1"}, NOW)
|
||||
assert (row["host"], row["drained_by"], row["label"], row["active"]) == (
|
||||
|
||||
Reference in New Issue
Block a user