fix(maintenance): a late return, once back, stays back, and every reader agrees (#6175)
18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -175,4 +175,18 @@ def test_a_host_that_came_back_late_reads_failed_only_until_it_is_back():
|
||||
await _post(app, ta)
|
||||
assert (await _tick(app))[a.name] == 0.0
|
||||
|
||||
# #6175: once back, a later wobble doesn't flag it again, and the
|
||||
# overview agrees with the figure.
|
||||
await _sql("UPDATE host_agent_registrations SET last_seen_at = now() - interval '1 hour' "
|
||||
"WHERE host_id = :h")
|
||||
assert (await _tick(app))[a.name] == 0.0
|
||||
mcp_tools = _ha("mcp_tools")
|
||||
async with app.db_sessionmaker() as s:
|
||||
(came_back,) = (await s.execute(text(
|
||||
"SELECT came_back_at FROM host_maintenance_jobs WHERE host_id = :h"),
|
||||
{"h": a.id})).one()
|
||||
flagged = await mcp_tools._failed_jobs(s, datetime.now(timezone.utc))
|
||||
assert came_back is not None
|
||||
assert a.name not in [name for _, name in flagged]
|
||||
|
||||
asyncio.run(_go())
|
||||
|
||||
Reference in New Issue
Block a user