feat(host_agent): Steward hands a host a maintenance job and follows it through the reboot (#5767)
#5767 "Steward hands a host a maintenance job and follows it to the end, through its own restarts", milestone 514. A host_maintenance_jobs row per job: queued → updating → [draining →] rebooting → [undraining →] done, or failed / timed_out. Steward's answer to an ingest post carries the step that host's agent should carry out now; the agent (1.18.0) leaves each step id for the root helper once, and the helper refuses an id it already ran, so repeats are harmless. Results come back as the #5766 maintenance events, matched by step id; the reboot completes when the host's `started` event has a boot_at after the reboot was handed out. The job reboots only when the update says one is needed (or never, if asked). A swarm node is drained first by itself if it's a manager or by a manager with the helper, and made active after; with no manager to drain it the job is refused. A minute tick times out a job past its state's deadline. A maintenance window over host_agent/<host> lasts the rest of the job. Everything is on the row, so a Steward restart, or the reboot of the host Steward runs on, resumes from the next post. Ingest runs it in its own savepoint so a job going wrong never costs the samples. The Agent panel leads with a running job and says what a finished one did; Manage agent has Update now (with "never reboot"), or says why a job can't start. Integration test: hand off, Steward restarts, the post-boot events close the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -119,3 +119,25 @@ def make_task(app) -> ScheduledTask:
|
||||
interval_seconds=60,
|
||||
run_on_startup=False,
|
||||
)
|
||||
|
||||
|
||||
def make_maintenance_task(app) -> ScheduledTask:
|
||||
"""Gives up on maintenance jobs past their deadline (#5767). Every other
|
||||
move a job makes comes from its agent's posts."""
|
||||
async def _tick():
|
||||
from . import maintenance
|
||||
try:
|
||||
async with app.db_sessionmaker() as session:
|
||||
async with session.begin():
|
||||
gave_up = await maintenance.tick(session, datetime.now(timezone.utc))
|
||||
if gave_up:
|
||||
logger.warning("host_agent: %d maintenance job(s) timed out", gave_up)
|
||||
except Exception:
|
||||
logger.exception("host_agent: maintenance tick failed")
|
||||
|
||||
return ScheduledTask(
|
||||
name="host_agent_maintenance",
|
||||
coro_factory=_tick,
|
||||
interval_seconds=60,
|
||||
run_on_startup=False,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user