feat(host_agent): a group's updates and reboots on a weekly cadence, one host at a time (#5768)

#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

plugins/host_agent/scheduler.py

@@ -122,16 +122,27 @@ def make_task(app) -> ScheduledTask:
 
 
 def make_maintenance_task(app) -> ScheduledTask:
-    """Gives up on maintenance jobs past their deadline (#5767). Every other
-    move a job makes comes from its agent's posts."""
+    """Gives up on maintenance jobs past their deadline (#5767), starts due
+    maintenance plans and moves their runs on one host at a time (#5768), and
+    records each host's maintenance_failed for an alert rule. Every move a job
+    itself makes comes from its agent's posts."""
     async def _tick():
-        from . import maintenance
+        from . import cadence, maintenance
+        now = datetime.now(timezone.utc)
         try:
             async with app.db_sessionmaker() as session:
                 async with session.begin():
-                    gave_up = await maintenance.tick(session, datetime.now(timezone.utc))
+                    gave_up = await maintenance.tick(session, now)
+                    said = await cadence.tick(session, now)
+                    batch = MetricBatch()
+                    for name, value in (await cadence.failed_hosts(session, now)).items():
+                        batch.add(source_module="host_agent", resource_name=name,
+                                  metric_name="maintenance_failed", value=value, recorded_at=now)
+                    await batch.record(session)
             if gave_up:
                 logger.warning("host_agent: %d maintenance job(s) timed out", gave_up)
+            for line in said:
+                logger.info("host_agent maintenance: %s", line)
         except Exception:
             logger.exception("host_agent: maintenance tick failed")