fix(monitors): run each monitor on its own interval, from its scheduled start (#5151)

The dispatcher ticked once per MONITORS_POLL_INTERVAL and stamped
last_checked_at when the whole batch finished. With a 60 s tick and a 60 s
monitor the next tick saw 59.x s elapsed and skipped, so the monitor ran
every 120 s; a 30 s monitor could never run faster than the tick.

- The dispatcher ticks every 5 s (DISPATCH_TICK_SECONDS) and reads the
  monitors and intervals fresh each tick, so a changed interval applies at
  the next one.
- Due is measured from the scheduled start, stamped before the check runs,
  with half a tick of slack so tick jitter never costs a whole tick. A
  long-overdue monitor runs once.
- Each check runs in its own task and records its own result, so a slow
  probe holds up nobody; a monitor still being checked is not started
  again, and it is freed in a finally.
- A check past CHECK_DEADLINE_SECONDS (120) is recorded down with "Check did
  not finish within 120 s" rather than holding its monitor forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 18:31:07 -04:00
co-authored by Claude Opus 5.5
parent fc9730a2e5
commit edea8482b5
3 changed files with 226 additions and 30 deletions
+4 -2
View File
@@ -367,7 +367,7 @@ def create_app(
def _register_core_tasks(app: Quart) -> None:
from .core.scheduler import ScheduledTask
poll_interval = app.config.get("MONITORS_POLL_INTERVAL", 60)
from .monitors.scheduler import DISPATCH_TICK_SECONDS as MONITOR_DISPATCH_TICK
cleanup_interval = 3600 # hourly
async def run_monitor_checks():
@@ -418,10 +418,12 @@ def _register_core_tasks(app: Quart) -> None:
interval_seconds=60,
run_on_startup=False,
),
# Ticks often and only starts checks: each monitor decides whether it
# is due from its own interval (steward/monitors/scheduler.py).
ScheduledTask(
name="monitor_check",
coro_factory=run_monitor_checks,
interval_seconds=poll_interval,
interval_seconds=MONITOR_DISPATCH_TICK,
run_on_startup=True,
),
ScheduledTask(