fix(monitors): run each monitor on its own interval, from its scheduled start (#5151)
The dispatcher ticked once per MONITORS_POLL_INTERVAL and stamped last_checked_at when the whole batch finished. With a 60 s tick and a 60 s monitor the next tick saw 59.x s elapsed and skipped, so the monitor ran every 120 s; a 30 s monitor could never run faster than the tick. - The dispatcher ticks every 5 s (DISPATCH_TICK_SECONDS) and reads the monitors and intervals fresh each tick, so a changed interval applies at the next one. - Due is measured from the scheduled start, stamped before the check runs, with half a tick of slack so tick jitter never costs a whole tick. A long-overdue monitor runs once. - Each check runs in its own task and records its own result, so a slow probe holds up nobody; a monitor still being checked is not started again, and it is freed in a finally. - A check past CHECK_DEADLINE_SECONDS (120) is recorded down with "Check did not finish within 120 s" rather than holding its monitor forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+4
-2
@@ -367,7 +367,7 @@ def create_app(
|
||||
def _register_core_tasks(app: Quart) -> None:
|
||||
from .core.scheduler import ScheduledTask
|
||||
|
||||
poll_interval = app.config.get("MONITORS_POLL_INTERVAL", 60)
|
||||
from .monitors.scheduler import DISPATCH_TICK_SECONDS as MONITOR_DISPATCH_TICK
|
||||
cleanup_interval = 3600 # hourly
|
||||
|
||||
async def run_monitor_checks():
|
||||
@@ -418,10 +418,12 @@ def _register_core_tasks(app: Quart) -> None:
|
||||
interval_seconds=60,
|
||||
run_on_startup=False,
|
||||
),
|
||||
# Ticks often and only starts checks: each monitor decides whether it
|
||||
# is due from its own interval (steward/monitors/scheduler.py).
|
||||
ScheduledTask(
|
||||
name="monitor_check",
|
||||
coro_factory=run_monitor_checks,
|
||||
interval_seconds=poll_interval,
|
||||
interval_seconds=MONITOR_DISPATCH_TICK,
|
||||
run_on_startup=True,
|
||||
),
|
||||
ScheduledTask(
|
||||
|
||||
Reference in New Issue
Block a user