feat(norms): Steward learns each metric's normal per hour of day, then week (#5739)
CI / unit (push) Failing after 0s
CI / lint (push) Failing after 0s
CI / integration (push) Failing after 1s
CI / publish (push) Skipped

A metric registered with learn_normal() gets, for every resource reporting
it, a usual hold (the 90th percentile across past days of each hour's p95)
and a usual peak for each slot:
- an hour of the day once 5 days of history exist;
- an hour of the week once history reaches back 3 weeks, so weekly jobs are
  learned too.
Each slot takes in the hours either side, so a job that runs a little late
stays normal. One SQL statement relearns everything hourly into
metric_norms, from the rollup plus the raw window not rolled up yet;
norms_at() reads the slot a moment falls in. Nothing is registered yet:
Traefik (#5741) and hosts (#5742) opt in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 12:48:44 -04:00
co-authored by Claude Opus 5.5
parent cbbf186f17
commit 4c84bc8560
8 changed files with 385 additions and 1 deletions
+13
View File
@@ -445,6 +445,10 @@ def _register_core_tasks(app: Quart) -> None:
from .backup.job import tick
await tick(app)
async def run_metric_norms():
from .core.norms import run_norms
await run_norms(app)
async def run_email_outbox():
from .core.outbox import send_due
await send_due(app)
@@ -507,6 +511,15 @@ def _register_core_tasks(app: Quart) -> None:
interval_seconds=cleanup_interval,
run_on_startup=False,
),
# Each learned metric's normal per hour (#5739). Hourly, since a slot
# gains one hour of history an hour; the table outlives a restart, so
# there is nothing to rebuild at startup.
ScheduledTask(
name="metric_norms",
coro_factory=run_metric_norms,
interval_seconds=3600,
run_on_startup=False,
),
])
# Pull Ansible git sources as each falls due. One task reads the sources