feat(norms): Steward learns each metric's normal per hour of day, then week (#5739)
CI / unit (push) Failing after 0s
CI / lint (push) Failing after 0s
CI / integration (push) Failing after 1s
CI / publish (push) Skipped

A metric registered with learn_normal() gets, for every resource reporting
it, a usual hold (the 90th percentile across past days of each hour's p95)
and a usual peak for each slot:
- an hour of the day once 5 days of history exist;
- an hour of the week once history reaches back 3 weeks, so weekly jobs are
  learned too.
Each slot takes in the hours either side, so a job that runs a little late
stays normal. One SQL statement relearns everything hourly into
metric_norms, from the rollup plus the raw window not rolled up yet;
norms_at() reads the slot a moment falls in. Nothing is registered yet:
Traefik (#5741) and hosts (#5742) opt in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 12:48:44 -04:00
co-authored by Claude Opus 5.5
parent cbbf186f17
commit 4c84bc8560
8 changed files with 385 additions and 1 deletions
+2 -1
View File
@@ -6,6 +6,7 @@ from .password_resets import PasswordReset
from .hosts import Host
from .monitors import Monitor, MonitorResult, MonitorType
from .metrics import PluginMetric
from .norms import MetricNorm
from .alerts import AlertRule, AlertState, AlertEvent, AlertOperator, AlertStateEnum
from .ansible import AnsibleRun, AnsibleRunStatus
from .ansible_inventory import AnsibleTarget, AnsibleGroup, ansible_target_groups
@@ -18,7 +19,7 @@ __all__ = [
"Base", "User", "ApiToken", "UserInvite", "PasswordReset",
"Host",
"Monitor", "MonitorResult", "MonitorType",
"PluginMetric",
"PluginMetric", "MetricNorm",
"AlertRule", "AlertState", "AlertEvent", "AlertOperator", "AlertStateEnum",
"AnsibleRun", "AnsibleRunStatus",
"AnsibleTarget", "AnsibleGroup", "ansible_target_groups",
+27
View File
@@ -0,0 +1,27 @@
from __future__ import annotations
from datetime import datetime
from sqlalchemy import DateTime, Float, Integer, SmallInteger, String
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class MetricNorm(Base):
"""What one metric usually does in one time slot (#5739), learned from
its own history by steward.core.norms.rebuild_norms.
`kind` says what a slot is: "day" (0–23, the UTC hour of day) or "week"
(0–167, Monday 00:00 UTC first). A key has slots of one kind at a time.
The whole table is rebuilt each pass, so a row is never edited."""
__tablename__ = "metric_norms"
source_module: Mapped[str] = mapped_column(String(64), primary_key=True)
resource_name: Mapped[str] = mapped_column(String(255), primary_key=True)
metric_name: Mapped[str] = mapped_column(String(128), primary_key=True)
slot: Mapped[int] = mapped_column(SmallInteger, primary_key=True)
kind: Mapped[str] = mapped_column(String(8), nullable=False)
# The level the slot's hours usually held, and how high they usually
# peaked: the 90th percentile, across past days, of each hour's p95 and max.
usual_hold: Mapped[float] = mapped_column(Float, nullable=False)
usual_peak: Mapped[float] = mapped_column(Float, nullable=False)
days_seen: Mapped[int] = mapped_column(Integer, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)