feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -26,12 +26,12 @@ def setup(app: "Quart") -> None:
|
||||
# learned from the scrape-by-scrape figures only: before #5755 it was a
|
||||
# percentile of every response since Traefik started.
|
||||
from datetime import datetime, timezone
|
||||
from steward.core.norms import learn_normal
|
||||
from .routes import FLOOR_4XX, FLOOR_P95, STATE_WINDOW, min_requests
|
||||
from steward.core.norms import floor_setting, learn_normal
|
||||
from .routes import STATE_WINDOW, min_requests
|
||||
gate = ("request_rate",
|
||||
lambda: min_requests(app.config.get("THRESHOLDS", {})) / STATE_WINDOW.total_seconds())
|
||||
learn_normal("traefik", "error_rate_4xx_pct", FLOOR_4XX, gate=gate)
|
||||
learn_normal("traefik", "latency_p95_ms", FLOOR_P95, gate=gate,
|
||||
learn_normal("traefik", "error_rate_4xx_pct", floor_setting(app, "router_4xx_pct"), gate=gate)
|
||||
learn_normal("traefik", "latency_p95_ms", floor_setting(app, "router_p95_ms"), gate=gate,
|
||||
since=datetime(2026, 10, 14, 18, 47, tzinfo=timezone.utc))
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user