feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -233,16 +233,23 @@ def required(value: Any, what: str) -> Any:
|
||||
_SEVERITY = {"crit": 2, "warn": 1}
|
||||
|
||||
|
||||
def worst_reading(readings, thresholds: dict):
|
||||
def worst_reading(readings, thresholds: dict, normals: dict | None = None):
|
||||
"""The reading furthest past its cutoff, as (label, value, grade), or None
|
||||
when every reading is within its cutoffs or missing.
|
||||
|
||||
`readings` is (label, value, kind) triples, graded like status_class; the
|
||||
first of equally bad readings wins, so list them in the order a reader
|
||||
should hear about them."""
|
||||
should hear about them. `normals` ({kind: (Norm, floor)}) are the
|
||||
resource's own normals for this hour: a reading past its cutoff but
|
||||
within its normal is not past it (#5742)."""
|
||||
from steward.core.anomalies import graded
|
||||
normals = normals or {}
|
||||
worst, rank = None, 0
|
||||
for label, value, kind in readings:
|
||||
grade = status_class(value, kind, thresholds)
|
||||
if kind in normals:
|
||||
grade = graded(value, kind, thresholds, *normals[kind], "%")[0]
|
||||
else:
|
||||
grade = status_class(value, kind, thresholds)
|
||||
if _SEVERITY.get(grade, 0) > rank:
|
||||
worst, rank = (label, value, grade), _SEVERITY[grade]
|
||||
return worst
|
||||
@@ -327,6 +334,7 @@ METRICS: dict[str, tuple[str, str]] = {
|
||||
"load_1m": ("Load (1 min)", "runnable processes, averaged over a minute"),
|
||||
"load_5m": ("Load (5 min)", "runnable processes, averaged over 5 minutes"),
|
||||
"load_15m": ("Load (15 min)", "runnable processes, averaged over 15 minutes"),
|
||||
"load_pct": ("Load per core", "1-minute load as a % of the host's cores; 100% keeps every core busy"),
|
||||
"uptime_secs": ("Uptime", "time since the host booted"),
|
||||
"seconds_since_seen": ("Time since the agent reported", "over 3 minutes means the agent is offline"),
|
||||
"clock_off_secs": ("Clock off by", "how far the host's clock is from Steward's, either way"),
|
||||
|
||||
Reference in New Issue
Block a user