feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -111,6 +111,9 @@ def agent_findings(hosts, agent: dict, thresholds: dict) -> list[dict]:
|
||||
"host agent has stopped reporting; its metrics are out of date",
|
||||
since=a.get("last_at"), ids=ids, next=f'get_host(host="{h.id}")'))
|
||||
continue # stale values would only add noise
|
||||
# CPU and memory within the host's own normal for the hour aren't
|
||||
# findings (#5742); the fullest disk keeps its cutoff.
|
||||
normals = host_svc.host_normals(a.get("norms"), thresholds)
|
||||
for metric, kind, label in _AGENT_CHECKS:
|
||||
value = a.get(metric)
|
||||
th = thresholds.get(kind) or {}
|
||||
@@ -122,6 +125,8 @@ def agent_findings(hosts, agent: dict, thresholds: dict) -> list[dict]:
|
||||
sev = "warning"
|
||||
else:
|
||||
continue
|
||||
if kind in normals and not anom.against(value, normals[kind][0], normals[kind][1], "%")[0]:
|
||||
continue
|
||||
out.append(finding(
|
||||
sev, f"host_{kind}_high", h.name,
|
||||
f"{label} at {rnd(value, 1)}% (warn {th['warn']}%, crit {th.get('crit')}%)",
|
||||
|
||||
Reference in New Issue
Block a user