feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+33
View File
@@ -100,6 +100,19 @@ DEFAULTS: dict[str, Any] = {
# A router's 4xx share over fewer requests than this in 15 minutes isn't
# graded, on the page or for anomalies: one 404 of three is 33% (#5741).
"traefik.threshold.min_requests": 30,
# The smallest departure from a metric's normal worth calling an anomaly
# (#5742), on top of being a quarter over its usual hold: a share's floor
# is in percentage points, a response time's in ms. 5 points is the
# baseline; bursty figures (CPU, load) get 10 so an ordinary build or
# backup minute isn't news.
"norms.floor.host_cpu_pct": 10,
"norms.floor.host_mem_pct": 5,
"norms.floor.host_load_pct": 10,
"norms.floor.container_cpu_pct": 10,
"norms.floor.container_mem_pct": 5,
"norms.floor.monitor_ms": 100,
"norms.floor.router_4xx_pct": 2,
"norms.floor.router_p95_ms": 50,
# Degraded/critical cutoffs for metric coloring (warn=amber, crit=red).
# cpu/mem/disk/load are percentages (load is load-per-core %); temp in °C.
# uptime is "higher is better" so its warn/crit are floors. Ping latency
@@ -540,9 +553,23 @@ def to_thresholds_cfg(settings: dict[str, Any]) -> dict:
# Fewer requests than this in 15 minutes and a router's 4xx share isn't graded.
"router_requests": {"warn": g("traefik.threshold.min_requests", 30),
"crit": g("traefik.threshold.min_requests", 30), "dir": "low"},
# Not a cutoff: each normal's floor (norms.floor.*), read by floor_of().
"floors": {name: float(g(f"norms.floor.{name}", DEFAULTS[f"norms.floor.{name}"]))
for name in NORM_FLOORS},
}
NORM_FLOORS = ("host_cpu_pct", "host_mem_pct", "host_load_pct", "container_cpu_pct",
"container_mem_pct", "monitor_ms", "router_4xx_pct", "router_p95_ms")
def floor_of(thresholds: dict, name: str) -> float:
"""A normal's floor (one of NORM_FLOORS) from to_thresholds_cfg's output:
the smallest departure from it worth calling an anomaly (#5742)."""
got = (thresholds.get("floors") or {}).get(name)
return float(got if got is not None else DEFAULTS[f"norms.floor.{name}"])
_STATUS_STYLE = {"warn": "color:var(--yellow);font-weight:500;",
"crit": "color:var(--red);font-weight:500;"}
@@ -557,6 +584,12 @@ def threshold_style_for(value: Any, kind: str, thresholds: dict) -> str:
return _STATUS_STYLE.get(status_class(value, kind, thresholds), "")
def grade_style(grade: str | None) -> str:
"""threshold_style_for's style for a grade already worked out ("warn" /
"crit"); empty for anything else."""
return _STATUS_STYLE.get(grade or "", "")
def to_plugins_cfg(settings: dict[str, Any]) -> dict:
"""Assemble {plugin_name: {...config}} from all plugin.* keys."""
result = {}