feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+40 -4
View File
@@ -35,12 +35,12 @@
Higher is worse for everything except uptime, where the cutoffs are floors.
</p>
{{ pair("CPU usage", "cpu_warn", "thresholds.cpu_warn", "cpu_crit", "thresholds.cpu_crit", "%", "Average CPU utilization per host.") }}
{{ pair("Memory usage", "mem_warn", "thresholds.mem_warn", "mem_crit", "thresholds.mem_crit", "%", "Memory in use per host.") }}
{{ pair("CPU usage", "cpu_warn", "thresholds.cpu_warn", "cpu_crit", "thresholds.cpu_crit", "%", "Average CPU utilization per host. A host or container with its own normal is judged against that instead (see Normals below).") }}
{{ pair("Memory usage", "mem_warn", "thresholds.mem_warn", "mem_crit", "thresholds.mem_crit", "%", "Memory in use per host, or a container's share of its limit. Judged against a normal once there is one.") }}
{{ pair("Disk usage", "disk_warn", "thresholds.disk_warn", "disk_crit", "thresholds.disk_crit", "%", "Filesystem usage.") }}
{{ pair("Load", "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core).") }}
{{ pair("Load", "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core). Judged against a host's normal once it has one.") }}
{{ pair("Temperature", "temp_warn", "thresholds.temp_warn", "temp_crit", "thresholds.temp_crit", "°C", "Hottest reported sensor per host.") }}
{{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 (until a router has its own normal) and the UniFi WAN round trip.") }}
{{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 and the UniFi WAN round trip. A monitor or router with its own normal is judged against that instead.") }}
{{ pair("Uptime / SLA", "uptime_warn", "thresholds.uptime_warn", "uptime_crit", "thresholds.uptime_crit", "%", "Lower is worse — amber below warn, red below crit.", step="0.1") }}
{{ pair("TLS certificate expiry", "tls_warn", "thresholds.tls_warn_days", "tls_crit", "thresholds.tls_crit_days", "days left", "HTTPS monitors whose certificate expires within these many days are flagged. Lower is worse.") }}
{{ pair("Clock drift", "clock_warn", "thresholds.clock_warn_s", "clock_crit", "thresholds.clock_crit_s", "seconds", "How far a host's clock may be from Steward's, either way, before its page and the Hosts list call it out.") }}
@@ -99,6 +99,42 @@
</div>
{% endmacro %}
{# Each normal's floor (#5742): the smallest departure from it worth calling an anomaly. #}
{% macro floor(label, key, unit, hint, step="1") %}
<div class="form-group" style="margin-bottom:1.1rem;">
<label for="{{ key }}">{{ label }} <span style="color:var(--text-muted);font-size:0.8rem;">({{ unit }})</span></label>
<input type="number" id="{{ key }}" name="{{ key }}" min="0" step="{{ step }}"
value="{{ settings[key] }}" style="max-width:110px;margin-top:0.25rem;display:block;">
{% if hint %}<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">{{ hint }}</div>{% endif %}
</div>
{% endmacro %}
<div class="card" id="normals" style="max-width:640px;margin-top:1.5rem;">
<h2 class="section-title" style="margin-bottom:0.5rem;">Normals</h2>
<p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">
Once one has 5 days of history, these learn what is normal for each host, container, monitor or router
at each hour, and are judged against that instead of the cutoffs above: a reading over a cutoff but usual
for that hour shows healthy, and one well above its usual level is a warning and an anomaly to review.
Disk space, a monitor that is down, an agent that stopped reporting and a router's 5xx share are failures
and always keep their cutoffs. A reading counts as out of its normal only when it is a quarter above its
usual level <em>and</em> at least this much above it:
</p>
{% if "host_agent" in (plugins_on or ()) %}
{{ floor("Host CPU", "norms.floor.host_cpu_pct", "% points", "CPU is bursty: a build or a backup moves it this much without anything being wrong.") }}
{{ floor("Host memory", "norms.floor.host_mem_pct", "% points", "Memory moves slowly, so a smaller step already means something changed.") }}
{{ floor("Host load per core", "norms.floor.host_load_pct", "% points", "Bursty, like CPU.") }}
{% endif %}
{% if "docker" in (plugins_on or ()) %}
{{ floor("Container CPU", "norms.floor.container_cpu_pct", "% of one core", "10 is a tenth of a core; with many containers, smaller steps are everyday noise.") }}
{{ floor("Container memory", "norms.floor.container_mem_pct", "% points", "Of its limit, or of the host's memory when it has none, so a container without a limit only counts large growth.") }}
{% endif %}
{{ floor("Monitor response time", "norms.floor.monitor_ms", "ms", "One check per interval is noisy; a slower reply than this over its usual is worth a look.") }}
{% if traefik_on %}
{{ floor("Router 4xx share", "norms.floor.router_4xx_pct", "% points", "", step="0.1") }}
{{ floor("Router p95", "norms.floor.router_p95_ms", "ms", "") }}
{% endif %}
</div>
<div class="card" style="max-width:640px;margin-top:1.5rem;">
<h2 class="section-title" style="margin-bottom:0.5rem;">Metric anomalies</h2>
<p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">