feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+8 -4
View File
@@ -1,6 +1,8 @@
{# hosts/overview_widget.html — the dashboard Hosts widget (#5251): a verdict,
then one row per host, worst first, with its resources under one header.
Load is shown against the host's cores and graded per core. Reachability
Load is shown against the host's cores and graded per core. CPU, memory
and load within the host's own normal for the hour read healthy (#5742),
with what is usual on hover. Reachability
and uptime belong to the Monitors widget, so they aren't repeated here. #}
{% from "_macros.html" import status_dot %}
{% if not rows %}
@@ -30,10 +32,12 @@
{%- elif not a %}<span class="w-sub">no agent</span>{% endif %}
</td>
{% for v, kind in ((a.get('cpu_pct'), 'cpu'), (a.get('mem_used_pct'), 'mem'), (a.get('disk_root'), 'disk')) %}
<td style="{{ threshold_style(v, kind) if fresh and v is not none }}">{% if v is none %}—{% else %}{{ "%.0f"|format(v) }}%{% endif %}</td>
{% set g = (r.grades or {}).get(kind) or (none, none) %}
<td style="{{ grade_style(g[0]) }}"{% if g[1] %} title="{{ g[1]|capitalize }}"{% endif %}>{% if v is none %}—{% else %}{{ "%.0f"|format(v) }}%{% endif %}</td>
{% endfor %}
<td style="{{ threshold_style(a.get('load_per_core'), 'load') if fresh and a.get('load_per_core') is not none }}"
title="{% if a.get('cores') %}Load over 1 minute on {{ a.cores }} cores: {{ a.load_per_core }}% of them busy or waiting{% endif %}">
{% set g = (r.grades or {}).get('load') or (none, none) %}
<td style="{{ grade_style(g[0]) }}"
title="{% if a.get('cores') %}Load over 1 minute on {{ a.cores }} cores: {{ a.load_per_core }}% of them busy or waiting{% endif %}{% if g[1] %}; {{ g[1] }}{% endif %}">
{%- if a.get('load_1m') is none %}—{% elif a.get('cores') %}{{ "%.1f"|format(a.load_1m) }} / {{ a.cores }} cores{% else %}{{ "%.1f"|format(a.load_1m) }}{% endif -%}
</td>
</tr>