feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+2 -2
View File
@@ -26,10 +26,10 @@
{{- status_dot(d.state) -}}
{% if session.user_id %}<a href="/monitors/{{ m.id }}">{{ m.name }}</a>{% else %}{{ m.name }}{% endif %}
{%- if d.state == 'down' %}<span class="w-sub tone-crit">went down {{ d.down_since|ago if d.down_since else '' }}{% if d.down_reason %} · {{ d.down_reason }}{% endif %}</span>
{%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if latest and status_class(latest.response_ms, 'latency') == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
{%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if d.latency == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
{%- elif d.state == 'pending' %}<span class="w-sub">not checked yet</span>{% endif %}
</td>
<td style="{{ threshold_style(latest.response_ms, 'latency') if latest and latest.is_up and latest.response_ms is not none }}">
<td class="{{ 'tone-' ~ d.latency if d.latency in ('warn', 'crit') }}"{% if d.latency_usual %} title="{{ d.latency_usual|capitalize }}"{% endif %}>
{%- if latest and latest.is_up and latest.response_ms is not none %}{{ "%.0f"|format(latest.response_ms) }} ms{% else %}—{% endif -%}
</td>
{% for k in ("24h", "7d", "30d") %}{% set u = d.uptime.get(k) %}