feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+5 -2
View File
@@ -1,7 +1,8 @@
{% extends "base.html" %}
{% from "_macros.html" import crumbs, page_header, stat_tile %}
{# One monitor: its state and for how long, latency and uptime against
Settings → Thresholds, its response time over the last day, its last
Settings → Thresholds (latency against its own normal once it has one,
with its anomalies), its response time over the last day, its last
checks, the failures behind them and how it is set up. #}
{% block title %}{{ monitor.name }} — Monitors — Steward{% endblock %}
{% block breadcrumb %}{{ crumbs([("Monitors", "/monitors/"), (monitor.name, "")]) }}{% endblock %}
@@ -23,7 +24,7 @@
{% set since_hint = ((("for at least " if since_at_least else "for ") ~ (since|ago|striptags|replace(" ago", ""))) if since and state in ("up", "down", "degraded") else none) %}
{{ stat_tile("State", status_info(state)[1], state=state, hint=since_hint) }}
{% if latest and latest.response_ms is not none %}
{{ stat_tile("Latency", "%.0f ms"|format(latest.response_ms), state=status_class(latest.response_ms, "latency")) }}
{{ stat_tile("Latency", "%.0f ms"|format(latest.response_ms), state=latency[0], hint=latency[1]) }}
{% endif %}
{% for window in ("24h", "7d", "30d") %}
{% set pct = uptime.get(window) %}
@@ -34,6 +35,8 @@
{% endif %}
</div>
{% with back="/monitors/" ~ monitor.id, base_resource=monitor.name %}{% include "_anomaly_card.html" %}{% endwith %}
{% set timed = history | selectattr("ms", "ne", none) | list %}
{% if timed|length > 1 %}
<section class="card" aria-labelledby="m-chart">
+1 -1
View File
@@ -51,7 +51,7 @@
{% if d.state in ('down', 'paused', 'pending') %}{{ status_badge(d.state) }}
{% elif latest.response_ms is not none %}
{% if d.spark %}<span class="mon-spark" title="Response time over the last 30 checks">{{ d.spark|safe }}</span>{% endif %}
<span class="tone-{{ status_class(latest.response_ms, 'latency') }}">{{ "%.0f"|format(latest.response_ms) }} ms</span>
<span class="tone-{{ d.latency }}"{% if d.latency_usual %} title="{{ d.latency_usual|capitalize }}"{% endif %}>{{ "%.0f"|format(latest.response_ms) }} ms</span>
{% elif m.type == 'dns' and latest.resolved_ip %}<span class="tone-ok">{{ latest.resolved_ip }}</span>
{% else %}{{ status_badge('up') }}{% endif %}
</div>
+2 -2
View File
@@ -26,10 +26,10 @@
{{- status_dot(d.state) -}}
{% if session.user_id %}<a href="/monitors/{{ m.id }}">{{ m.name }}</a>{% else %}{{ m.name }}{% endif %}
{%- if d.state == 'down' %}<span class="w-sub tone-crit">went down {{ d.down_since|ago if d.down_since else '' }}{% if d.down_reason %} · {{ d.down_reason }}{% endif %}</span>
{%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if latest and status_class(latest.response_ms, 'latency') == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
{%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if d.latency == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
{%- elif d.state == 'pending' %}<span class="w-sub">not checked yet</span>{% endif %}
</td>
<td style="{{ threshold_style(latest.response_ms, 'latency') if latest and latest.is_up and latest.response_ms is not none }}">
<td class="{{ 'tone-' ~ d.latency if d.latency in ('warn', 'crit') }}"{% if d.latency_usual %} title="{{ d.latency_usual|capitalize }}"{% endif %}>
{%- if latest and latest.is_up and latest.response_ms is not none %}{{ "%.0f"|format(latest.response_ms) }} ms{% else %}—{% endif -%}
</td>
{% for k in ("24h", "7d", "30d") %}{% set u = d.uptime.get(k) %}