feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/templates/hosts/overview_widget.html

@@ -1,6 +1,8 @@
 {# hosts/overview_widget.html — the dashboard Hosts widget (#5251): a verdict,
    then one row per host, worst first, with its resources under one header.
-   Load is shown against the host's cores and graded per core. Reachability
+   Load is shown against the host's cores and graded per core. CPU, memory
+   and load within the host's own normal for the hour read healthy (#5742),
+   with what is usual on hover. Reachability
    and uptime belong to the Monitors widget, so they aren't repeated here. #}
 {% from "_macros.html" import status_dot %}
 {% if not rows %}
@@ -30,10 +32,12 @@
       {%- elif not a %}<span class="w-sub">no agent</span>{% endif %}
     </td>
     {% for v, kind in ((a.get('cpu_pct'), 'cpu'), (a.get('mem_used_pct'), 'mem'), (a.get('disk_root'), 'disk')) %}
-    <td style="{{ threshold_style(v, kind) if fresh and v is not none }}">{% if v is none %}—{% else %}{{ "%.0f"|format(v) }}%{% endif %}</td>
+    {% set g = (r.grades or {}).get(kind) or (none, none) %}
+    <td style="{{ grade_style(g[0]) }}"{% if g[1] %} title="{{ g[1]|capitalize }}"{% endif %}>{% if v is none %}—{% else %}{{ "%.0f"|format(v) }}%{% endif %}</td>
     {% endfor %}
-    <td style="{{ threshold_style(a.get('load_per_core'), 'load') if fresh and a.get('load_per_core') is not none }}"
-        title="{% if a.get('cores') %}Load over 1 minute on {{ a.cores }} cores: {{ a.load_per_core }}% of them busy or waiting{% endif %}">
+    {% set g = (r.grades or {}).get('load') or (none, none) %}
+    <td style="{{ grade_style(g[0]) }}"
+        title="{% if a.get('cores') %}Load over 1 minute on {{ a.cores }} cores: {{ a.load_per_core }}% of them busy or waiting{% endif %}{% if g[1] %}; {{ g[1] }}{% endif %}">
       {%- if a.get('load_1m') is none %}—{% elif a.get('cores') %}{{ "%.1f"|format(a.load_1m) }} / {{ a.cores }} cores{% else %}{{ "%.1f"|format(a.load_1m) }}{% endif -%}
     </td>
   </tr>

steward/templates/monitors/detail.html

@@ -1,7 +1,8 @@
 {% extends "base.html" %}
 {% from "_macros.html" import crumbs, page_header, stat_tile %}
 {# One monitor: its state and for how long, latency and uptime against
-   Settings → Thresholds, its response time over the last day, its last
+   Settings → Thresholds (latency against its own normal once it has one,
+   with its anomalies), its response time over the last day, its last
    checks, the failures behind them and how it is set up. #}
 {% block title %}{{ monitor.name }} — Monitors — Steward{% endblock %}
 {% block breadcrumb %}{{ crumbs([("Monitors", "/monitors/"), (monitor.name, "")]) }}{% endblock %}
@@ -23,7 +24,7 @@
   {% set since_hint = ((("for at least " if since_at_least else "for ") ~ (since|ago|striptags|replace(" ago", ""))) if since and state in ("up", "down", "degraded") else none) %}
   {{ stat_tile("State", status_info(state)[1], state=state, hint=since_hint) }}
   {% if latest and latest.response_ms is not none %}
-  {{ stat_tile("Latency", "%.0f ms"|format(latest.response_ms), state=status_class(latest.response_ms, "latency")) }}
+  {{ stat_tile("Latency", "%.0f ms"|format(latest.response_ms), state=latency[0], hint=latency[1]) }}
   {% endif %}
   {% for window in ("24h", "7d", "30d") %}
   {% set pct = uptime.get(window) %}
@@ -34,6 +35,8 @@
   {% endif %}
 </div>
 
+{% with back="/monitors/" ~ monitor.id, base_resource=monitor.name %}{% include "_anomaly_card.html" %}{% endwith %}
+
 {% set timed = history | selectattr("ms", "ne", none) | list %}
 {% if timed|length > 1 %}
 <section class="card" aria-labelledby="m-chart">

steward/templates/monitors/rows.html

@@ -51,7 +51,7 @@
     {% if d.state in ('down', 'paused', 'pending') %}{{ status_badge(d.state) }}
     {% elif latest.response_ms is not none %}
       {% if d.spark %}<span class="mon-spark" title="Response time over the last 30 checks">{{ d.spark|safe }}</span>{% endif %}
-      <span class="tone-{{ status_class(latest.response_ms, 'latency') }}">{{ "%.0f"|format(latest.response_ms) }} ms</span>
+      <span class="tone-{{ d.latency }}"{% if d.latency_usual %} title="{{ d.latency_usual|capitalize }}"{% endif %}>{{ "%.0f"|format(latest.response_ms) }} ms</span>
     {% elif m.type == 'dns' and latest.resolved_ip %}<span class="tone-ok">{{ latest.resolved_ip }}</span>
     {% else %}{{ status_badge('up') }}{% endif %}
   </div>

steward/templates/monitors/widget.html

@@ -26,10 +26,10 @@
       {{- status_dot(d.state) -}}
       {% if session.user_id %}<a href="/monitors/{{ m.id }}">{{ m.name }}</a>{% else %}{{ m.name }}{% endif %}
       {%- if d.state == 'down' %}<span class="w-sub tone-crit">went down {{ d.down_since|ago if d.down_since else '' }}{% if d.down_reason %} · {{ d.down_reason }}{% endif %}</span>
-      {%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if latest and status_class(latest.response_ms, 'latency') == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
+      {%- elif d.state == 'degraded' %}<span class="w-sub tone-warn">{% if d.latency == 'crit' %}slow replies{% else %}checks failing now and then{% endif %}</span>
       {%- elif d.state == 'pending' %}<span class="w-sub">not checked yet</span>{% endif %}
     </td>
-    <td style="{{ threshold_style(latest.response_ms, 'latency') if latest and latest.is_up and latest.response_ms is not none }}">
+    <td class="{{ 'tone-' ~ d.latency if d.latency in ('warn', 'crit') }}"{% if d.latency_usual %} title="{{ d.latency_usual|capitalize }}"{% endif %}>
       {%- if latest and latest.is_up and latest.response_ms is not none %}{{ "%.0f"|format(latest.response_ms) }} ms{% else %}—{% endif -%}
     </td>
     {% for k in ("24h", "7d", "30d") %}{% set u = d.uptime.get(k) %}

steward/templates/settings/thresholds.html

@@ -35,12 +35,12 @@
     Higher is worse for everything except uptime, where the cutoffs are floors.
   </p>
 
-  {{ pair("CPU usage",    "cpu_warn",  "thresholds.cpu_warn",  "cpu_crit",  "thresholds.cpu_crit",  "%", "Average CPU utilization per host.") }}
-  {{ pair("Memory usage", "mem_warn",  "thresholds.mem_warn",  "mem_crit",  "thresholds.mem_crit",  "%", "Memory in use per host.") }}
+  {{ pair("CPU usage",    "cpu_warn",  "thresholds.cpu_warn",  "cpu_crit",  "thresholds.cpu_crit",  "%", "Average CPU utilization per host. A host or container with its own normal is judged against that instead (see Normals below).") }}
+  {{ pair("Memory usage", "mem_warn",  "thresholds.mem_warn",  "mem_crit",  "thresholds.mem_crit",  "%", "Memory in use per host, or a container's share of its limit. Judged against a normal once there is one.") }}
   {{ pair("Disk usage",   "disk_warn", "thresholds.disk_warn", "disk_crit", "thresholds.disk_crit", "%", "Filesystem usage.") }}
-  {{ pair("Load",         "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core).") }}
+  {{ pair("Load",         "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core). Judged against a host's normal once it has one.") }}
   {{ pair("Temperature",  "temp_warn", "thresholds.temp_warn", "temp_crit", "thresholds.temp_crit", "°C", "Hottest reported sensor per host.") }}
-  {{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 (until a router has its own normal) and the UniFi WAN round trip.") }}
+  {{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 and the UniFi WAN round trip. A monitor or router with its own normal is judged against that instead.") }}
   {{ pair("Uptime / SLA", "uptime_warn", "thresholds.uptime_warn", "uptime_crit", "thresholds.uptime_crit", "%", "Lower is worse — amber below warn, red below crit.", step="0.1") }}
   {{ pair("TLS certificate expiry", "tls_warn", "thresholds.tls_warn_days", "tls_crit", "thresholds.tls_crit_days", "days left", "HTTPS monitors whose certificate expires within these many days are flagged. Lower is worse.") }}
   {{ pair("Clock drift", "clock_warn", "thresholds.clock_warn_s", "clock_crit", "thresholds.clock_crit_s", "seconds", "How far a host's clock may be from Steward's, either way, before its page and the Hosts list call it out.") }}
@@ -99,6 +99,42 @@
 </div>
 {% endmacro %}
 
+{# Each normal's floor (#5742): the smallest departure from it worth calling an anomaly. #}
+{% macro floor(label, key, unit, hint, step="1") %}
+<div class="form-group" style="margin-bottom:1.1rem;">
+  <label for="{{ key }}">{{ label }} <span style="color:var(--text-muted);font-size:0.8rem;">({{ unit }})</span></label>
+  <input type="number" id="{{ key }}" name="{{ key }}" min="0" step="{{ step }}"
+         value="{{ settings[key] }}" style="max-width:110px;margin-top:0.25rem;display:block;">
+  {% if hint %}<div style="font-size:0.78rem;color:var(--text-muted);margin-top:0.3rem;">{{ hint }}</div>{% endif %}
+</div>
+{% endmacro %}
+
+<div class="card" id="normals" style="max-width:640px;margin-top:1.5rem;">
+  <h2 class="section-title" style="margin-bottom:0.5rem;">Normals</h2>
+  <p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">
+    Once one has 5 days of history, these learn what is normal for each host, container, monitor or router
+    at each hour, and are judged against that instead of the cutoffs above: a reading over a cutoff but usual
+    for that hour shows healthy, and one well above its usual level is a warning and an anomaly to review.
+    Disk space, a monitor that is down, an agent that stopped reporting and a router's 5xx share are failures
+    and always keep their cutoffs. A reading counts as out of its normal only when it is a quarter above its
+    usual level <em>and</em> at least this much above it:
+  </p>
+  {% if "host_agent" in (plugins_on or ()) %}
+  {{ floor("Host CPU", "norms.floor.host_cpu_pct", "% points", "CPU is bursty: a build or a backup moves it this much without anything being wrong.") }}
+  {{ floor("Host memory", "norms.floor.host_mem_pct", "% points", "Memory moves slowly, so a smaller step already means something changed.") }}
+  {{ floor("Host load per core", "norms.floor.host_load_pct", "% points", "Bursty, like CPU.") }}
+  {% endif %}
+  {% if "docker" in (plugins_on or ()) %}
+  {{ floor("Container CPU", "norms.floor.container_cpu_pct", "% of one core", "10 is a tenth of a core; with many containers, smaller steps are everyday noise.") }}
+  {{ floor("Container memory", "norms.floor.container_mem_pct", "% points", "Of its limit, or of the host's memory when it has none, so a container without a limit only counts large growth.") }}
+  {% endif %}
+  {{ floor("Monitor response time", "norms.floor.monitor_ms", "ms", "One check per interval is noisy; a slower reply than this over its usual is worth a look.") }}
+  {% if traefik_on %}
+  {{ floor("Router 4xx share", "norms.floor.router_4xx_pct", "% points", "", step="0.1") }}
+  {{ floor("Router p95", "norms.floor.router_p95_ms", "ms", "") }}
+  {% endif %}
+</div>
+
 <div class="card" style="max-width:640px;margin-top:1.5rem;">
   <h2 class="section-title" style="margin-bottom:0.5rem;">Metric anomalies</h2>
   <p style="font-size:0.82rem;color:var(--text-muted);margin-bottom:1.25rem;">