feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m26s
CI / integration (push) Failing after 1m27s
CI / publish (push) Skipped

Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.

- Each normal's floor, the smallest departure that counts, is a setting:
  Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
  points, host memory 5, load per core 10, container CPU 10 points of one
  core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
  and 50 ms p95 floors move there from constants. learn_normal takes a
  floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
  hosts' host_grades, docker usage(thresholds=, norms=), monitors'
  latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
  Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 16:17:04 -04:00
co-authored by Claude Opus 5.5
parent f3cfb89612
commit f3433fe3ce
30 changed files with 638 additions and 93 deletions
+7 -5
View File
@@ -3,7 +3,9 @@
Docker counts CPU per core (100% = one core busy), so CPU reads as cores,
toned by its share of the host's CPUs when the host's core count is known.
Memory reads in bytes, against a limit only when one is actually set;
mem_pct is then a share of that limit, otherwise of the host's memory. -#}
mem_pct is then a share of that limit, otherwise of the host's memory.
Each is toned by usage()'s grade when it has one: within the container's
own normal for the hour reads healthy, and says what is usual (#5742). -#}
{% macro cores(n) -%}
{{ "%.2f"|format(n) if n < 1 else "%.1f"|format(n) }} {{ "core" if n == 1 else "cores" }}
@@ -12,15 +14,15 @@
{% macro cpu(u) -%}
{%- if u.cores is none -%}<span class="tone-dim">—</span>
{%- else -%}
<span class="tone-{{ status_class(u.cpu_share, 'cpu') }}"
title="Docker counts CPU per core: {{ '%.0f'|format(u.cores * 100) }}% of one core{% if u.cpus %}, {{ '%.1f'|format(u.cpu_share) }}% of this host's {{ u.cpus }} core{{ 's' if u.cpus != 1 }}{% endif %}">{{ cores(u.cores) }}</span>
<span class="tone-{{ u.cpu_grade or status_class(u.cpu_share, 'cpu') }}"
title="Docker counts CPU per core: {{ '%.0f'|format(u.cores * 100) }}% of one core{% if u.cpus %}, {{ '%.1f'|format(u.cpu_share) }}% of this host's {{ u.cpus }} core{{ 's' if u.cpus != 1 }}{% endif %}{% if u.cpu_usual %}; {{ u.cpu_usual }}{% endif %}">{{ cores(u.cores) }}</span>
{%- endif -%}
{%- endmacro %}
{% macro mem(u) -%}
{%- if u.mem_bytes is none -%}<span class="tone-dim">—</span>
{%- else -%}
<span class="tone-{{ status_class(u.mem_pct, 'mem') if u.mem_of else 'dim' }}"
title="{% if u.mem_pct is not none and u.mem_of %}{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% else %}No memory limit set{% endif %}">{{ u.mem_bytes|bytes }}{% if u.mem_limit %} / {{ u.mem_limit|bytes }}{% endif %}</span>
<span class="tone-{{ u.mem_grade or (status_class(u.mem_pct, 'mem') if u.mem_of else 'dim') }}"
title="{% if u.mem_pct is not none and u.mem_of %}{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% if u.mem_usual %}, {{ u.mem_usual }}{% endif %}{% else %}No memory limit set{% endif %}">{{ u.mem_bytes|bytes }}{% if u.mem_limit %} / {{ u.mem_limit|bytes }}{% endif %}</span>
{%- endif -%}
{%- endmacro %}
@@ -1,7 +1,8 @@
{# docker/container_detail.html — one container (#5310), shaped like a
service's page: a header that says whether it is well and what it is; why
it stopped, only while it has; what it moved since it started; its CPU and
memory over time (the one home for what it uses now); its lifecycle.
memory over time (the one home for what it uses now), with its metric
anomalies above when it has any; its lifecycle.
A swarm task is titled by its service and slot, the raw name under it. #}
{% extends "base.html" %}
{% from "_macros.html" import crumbs, status_badge %}
@@ -63,6 +64,9 @@
</section>
{% endif %}
{# ── Its CPU or memory out of its normal for the hour, and past anomalies (#5742) #}
{% with back=anomaly_back, base_resource=anomaly_resource %}{% include "_anomaly_card.html" %}{% endwith %}
{# ── What it is set up with, and what it moved since it started ─────────── #}
<div class="card">
<dl class="kv">
@@ -33,13 +33,13 @@
{%- if session.user_id %}<a href="/plugins/docker/container/{{ r.c.host_id }}/{{ r.c.name }}">{{ r.label }}</a>{% else %}{{ r.label }}{% endif -%}
</td>
<td class="w-wide-only w-text" title="{{ r.host_name }}">{{ r.host_name }}</td>
<td class="tone-{{ status_class(u.cpu_share, 'cpu') if u.cpus and u.cores is not none else 'dim' }}"
{% if u.cpus and u.cores is not none %}title="{{ '%.1f'|format(u.cpu_share) }}% of {{ r.host_name }}'s {{ u.cpus }} cores"{% endif %}>
<td class="tone-{{ (u.cpu_grade or status_class(u.cpu_share, 'cpu')) if u.cpus and u.cores is not none else 'dim' }}"
{% if u.cpus and u.cores is not none %}title="{{ '%.1f'|format(u.cpu_share) }}% of {{ r.host_name }}'s {{ u.cpus }} cores{% if u.cpu_usual %}; {{ u.cpu_usual }}{% endif %}"{% endif %}>
{{- bars(r.cpu_bars, (cores(r.cpu_bars.peak) ~ " cores") if r.cpu_bars.peak is not none else "", cores(r.cpu_bars.roof) ~ " cores") -}}
<span class="w-fig">{% if u.cores is none %}—{% else %}{{ cores(u.cores) }} cores{% endif %}</span>
</td>
<td class="tone-{{ status_class(u.mem_pct, 'mem') if u.mem_of and u.mem_bytes is not none else 'dim' }}"
{% if u.mem_pct is not none and u.mem_of and u.mem_bytes is not none %}title="{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}"{% endif %}>
<td class="tone-{{ (u.mem_grade or status_class(u.mem_pct, 'mem')) if u.mem_of and u.mem_bytes is not none else 'dim' }}"
{% if u.mem_pct is not none and u.mem_of and u.mem_bytes is not none %}title="{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% if u.mem_usual %}, {{ u.mem_usual }}{% endif %}"{% endif %}>
{{- bars(r.mem_bars, r.mem_bars.peak|bytes if r.mem_bars.peak is not none else "", r.mem_bars.roof|bytes) -}}
<span class="w-fig">{% if u.mem_bytes is none %}—{% else %}{{ u.mem_bytes|bytes }}{% endif %}</span>
</td>