feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -3,7 +3,9 @@
|
||||
Docker counts CPU per core (100% = one core busy), so CPU reads as cores,
|
||||
toned by its share of the host's CPUs when the host's core count is known.
|
||||
Memory reads in bytes, against a limit only when one is actually set;
|
||||
mem_pct is then a share of that limit, otherwise of the host's memory. -#}
|
||||
mem_pct is then a share of that limit, otherwise of the host's memory.
|
||||
Each is toned by usage()'s grade when it has one: within the container's
|
||||
own normal for the hour reads healthy, and says what is usual (#5742). -#}
|
||||
|
||||
{% macro cores(n) -%}
|
||||
{{ "%.2f"|format(n) if n < 1 else "%.1f"|format(n) }} {{ "core" if n == 1 else "cores" }}
|
||||
@@ -12,15 +14,15 @@
|
||||
{% macro cpu(u) -%}
|
||||
{%- if u.cores is none -%}<span class="tone-dim">—</span>
|
||||
{%- else -%}
|
||||
<span class="tone-{{ status_class(u.cpu_share, 'cpu') }}"
|
||||
title="Docker counts CPU per core: {{ '%.0f'|format(u.cores * 100) }}% of one core{% if u.cpus %}, {{ '%.1f'|format(u.cpu_share) }}% of this host's {{ u.cpus }} core{{ 's' if u.cpus != 1 }}{% endif %}">{{ cores(u.cores) }}</span>
|
||||
<span class="tone-{{ u.cpu_grade or status_class(u.cpu_share, 'cpu') }}"
|
||||
title="Docker counts CPU per core: {{ '%.0f'|format(u.cores * 100) }}% of one core{% if u.cpus %}, {{ '%.1f'|format(u.cpu_share) }}% of this host's {{ u.cpus }} core{{ 's' if u.cpus != 1 }}{% endif %}{% if u.cpu_usual %}; {{ u.cpu_usual }}{% endif %}">{{ cores(u.cores) }}</span>
|
||||
{%- endif -%}
|
||||
{%- endmacro %}
|
||||
|
||||
{% macro mem(u) -%}
|
||||
{%- if u.mem_bytes is none -%}<span class="tone-dim">—</span>
|
||||
{%- else -%}
|
||||
<span class="tone-{{ status_class(u.mem_pct, 'mem') if u.mem_of else 'dim' }}"
|
||||
title="{% if u.mem_pct is not none and u.mem_of %}{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% else %}No memory limit set{% endif %}">{{ u.mem_bytes|bytes }}{% if u.mem_limit %} / {{ u.mem_limit|bytes }}{% endif %}</span>
|
||||
<span class="tone-{{ u.mem_grade or (status_class(u.mem_pct, 'mem') if u.mem_of else 'dim') }}"
|
||||
title="{% if u.mem_pct is not none and u.mem_of %}{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% if u.mem_usual %}, {{ u.mem_usual }}{% endif %}{% else %}No memory limit set{% endif %}">{{ u.mem_bytes|bytes }}{% if u.mem_limit %} / {{ u.mem_limit|bytes }}{% endif %}</span>
|
||||
{%- endif -%}
|
||||
{%- endmacro %}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
{# docker/container_detail.html — one container (#5310), shaped like a
|
||||
service's page: a header that says whether it is well and what it is; why
|
||||
it stopped, only while it has; what it moved since it started; its CPU and
|
||||
memory over time (the one home for what it uses now); its lifecycle.
|
||||
memory over time (the one home for what it uses now), with its metric
|
||||
anomalies above when it has any; its lifecycle.
|
||||
A swarm task is titled by its service and slot, the raw name under it. #}
|
||||
{% extends "base.html" %}
|
||||
{% from "_macros.html" import crumbs, status_badge %}
|
||||
@@ -63,6 +64,9 @@
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{# ── Its CPU or memory out of its normal for the hour, and past anomalies (#5742) #}
|
||||
{% with back=anomaly_back, base_resource=anomaly_resource %}{% include "_anomaly_card.html" %}{% endwith %}
|
||||
|
||||
{# ── What it is set up with, and what it moved since it started ─────────── #}
|
||||
<div class="card">
|
||||
<dl class="kv">
|
||||
|
||||
@@ -33,13 +33,13 @@
|
||||
{%- if session.user_id %}<a href="/plugins/docker/container/{{ r.c.host_id }}/{{ r.c.name }}">{{ r.label }}</a>{% else %}{{ r.label }}{% endif -%}
|
||||
</td>
|
||||
<td class="w-wide-only w-text" title="{{ r.host_name }}">{{ r.host_name }}</td>
|
||||
<td class="tone-{{ status_class(u.cpu_share, 'cpu') if u.cpus and u.cores is not none else 'dim' }}"
|
||||
{% if u.cpus and u.cores is not none %}title="{{ '%.1f'|format(u.cpu_share) }}% of {{ r.host_name }}'s {{ u.cpus }} cores"{% endif %}>
|
||||
<td class="tone-{{ (u.cpu_grade or status_class(u.cpu_share, 'cpu')) if u.cpus and u.cores is not none else 'dim' }}"
|
||||
{% if u.cpus and u.cores is not none %}title="{{ '%.1f'|format(u.cpu_share) }}% of {{ r.host_name }}'s {{ u.cpus }} cores{% if u.cpu_usual %}; {{ u.cpu_usual }}{% endif %}"{% endif %}>
|
||||
{{- bars(r.cpu_bars, (cores(r.cpu_bars.peak) ~ " cores") if r.cpu_bars.peak is not none else "", cores(r.cpu_bars.roof) ~ " cores") -}}
|
||||
<span class="w-fig">{% if u.cores is none %}—{% else %}{{ cores(u.cores) }} cores{% endif %}</span>
|
||||
</td>
|
||||
<td class="tone-{{ status_class(u.mem_pct, 'mem') if u.mem_of and u.mem_bytes is not none else 'dim' }}"
|
||||
{% if u.mem_pct is not none and u.mem_of and u.mem_bytes is not none %}title="{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}"{% endif %}>
|
||||
<td class="tone-{{ (u.mem_grade or status_class(u.mem_pct, 'mem')) if u.mem_of and u.mem_bytes is not none else 'dim' }}"
|
||||
{% if u.mem_pct is not none and u.mem_of and u.mem_bytes is not none %}title="{{ '%.1f'|format(u.mem_pct) }}% of {{ u.mem_of }}{% if u.mem_usual %}, {{ u.mem_usual }}{% endif %}"{% endif %}>
|
||||
{{- bars(r.mem_bars, r.mem_bars.peak|bytes if r.mem_bars.peak is not none else "", r.mem_bars.roof|bytes) -}}
|
||||
<span class="w-fig">{% if u.mem_bytes is none %}—{% else %}{{ u.mem_bytes|bytes }}{% endif %}</span>
|
||||
</td>
|
||||
|
||||
Reference in New Issue
Block a user