{# steward/templates/settings/thresholds.html — tunable degraded/critical cutoffs #} {% extends "base.html" %} {% from "_macros.html" import crumbs %} {% block title %}Settings — Thresholds & Retention — Steward{% endblock %} {% from "settings/_tabs.html" import settings_tabs %} {% block section_tabs %}{{ settings_tabs("thresholds") }}{% endblock %} {% block breadcrumb %}{{ crumbs([("Settings", "/settings/"), ("Thresholds & retention", "")]) }}{% endblock %} {% block content %} {% macro pair(label, warn_field, warn_key, crit_field, crit_key, unit, hint, step="1") %}
warn crit
{% if hint %}
{{ hint }}
{% endif %}
{% endmacro %}
{{ csrf_field() }}

Monitoring thresholds

Values at or beyond warn show amber; at or beyond crit show red — across dashboard widgets and host views. Higher is worse for everything except uptime, where the cutoffs are floors.

{{ pair("CPU usage", "cpu_warn", "thresholds.cpu_warn", "cpu_crit", "thresholds.cpu_crit", "%", "Average CPU utilization per host. A host or container with its own normal is judged against that instead (see Normals below).") }} {{ pair("Memory usage", "mem_warn", "thresholds.mem_warn", "mem_crit", "thresholds.mem_crit", "%", "Memory in use per host, or a container's share of its limit. Judged against a normal once there is one.") }} {{ pair("Disk usage", "disk_warn", "thresholds.disk_warn", "disk_crit", "thresholds.disk_crit", "%", "Filesystem usage.") }} {{ pair("Load", "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core). Judged against a host's normal once it has one.") }} {{ pair("Temperature", "temp_warn", "thresholds.temp_warn", "temp_crit", "thresholds.temp_crit", "°C", "Hottest reported sensor per host.") }} {{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 and the UniFi WAN round trip. A monitor or router with its own normal is judged against that instead.") }} {{ pair("Uptime / SLA", "uptime_warn", "thresholds.uptime_warn", "uptime_crit", "thresholds.uptime_crit", "%", "Lower is worse — amber below warn, red below crit.", step="0.1") }} {{ pair("TLS certificate expiry", "tls_warn", "thresholds.tls_warn_days", "tls_crit", "thresholds.tls_crit_days", "days left", "HTTPS monitors whose certificate expires within these many days are flagged. Lower is worse.") }} {{ pair("Clock drift", "clock_warn", "thresholds.clock_warn_s", "clock_crit", "thresholds.clock_crit_s", "seconds", "How far a host's clock may be from Steward's, either way, before its page and the Hosts list call it out.") }}
A container that dies (or is OOM-killed) this many times within an hour is reported as crash-looping.
{# Traefik's router cutoffs, while the plugin runs; its page links here (#5380). #} {% if traefik_on %}

Traefik routers

How much of a router's traffic, over the last 15 minutes, may end in an error before the Traefik page and the Routers widget list it.

Above this share, a router is failing (red), whatever its history.
Above this share, a router is worth a look (amber): missing routes, failed sign-ins. Once a router has 5 days of history it is judged against its own normal for the hour instead, as is its p95, so one that always answers some 4xx stops being listed.
Below this, a router's 4xx share isn't judged, on its page or as an anomaly: one 404 of three requests is 33%. 5xx is always judged.
{% endif %} {% macro days(label, field, key, hint) %}
{% if hint %}
{{ hint }}
{% endif %}
{% endmacro %} {# Each normal's floor (#5742): the smallest departure from it worth calling an anomaly. #} {% macro floor(label, key, unit, hint, step="1") %}
{% if hint %}
{{ hint }}
{% endif %}
{% endmacro %}

Normals

Once one has 5 days of history, these learn what is normal for each host, container, monitor or router at each hour, and are judged against that instead of the cutoffs above: a reading over a cutoff but usual for that hour shows healthy, and one well above its usual level is a warning and an anomaly to review. Disk space, a monitor that is down, an agent that stopped reporting and a router's 5xx share are failures and always keep their cutoffs. A reading counts as out of its normal only when it is a quarter above its usual level and at least this much above it:

{% if "host_agent" in (plugins_on or ()) %} {{ floor("Host CPU", "norms.floor.host_cpu_pct", "% points", "CPU is bursty: a build or a backup moves it this much without anything being wrong.") }} {{ floor("Host memory", "norms.floor.host_mem_pct", "% points", "Memory moves slowly, so a smaller step already means something changed.") }} {{ floor("Host load per core", "norms.floor.host_load_pct", "% points", "Bursty, like CPU.") }} {% endif %} {% if "docker" in (plugins_on or ()) %} {{ floor("Container CPU", "norms.floor.container_cpu_pct", "% of one core", "10 is a tenth of a core; with many containers, smaller steps are everyday noise.") }} {{ floor("Container memory", "norms.floor.container_mem_pct", "% points", "Of its limit, or of the host's memory when it has none, so a container without a limit only counts large growth.") }} {% endif %} {{ floor("Monitor response time", "norms.floor.monitor_ms", "ms", "One check per interval is noisy; a slower reply than this over its usual is worth a look.") }} {% if traefik_on %} {{ floor("Router 4xx share", "norms.floor.router_4xx_pct", "% points", "", step="0.1") }} {{ floor("Router p95", "norms.floor.router_p95_ms", "ms", "") }} {% endif %}

Metric anomalies

A metric that leaves what is normal for it at that hour is listed under Alerts → Anomalies to review. One nobody marks normal or real within this window, once it has ended, lapses: it leaves the list and is kept as not reviewed, and its hours are never learned.

{{ days("Review window", "anomalies_review_days", "anomalies.review_days", "An anomaly still going never lapses.") }}

Docker lifecycle events

How long container start, stop, die and health changes are kept. Container CPU and memory are kept with the other metrics, under Metric retention below. Applied by the hourly cleanup task — a change takes effect on its next run.

{{ days("Lifecycle events", "docker_events_days", "docker.retention.events_days", "Keep container start/stop/die/health history this long.") }}

Container logs

The host agent tails each container's logs and pushes them here, viewable per container. On by default for every container; storage is bounded by a per-container ring (oldest lines rotate out once the age or size cap is hit, whichever comes first). Applied by the hourly cleanup and on ingest.

Global kill-switch. When off, pushed log lines are dropped and no new logs are stored.
Comma-separated container names whose logs are dropped on ingest (e.g. known-noisy ones).
{{ days("Log retention", "docker_logs_retention_days", "docker.logs.retention_days", "Keep each container's log lines at most this long before rotating them out.") }}
Newest lines are kept up to this size per container; older lines rotate out first.

Metric retention

Bounds how much metric history is stored: hosts (CPU, memory, disk, network, temps, …), containers (CPU and memory), monitors and the other sources. Raw per-sample points are kept for the raw window, then rolled up into hourly figures (average, peak, and the level held) kept for the rollup window. Charts read raw for the recent part of a range and hourly for older data. Applied by the hourly cleanup.

{{ days("Raw metrics", "metrics_raw_days", "metrics.retention.raw_days", "Keep per-sample metrics this long, then roll up to hourly.") }} {{ days("Rolled-up metrics", "metrics_rollup_days", "metrics.retention.rollup_days", "Keep the hourly-averaged series this long for multi-week history.") }}
Takes effect immediately — no restart.
{% endblock %}