{# steward/templates/settings/thresholds.html — tunable degraded/critical cutoffs #} {% extends "base.html" %} {% from "_macros.html" import crumbs %} {% block title %}Settings — Thresholds & Retention — Steward{% endblock %} {% from "settings/_tabs.html" import settings_tabs %} {% block section_tabs %}{{ settings_tabs("thresholds") }}{% endblock %} {% block breadcrumb %}{{ crumbs([("Settings", "/settings/"), ("Thresholds & retention", "")]) }}{% endblock %} {% block content %} {% macro pair(label, warn_field, warn_key, crit_field, crit_key, unit, hint, step="1") %}
warn crit
{% if hint %}
{{ hint }}
{% endif %}
{% endmacro %}
{{ csrf_field() }}

Monitoring thresholds

Values at or beyond warn show amber; at or beyond crit show red — across dashboard widgets and host views. Higher is worse for everything except uptime, where the cutoffs are floors.

{{ pair("CPU usage", "cpu_warn", "thresholds.cpu_warn", "cpu_crit", "thresholds.cpu_crit", "%", "Average CPU utilization per host.") }} {{ pair("Memory usage", "mem_warn", "thresholds.mem_warn", "mem_crit", "thresholds.mem_crit", "%", "Memory in use per host.") }} {{ pair("Disk usage", "disk_warn", "thresholds.disk_warn", "disk_crit", "thresholds.disk_crit", "%", "Filesystem usage.") }} {{ pair("Load", "load_warn", "thresholds.load_warn", "load_crit", "thresholds.load_crit", "% per core", "Load average normalized by core count (100% = one full core per core).") }} {{ pair("Temperature", "temp_warn", "thresholds.temp_warn", "temp_crit", "thresholds.temp_crit", "°C", "Hottest reported sensor per host.") }} {{ pair("Response time", "latency_warn", "ping.threshold.good_ms", "latency_crit", "ping.threshold.warn_ms", "ms", "How long a monitor's check takes to answer. Also grades Traefik routers' p95 (until a router has its own normal) and the UniFi WAN round trip.") }} {{ pair("Uptime / SLA", "uptime_warn", "thresholds.uptime_warn", "uptime_crit", "thresholds.uptime_crit", "%", "Lower is worse — amber below warn, red below crit.", step="0.1") }} {{ pair("TLS certificate expiry", "tls_warn", "thresholds.tls_warn_days", "tls_crit", "thresholds.tls_crit_days", "days left", "HTTPS monitors whose certificate expires within these many days are flagged. Lower is worse.") }} {{ pair("Clock drift", "clock_warn", "thresholds.clock_warn_s", "clock_crit", "thresholds.clock_crit_s", "seconds", "How far a host's clock may be from Steward's, either way, before its page and the Hosts list call it out.") }}
A container that dies (or is OOM-killed) this many times within an hour is reported as crash-looping.
{# Traefik's router cutoffs, while the plugin runs; its page links here (#5380). #} {% if traefik_on %}

Traefik routers

How much of a router's traffic, over the last 15 minutes, may end in an error before the Traefik page and the Routers widget list it.

Above this share, a router is failing (red), whatever its history.
Above this share, a router is worth a look (amber): missing routes, failed sign-ins. Once a router has 5 days of history it is judged against its own normal for the hour instead, as is its p95, so one that always answers some 4xx stops being listed.
Below this, a router's 4xx share isn't judged, on its page or as an anomaly: one 404 of three requests is 33%. 5xx is always judged.
{% endif %} {% macro days(label, field, key, hint) %}
{% if hint %}
{{ hint }}
{% endif %}
{% endmacro %}

Metric anomalies

A metric that leaves what is normal for it at that hour is listed under Alerts → Anomalies to review. One nobody marks normal or real within this window, once it has ended, lapses: it leaves the list and is kept as not reviewed, and its hours are never learned.

{{ days("Review window", "anomalies_review_days", "anomalies.review_days", "An anomaly still going never lapses.") }}

Docker lifecycle events

How long container start, stop, die and health changes are kept. Container CPU and memory are kept with the other metrics, under Metric retention below. Applied by the hourly cleanup task — a change takes effect on its next run.

{{ days("Lifecycle events", "docker_events_days", "docker.retention.events_days", "Keep container start/stop/die/health history this long.") }}

Container logs

The host agent tails each container's logs and pushes them here, viewable per container. On by default for every container; storage is bounded by a per-container ring (oldest lines rotate out once the age or size cap is hit, whichever comes first). Applied by the hourly cleanup and on ingest.

Global kill-switch. When off, pushed log lines are dropped and no new logs are stored.
Comma-separated container names whose logs are dropped on ingest (e.g. known-noisy ones).
{{ days("Log retention", "docker_logs_retention_days", "docker.logs.retention_days", "Keep each container's log lines at most this long before rotating them out.") }}
Newest lines are kept up to this size per container; older lines rotate out first.

Metric retention

Bounds how much metric history is stored: hosts (CPU, memory, disk, network, temps, …), containers (CPU and memory), monitors and the other sources. Raw per-sample points are kept for the raw window, then rolled up into hourly figures (average, peak, and the level held) kept for the rollup window. Charts read raw for the recent part of a range and hourly for older data. Applied by the hourly cleanup.

{{ days("Raw metrics", "metrics_raw_days", "metrics.retention.raw_days", "Keep per-sample metrics this long, then roll up to hourly.") }} {{ days("Rolled-up metrics", "metrics_rollup_days", "metrics.retention.rollup_days", "Keep the hourly-averaged series this long for multi-week history.") }}
Takes effect immediately — no restart.
{% endblock %}