#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.
- Every minute each learned key's last 15 minutes is judged against its
hour's normal: above the usual hold x1.25 and by the metric's floor opens
an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
to review; the host page shows its own; the weekly report lists the
week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
in backups like alert history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A metric registered with learn_normal() gets, for every resource reporting
it, a usual hold (the 90th percentile across past days of each hour's p95)
and a usual peak for each slot:
- an hour of the day once 5 days of history exist;
- an hour of the week once history reaches back 3 weeks, so weekly jobs are
learned too.
Each slot takes in the hours either side, so a job that runs a little late
stays normal. One SQL statement relearns everything hourly into
metric_norms, from the rollup plus the raw window not rolled up yet;
norms_at() reads the slot a moment falls in. Nothing is registered yet:
Traefik (#5741) and hosts (#5742) opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scraper folded traefik_service_requests_total{code} into two shares and
dropped the code, so a problem couldn't say whether it was scanners (404), a
login wall (401/403) or the app down (502/503/504). Each scrape now stores the
responses per router and code that occurred (traefik_status_codes, migration
traefik_006), pruned with the router metrics and left out of backups (System
180 ruling: Traefik history is not backed up). A problem card lists its codes
over the last 15 minutes, most first, or says none occurred, or that codes
aren't recorded yet.
A counter series new since the previous scrape now counts from 0 instead of
being skipped: Traefik creates a router's 502 series at its first 502, so a
first burst of 5xx read as 0% until it happened again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus gains a "Widget templates" tab beside Targets. It lists the
built-in templates and an admin's own, any that no longer pass the checker
first. A template opens as its YAML beside a live preview drawn over a
chosen target exactly as the dashboard widget draws it (routes.draw, now
shared by both). An admin writes or pastes a new one, reads a .yaml upload
into the editor to look over, duplicates any template under a free name,
edits and deletes their own; anyone can export the YAML. Built-in
templates are read-only.
Custom templates are rows in prometheus_widget_templates (migration
prometheus_002_templates), kept in backups, keyed "custom:<id>" so a
rename keeps its widgets. Saving needs a template that passes the checker
and a name no other template has. Create, update and delete are audited
and link to the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backup file (#5672 "Backup file: pg_dump of config + audit + alert
history, the secret key and a manifest"): steward-<UTC>.tar holding a
pg_dump --format=custom with metrics', check results' and Docker/Traefik/
UniFi rows left out, the secret key, and a manifest. Every table is named
KEPT or EXCLUDED in steward/backup/contents.py, and a guard test fails on a
new one. The image gains postgresql-client-17 (approved, System 180).
The restore (#5674 "Restore: `steward restore` and a CI test that a dump
brings Steward back"): pg_restore in its three sections. Between data and
constraints, a kept row pointing at a left-out row gets what its foreign
key's ON DELETE says (alert_events.ansible_run_id is cleared). It refuses
a non-empty database unless --wipe, and puts the backup's key in
/data/secret.key. Migrations run on Steward's next start.
`steward` stays the server command: it is now a click group that serves
when given no subcommand.
The publish smoke step backs up the booted image's database, restores it
into a second one, checks a marker setting came back and plugin_metrics
is empty, and starts Steward on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5