The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.
Agent 1.22.0 (which also carries the backlog order from 41fc807).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A redeploy's new process ticks at once, so its once-a-minute samples never
broke for 90 s and get_anomaly showed none of the day's restarts. The first
self-monitor tick in a process now records restart_gap_secs (time since the
last process's last sample), and steward_gaps shows each one as a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
temp_c_max was recorded on every ingest, but host_agent didn't offer it
to rules, so "host running hot" couldn't be watched. The catalog guard
didn't see the row(...) helper ingest builds its rows with. Now offered:
temp_c_max, temp_c per sensor, and network and disk throughput per host,
interface and disk. A new guard holds every literal row(...) name to the
catalog, except NOT_ALERTABLE, where each exclusion is given a reason.
temp_c_max reads in °C.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.
- One shared reader, _docker_get: each answer has an overall deadline and a
byte cap, not only a per-recv timeout. A trickling or endless body is cut
off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
within 10 s. A note says when older lines were skipped, and a container
whose answer is too big skips ahead to now instead of re-reading its
backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
and agent_cpu_pct, so a runaway agent shows before it goes silent.
Agent 1.15.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.
Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Util01's / ran out of inodes (956 free of 60.8M) with 274G of space
left; every file create failed and Steward's page read healthy at 66%.
- Agent 1.13.0 reports each mount's inodes (statvfs; a filesystem reporting
none, like btrfs, is left without).
- Ingest records inodes_used_pct per mount, and disk_used_pct_worst is the
fullest mount by space or inodes, so existing disk-full rules and the
overview see it.
- The host's health strip, the Hosts widget and the host glance grade / by
whichever is fuller and name inodes when they are ("Disk / 66% · inodes
100%", "Inodes on / ... past its critical threshold").
Hosts pick up the inode figures once their agent is updated to 1.13.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.
- Every minute each learned key's last 15 minutes is judged against its
hour's normal: above the usual hold x1.25 and by the metric's floor opens
an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
to review; the host page shows its own; the weekly report lists the
week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
in backups like alert history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
(an append-only key is enough; retention is the bucket's). Signing is held
to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
chosen on the reader's clock; a missed day is made up, and a failure is
retried an hour later. History of the last 30 runs, audit
(backup.made / backup.failed), and backup_age_secs / backup_bytes under
"Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
blank box keeps it), the schedule, Test the bucket, Back up now, and the
recent runs.
#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.
Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).
- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
writes the outcome to its row and its series through record_metrics, so
alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
_sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
quantiles are skipped. A labelled series is stored under
"<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
by metric with their newest values; a chart per series; the editor with
Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
prometheus), as the rule form does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 495 "Email goes to each account, in its own time zone", on the
operator's ruling (System 156): notifications go to each user, times in
their zone, defaults on, toggled per user or per role.
- users.notify_alerts / notify_report (migration 0036): None follows the
role's default, notify.<kind>.<role> in settings, all on to start.
- mail.pick_recipients: active accounts that want the email and have an
address mail can reach (an SSO placeholder like name@oidc is skipped and
flagged), each in its own zone with no UTC beside it, then the Also send
to addresses in their chosen zone with UTC beside. One copy per address,
all over one SMTP connection.
- Alerts, the weekly report and the test email (to you, in your zone, plus
the typed addresses) send that way. The save-time smtp.tz capture is gone.
- Notifications: a role × email table of defaults, who each email reaches
and who can't be reached, and Also send to with its own zone, which starts
as the first admin's to open the page.
- Account page: your address and your two switches. Users: an Emailed
column, and an admin sets an account's switches on its page.
- The Alerts page counts the accounts it emails. zone_select macro and
formatting.zones_by_region shared by Account and Notifications.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5335 "Emails Steward sends read as designed messages: an HTML alert email
and weekly report, with a plain-text twin".
- core/mail.py renders each email as multipart/alternative from
templates/email/: a table-and-inline-style HTML part and a plain-text
twin carrying the same facts. Colours are the design system's light
theme, resolved from theme.css; nothing is fetched when a mail opens.
- Alert email: state badge, a headline sentence ("Disk used on web01 is
94%."), rule, resource, reading, the condition in words, and the time in
the zone Notifications was saved in with UTC beside it; "View in
Steward" when Steward's address is set. Subject "[Steward] Firing: <rule>
— <resource> 94%". Alerts now carry their operator.
- Weekly report: monitors down, active alerts, the week's alert events,
uptime and busiest Traefik routers as small tables with links. Sent from
the configured From address; the "Generated … UTC" line and the stale
repo footer are gone.
- condition_text() states a condition from a rule's parts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator found "Today 08:03" in place of a date odd on the audit log,
and asked for the recency groups as dividers instead. |when now always
writes the date ("Thu 8 Oct 08:03"); mark_day_groups gives each entry its
group in the reader's zone, and a divider row opens each group. A scrolled
page carries the group the last one ended in, so it is not opened twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- users.time_zone (0034). An account without one takes the browser's zone
on its first page (POST /account/time-zone/adopt, never over a choice made
by hand); the session follows the account on every request.
- /account: who you are and the zone every page writes its times in,
reached from your name in the sidebar.
- formatting.when: 'Today 06:58', 'Yesterday 23:41', 'Tue 6 Oct 14:12',
'6 Oct 2025 14:12', 24-hour, in the reader's zone; |ago's hover is the
exact local moment instead of UTC; tz_label() for column headers.
- Audit log: When (EDT) with exact times, one detail fact a line, on classes.
- tzdata in the image, and the airgapped smoke checks a zone resolves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 3:
- Disk page: each tile says what it holds. "Writable layers" becomes
"Container files" and Build cache says it is safe to clear. Shared is
"Shared with other images" with a tooltip, and "dangling" is gone.
Untagged images show by id instead of "<none>", a capped list says
"The 50 largest of N images", and each host shows "as of" its report.
- Traefik: an expired certificate reads "expired 12 days ago", not "-12
days" (cert_left). The Traffic empty state knows whether the access log
is on: off points to Settings → Plugins → Traefik; on says to check the
mounted log path.
- Moments use |ago with an exact UTC tooltip: UniFi last seen, alarms,
events, speed test and DPI; AI token last used.
- Settings: the Auth tab is "Authentication" like its breadcrumb, plugin
fields read "Metrics URL" not metrics_url (field_label), a source's pull
interval is "every 1h", and the last report time drops the ISO "T".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:
- Audit log: each action reads as a sentence ("paused monitor"), with the
code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
scope by name, spans as spans, and no ids the entity column already gives.
The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
"Fullest disk >= 85%". Readings use metric_value, and history times use
|ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
when it is down/up" for an up/down metric instead of 0.5, and a metric
reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
a scope reads "Every alert on Host agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs read "update.yml on web-1" (run_title over the scope label, beside
describe_scope) on the runs list, the run page's title and breadcrumb, and
the run-started panel. The list shows started |ago (full time on hover),
|duration (counting "so far" while a run is going) and a dry-run tag.
Run statuses join STATES, so the list and the run page use status_badge:
queued dim, running info (blue, so work in progress doesn't read as a
problem), success ok "Succeeded", failed crit, interrupted warn, cancelled
dim. run_started.html leaves its hard-coded colours for tokens; the run
page's facts move to the .kv list and its failures to var(--red). Template
hex colours: 54 -> 47.
Also fixes#5124's sidebar count, which CI run 8389 failed on: the stylesheet
guard allows no weight above 500, and .nav-count asked for 600.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts page's active list now says each alert in one line around the
resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m",
with web-1 linking to its host, monitor or container page (or the plugin's
page for Traefik, UniFi and SNMP). Values take their unit from the metric
name through formatting.metric_value. A silent series reads "No cpu_pct from
web-1 for 20m"; an up/down metric says the word.
Actions follow the routes' roles: acknowledge, edit, toggle and New rule for
operators and admins, Delete for admins only, nothing for viewers.
The sidebar shows how many alerts need attention beside Alerts, from an
async context processor that skips htmx fragments, logged-out pages and the
database-down page, and renders without the count if the query fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page's vitals strip becomes its health header: Live or Stale,
then one sentence — "Memory 93% is past its critical threshold", "Every
reading is within its thresholds", or, for a stale host, "No report since
20m ago; the readings below are from then" — and last seen as |ago. CPU,
memory, root disk and load per core follow as stat tiles graded by
Settings → Thresholds, with their sparklines.
The full-metrics page drew its bars and gauges at a hard-coded 70/90
(temperatures 70/85) while every other page used the configured cutoffs
(80/90 by default), so a 75 % CPU was amber here and fine everywhere else.
bar() now takes the metric kind and colours its .meter-fill through
status_class; gauges, temperatures and per-core bars do the same.
The host's monitor rows move onto the Monitors list's shapes: a status dot,
the name linking to the monitor's page, Ping/TCP/DNS/HTTP, pill-up/
pill-down, latency and uptime through status_class, labelled icon buttons.
HEX_CEILING drops to 54.
New formatting.worst_reading(readings, thresholds) picks the reading
furthest past its cutoff, the first of equals winning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors list now opens with a count strip (down, up with how many
degraded, pending, paused) and sorts down, degraded, pending, up, paused,
by name within each. Degraded is up but worth a look: uptime at or below
its warning floor, or the latest response past its critical cutoff, both
from Settings → Thresholds; a slow-but-tolerable reply is coloured, not
sorted.
A monitor had no page of its own: every link went to the list. Now
/monitors/<id> shows its state and since when, latency, uptime for 24h/7d/
30d, TLS days, the last checks, recent failures with their errors, and its
settings (HTTP header names, never their values). Status links to it, and
adding or editing returns to it.
Adding is its own page (/monitors/new, form.html shared with edit) behind
an "Add monitor" button; a refused add re-renders the form with the error
and what was typed instead of a bare 400 text. Row actions are labelled
icon buttons, type names read Ping/TCP/DNS/HTTP (|monitor_type), and the
heartbeat uses pill-up/pill-down, so HEX_CEILING drops to 56.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5094 "Status and layout macros, and form classes".
- formatting.STATES: one state -> (tone, label) map for up, down, stale,
firing, pending, acknowledged, no_data, ok, warn, crit, resolved and
inactive; status_info() reads it (an unmapped state is drawn dim under its
own name). required() lets a macro refuse a missing argument.
- _macros.html: status_badge and status_dot (a dot always carries its label,
sr-only unless shown), stat_tile, empty_state, page_header (actions via a
call block) and icon_button, which raises without a label.
- steward.css: tone classes (badge-/dot-/stat- ok, warn, crit, info, dim),
.sr-only, .page-header, .stat-tile, .empty-state, .btn-icon and the form
classes .field, .field-label, .field-hint and .input. Info text is a 40%
mix (--blue): the house --fs-info-fg falls to 4.18:1 on its tint over a
hovered row, 40% holds 5.09:1.
- First users: the Alerts list, history and maintenance pages (headers,
state badges, empty states) and the alert forms' hints; the inline inp/lbl
style sets on the monitor and host forms become .input / .field-label.
- tests/core/test_macros.py renders each macro through the app environment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
