Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tcp_check swallowed every exception and icmp_check returned on a
non-zero exit without a reason, so every down ICMP/TCP monitor stored
error_msg NULL and the Monitors page and MCP showed it down with no why.
Each failure now names its cause: "No reply within 2 s", "Connection
refused on port N", "No answer on port N within 5 s", "Can't resolve
<name>: ...", the OS error, or ping's own first stderr line. An ICMP
check that fell back to TCP says so, and a ping that outlives its
timeout is killed rather than left running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DNS monitors take an optional Resolver, a server's IP. Set, the check asks
that server directly for A and AAAA over UDP and records how long it took
to answer; empty keeps today's lookup through the container's resolver.
"Does my local DNS still answer for public names" and "does 1.1.1.1" are
now separate monitors. No new dependency.
On a host's page the Ansible card moves above the Docker list, which can
run to dozens of containers and hid it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Monitor page: a response-time chart for the last day with failed checks
marked; Edit / Pause / Delete as labelled buttons; the state tile says
"for 29m" instead of "for at least 29m 25s ago"; the check strip fills
its card; no failures is one line, not an empty box; Settings names the
type, target and the actual global interval, with readable field names.
Add monitor: a line under the fields says what the selected type checks
and when it counts as up; Name says it defaults to the target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Status repeated the Monitors page's counts, check strip and uptime. Monitors
now has one header row and every column Status had: the last 30 checks,
latest response with its trend, uptime over 24h / 7d / 30d (the range picker
goes) and TLS days on every row. /status and /status/rows and the nav link
are removed outright; the dashboard Overview (/status/widget) stays and its
Details link goes to Monitors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors widget opens with a verdict ("1 of 5 monitors down."), then one
row per monitor under a single header: latency and uptime over 24h, 7d and
30d, down monitors first with why and since when, the name linking to the
monitor. Counts follow the type filter. Problems always show; the limit
trims only the healthy rest ("+N more up").
Uptime / SLA (uptime_summary) repeated those numbers per host; migration
0033 moves dashboards onto Monitors. Shared widget styles (.w-verdict,
.w-table, .w-sub, .w-more) carry the widget-shape ruling for the widgets
that follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
first batch.
- Alerts: the Active list reads firing, no data, acknowledged, pending, the
longest-running first, and re-polls itself. An acknowledged alert says how
long it has been breaching (from the alert's events; acknowledging moves
last_transition_at) and, apart from that, who acknowledged it and when.
Rules show what each is doing now; disabled rules sort last. The current
value reads the latest-value table (#5208) instead of plugin_metrics.
- Monitors: a down monitor says "Down 12m · connection refused" in the list.
- Hosts: sorted down, agent gone quiet, pending, healthy. The agent column
shows "—" for a missing reading (was 0%) and a Stale badge with its age.
agent_metrics_by_host is gone; the list reads agent_overview_by_host.
- Schedules: failing first (failed or interrupted run, or a firing that could
not start one), then running, then the rest, paused last; status badge
from the shared states.
- Maintenance windows: active (ending soonest), then upcoming (starting
soonest), with expired folded away; each says when it ends/starts/ended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dispatcher ticked once per MONITORS_POLL_INTERVAL and stamped
last_checked_at when the whole batch finished. With a 60 s tick and a 60 s
monitor the next tick saw 59.x s elapsed and skipped, so the monitor ran
every 120 s; a 30 s monitor could never run faster than the tick.
- The dispatcher ticks every 5 s (DISPATCH_TICK_SECONDS) and reads the
monitors and intervals fresh each tick, so a changed interval applies at
the next one.
- Due is measured from the scheduled start, stamped before the check runs,
with half a tick of slack so tick jitter never costs a whole tick. A
long-overdue monitor runs once.
- Each check runs in its own task and records its own result, so a slow
probe holds up nobody; a monitor still being checked is not started
again, and it is freed in a finally.
- A check past CHECK_DEADLINE_SECONDS (120) is recorded down with "Check did
not finish within 120 s" rather than holding its monitor forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors list now opens with a count strip (down, up with how many
degraded, pending, paused) and sorts down, degraded, pending, up, paused,
by name within each. Degraded is up but worth a look: uptime at or below
its warning floor, or the latest response past its critical cutoff, both
from Settings → Thresholds; a slow-but-tolerable reply is coloured, not
sorted.
A monitor had no page of its own: every link went to the list. Now
/monitors/<id> shows its state and since when, latency, uptime for 24h/7d/
30d, TLS days, the last checks, recent failures with their errors, and its
settings (HTTP header names, never their values). Status links to it, and
adding or editing returns to it.
Adding is its own page (/monitors/new, form.html shared with edit) behind
an "Add monitor" button; a refused add re-renders the form with the error
and what was typed instead of a bare 400 text. Row actions are labelled
icon buttons, type names read Ping/TCP/DNS/HTTP (|monitor_type), and the
heartbeat uses pill-up/pill-down, so HEX_CEILING drops to 56.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5065 "Renames carry their alerts". Alert rules, maintenance-window scopes
and metric history key a thing by (source, name), so a rename used to leave
the rules watching a name nothing reported under, and split the history.
- core/renames.carry_rename moves AlertRule.resource_name,
MaintenanceWindow.scope_resource, plugin_metrics and plugin_metrics_hourly
to the new name in the caller's transaction. A host rename also carries
its parts: host_agent "<host>:…" and docker "<host>/…" (HOST_SOURCES).
Matching is exact (a prefix compare, not LIKE or a collation range).
History left under the new name by something deleted is dropped first
rather than merged in.
- Web monitor edit, MCP update_monitor and web host edit call it; MCP
answers with `carried` (it used to return a warning naming stranded rules).
- A name another monitor of the same type, or another host, already has is
refused on create and rename: two would share one history.
- The rename is audited (monitor.updated / host.renamed) with what moved.
Tests: an integration test renames a firing monitor through the web route
and finds its rule still firing and cleared by a sample on the new name,
its window and raw+hourly history moved and an orphan's history dropped; a
host rename moves agent and container parts but not a longer-named host's
or another source's; a guard that every host_agent resource is "<host>" or
"<host>:…".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".
The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
object of strings, a key the type does not take is named. The forms now
answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
metrics_for, rule_fields (source, resource and metric from the same
choices the rule form offers), add_rule, acknowledge, window_fields.
Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Twelve read-only viewer tools:
- get_overview: what is wrong now, ranked critical-then-oldest. Failing
monitors with their error and when the run of failures began, firing,
pending and still-breaching acknowledged alerts, TLS certificates expiring
within 14 days, agents that stopped reporting, hosts over the configured
CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in
the last 24h, and findings from enabled plugins through a new optional
`mcp_overview(db)` plugin hook. Each finding names the call that shows
its evidence.
- list_hosts, get_host, get_host_metrics
- list_monitors, get_monitor (HTTP header values and URL passwords are
redacted)
- list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows
- query_metrics: the whole metric store, narrowed one argument at a time
(sources > resources > latest values > a series summary)
- list_audit_events
Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s,
bounded lists that say when and how they were cut, series as
min/avg/max/p95/last plus a ≤60-point series, lookup by id or
case-insensitive name with a did-you-mean, and a range argument that
refuses an unknown key instead of reading it as 24h.
New core reads in steward/core/metric_series.py (sources and resources
via loose index scans, latest values via DISTINCT ON, series across the
raw/hourly-rollup seam). monitors/service.py gains monitor_states,
down_since and recent_failures; agent_overview_by_host now carries the
fullest disk and the last sample time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:
- steward/hosts/service.py: uptime, status, monitors, agent metrics,
agent overview (stale threshold passed in, not read from current_app),
CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).
Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
