e445c83e676392829bcda1cc2d58ebf5e777b85e
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5.5
|
f3433fe3ce |
feat(norms): hosts, containers and monitor response times are graded against their own normal (#5742)
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
9f14620271 |
feat(traefik): routers are graded against their own normal (#5741)
#5741 "Traefik routers are graded against their own normal", milestone 513. - A router's 4xx share and p95 learn their normal (learn_normal in the plugin's setup). Once a router has 5 days of history, a reading over the fixed cutoff but inside its normal is healthy, and the healthy table says it is usual for that router. One out of its normal is a warning that says what is usual, and a metric anomaly. - The 5xx share is a failure: fixed cutoff always, whatever the history or traffic (System 182's ruling). - A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and isn't judged for anomalies. - Core: learn_normal(gate=(metric, least)) judges a key only while its resource's gate metric averages at least `least`; `least` may be a function, so it follows a setting. learn_normal(since=...) learns only from history after a moment: Traefik latency learns from #5755's per-scrape figures only. - The Traefik page lists the routers' metric anomalies of the last 30 days (_anomaly_card.html). The dashboard widget keeps its why to problems. - Thresholds and docs say when the cutoffs give way to a normal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
5a0e150789 |
refactor(docker): container CPU and memory live in the one metric store (#5752)
#5752 "Container CPU and memory history lives in the one metric store; docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does too, and every snapshot in an ingest (backlog included) is recorded at its own time. The container history chart, Busiest Containers, a service's history and the MCP container tools read plugin_metrics under resource "<host>/<container>", rolled up and pruned by the shared metric retention. docker_012_drop_metrics drops both tables. Their history is not copied (lesson #5221 "A data migration's cost is set by the live table"); the store already holds 7 days of container CPU and memory %. Docker retention now prunes only lifecycle events, and Settings -> Thresholds has one "Metric retention" card for every source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
5569f9934e |
feat(anomalies): a metric out of its normal is an anomaly to review (#5740)
#5740 "A metric that departs from its normal is recorded as an anomaly to review, and is findable after it ends", milestone 513. - Every minute each learned key's last 15 minutes is judged against its hour's normal: above the usual hold x1.25 and by the metric's floor opens an occurrence, which ends when the level comes back. - metric_anomalies holds one row per item to review; a repeat while it still waits joins it as an occurrence (metric_anomaly_spans). - An ended row nobody reviews within anomalies.review_days (14, Settings -> Thresholds) lapses; an ongoing one never does. - Learning skips every hour an occurrence covers unless its anomaly was marked normal: no normal is learned without review. - Alerts -> Anomalies lists those to review first, then the reviewed and lapsed under day dividers; marking is operator/admin and audited. - The Overview widget and get_overview warn of open ones and count those to review; the host page shows its own; the weekly report lists the week's and the lapsed; MCP list_anomalies and review_anomaly. - Reviewed and lapsed rows are pruned at data.retention_days, and kept in backups like alert history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
24700ee608 |
feat(settings): Settings brought into line with the rest of Steward (milestone 493)
The last page of #5214's walk. Settings still described Steward as it was before monitors were unified and SNMP was rebuilt, drew its tabs below the breadcrumb, and showed the report schedule in UTC. - Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in each page's section_tabs block, above the trail, as Ansible's are. The breadcrumb guard also fails any page that {% include %}s a tab strip. - Report time (#5378): reports.schedule_tz keeps the zone of whoever saved the schedule; check_and_send compares in it (report_due), and the form shows the day and hour on the reader's clock (schedule_in) under their zone's name. "Last sent" uses |when. - Wording (#5379): the report contents list what the report holds (monitors, not ping-enabled hosts); "Ping latency" is "Response time" and says what it grades; plugin detail says enabling loads now and disabling needs a restart; the monitor interval says it applies on the next check (the scheduler reads it every pass); the dead list-config branch is gone. - Traefik (#5380): the 5xx and 4xx cutoffs are settings (traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik routers" group on Thresholds while the plugin runs, read by router_state through error_cutoffs(). The Traefik page has a Thresholds button for admins, to /settings/thresholds/#traefik. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
96c1ea9407 |
feat(host_agent): how far each host's clock is from Steward's (milestone 490)
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the moment the post arrived: clock_offset_s / clock_checked_at on the registration (host_agent_003), and a clock_off_secs metric (either way) that alert rules can watch. Replaces the old skew log, which measured the newest sample and was off by up to an interval. - Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical. - Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning. - Hosts list: names hosts whose clock is off; when every reporting host is off by about the same amount, says Steward's own clock is the likely one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
414c942cbf |
feat(plugins): Traefik problems first, UniFi/SNMP/fleet on shared shapes, audit names its entities, templates on tokens (#5126)
Traefik: routers are graded from their latest scrape (router_state: crit on any real 5xx share, warn on p95 past the latency critical cutoff or a high 4xx share). Failing and slow routers lead as cards that say why; healthy ones fold into one table that stays open or shut across polls. Proxy counts are stat tiles; certificates are graded by Settings -> Thresholds (tls_days). UniFi: WAN health is a stat strip (latency graded by the thresholds), devices carry status dots, event categories and firewall actions are badges, the alarm count reuses .count-badge (renamed from .nav-count). SNMP: each device wears a status badge and its readings are stat tiles. Agent fleet: the table comes first, with a Status column (Reporting / Silent / Never reported, by the host page's stale_after_seconds), silent agents first, hosts linked, last seen as |ago. Audit: entries name their entity (one query per type, steward/audit/ service.py) and link it; something deleted since keeps the name its entry recorded and says "(deleted)". Tokens: charts read colours through chart-colors.js (stewardColor resolves a token to the rgb the browser computed; Chart.js defaults take the page's text and border colours), so host, history, Traefik and SNMP charts drop their hex. Template hex colours reach 0 (HEX_CEILING = 0), the last rgba pills go, and heavy weights (600/700/bold, 60 of them) become 500, the house pair being 400/500; a guard keeps templates there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
e864cd4b6c |
feat(ui): share view on steward.css; breadcrumbs everywhere they belong; focus ring; tables scroll in their card (#5095)
#5095 "Share view on the shared stylesheet; breadcrumbs, focus-visible and wide tables". - dashboard/share_view.html loads theme.css and steward.css; its own copy of the old purple palette is gone. Only the read-only frame and the x/y/w/h grid stay inline, written in tokens. HEX_CEILING 75 -> 58. - Breadcrumbs on settings/general, settings/thresholds, the dashboard share page and share-link-created. tests/test_breadcrumbs.py fails on any page extending base.html without a crumbs() trail, unless it is on the named list of section top pages, logged-out and error pages (rule 134). - :focus-visible draws --fs-focus-ring on links, buttons, summaries, inputs and anything with tabindex. - Any card holding a .table scrolls sideways (card:has(table.table)), so a wide table no longer widens the page at phone width; .card-flush scrolls too, main gets min-width:0, and .table-wrap covers a table outside a card. - tests/test_share_view.py renders the share view: shared sheets, no hex colour, escaped name, widgets loading through the share token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
96126b1c83 |
feat(auth): forged browser requests are refused (#5031)
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token on every non-GET, HTMX header". A page on another site could make a logged-in browser POST to Steward with the session cookie attached. - steward/auth/csrf.py: a per-session token, checked in one before_request for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field csrf_token, constant-time compare). The bearer-authenticated host-agent ingest is exempt; /mcp never reaches Quart. - Every POST form renders {{ csrf_field() }}; base.html carries the token in a meta tag and adds it to every HTMX request; the dashboard's fetch() calls send it. - The session cookie is SameSite=Lax; login issues a fresh token. - PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set when Settings → General was saved). - tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption, and a guard that every POST form in the templates carries the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
21338dbcd9 |
feat(mcp): docker and host-agent tools; TLS and crash-loop cutoffs in Settings
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". Contributed through each plugin's register_mcp_tools hook, so they exist only while the plugin is enabled. docker (viewer, read-only): - list_containers: filter by host, status, compose project, or problem (restarting, crash_looping, unhealthy, oom_killed, exited_nonzero); problems sort first and each row names what is wrong. - get_container: state, CPU/memory history across the raw/hourly seam, lifecycle events. A unique part of a swarm task name is enough. - list_swarm_services: replicas per node (agent-less placements as ghosts) and the swarm's nodes. - search_container_logs: text (% and _ literal), stream, range, context lines, paging by a full-precision `before` cursor; over-long lines keep both ends and say how much of the middle was cut. - get_docker_disk_usage: bytes per host, reclaimable first. - mcp_overview: failing containers and under-replicated swarm services join get_overview. host_agent: list_agents (reporting / stale / pending, version, OS), and an overview finding for agents registered over 30 minutes ago that never checked in. Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in code) and the crash-loop deaths-per-hour count (3), which get_overview and the docker tools now read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
8f1c8c5cf7 |
feat(docker): container log viewer + Settings controls [M79 step 5]
The user-facing half (rule 27). Per-container log viewer + admin controls, so container logs are something the operator can actually touch. Viewer (plugins/docker): - /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive text search; newest-first so the latest lines survive the poll's scroll reset; empty/loading states; a "View logs" link from the container detail page. Jinja auto-escapes log content (no markup injection). Settings (Thresholds & Retention tab): - global on/off toggle, comma-separated exclude list, retention days + max MB per container — DB-backed, no restart. The toggle + exclude are enforced authoritatively at ingest (_persist_logs drops disabled/excluded lines), since the push model has no channel to tell an agent to stop. Tests: route-defined smoke; template-parse covers the new templates; integration test for the ingest-time toggle + exclude enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
8af297670e |
feat(metrics): roll plugin_metrics up to hourly to bound storage
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
faecac3ec6 |
feat(docker): retention + hourly rollup for metrics/events with Settings windows
Bounds Docker time-series growth (the main scaling concern). New docker_metrics_hourly table + docker_006 migration; a plugin retention module (docker.run_retention capability) rolls raw docker_metrics older than the raw window into hourly averages (idempotent upsert), deletes the rolled raw rows, then prunes stale rollups + lifecycle events. Core cleanup.py drives it each hourly run via the capability (no plugin-model import), reading the three retention windows fresh from settings so changes apply without restart (rule 25). Settings → "Thresholds & Retention" gains a Docker retention card (raw / rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the hour-aligned cutoff/bucketing helpers; integration test exercises the real rollup-average + prune across both windows. Milestone 77 task #941. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
591706bd39 |
feat(settings): configurable monitoring thresholds
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed, live, no restart). New thresholds.* keys + to_thresholds_cfg() + a threshold_style(value, kind) jinja global that reads them; latency reuses the existing ping good/warn keys, uptime is direction-aware (floors). Replace the _macros metric_style/uptime_style macros (now removed) with the global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and the ping page uptime column — all now honor the configured cutoffs. Uptime keeps its green 'good' look when not degraded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |