c2a79c8bb05ecc052d70e8f79b4c71adc47e1fab
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c2a79c8bb0 |
feat(ansible): runs named by what ran where, with ago, duration and one status badge (#5125)
Runs read "update.yml on web-1" (run_title over the scope label, beside describe_scope) on the runs list, the run page's title and breadcrumb, and the run-started panel. The list shows started |ago (full time on hover), |duration (counting "so far" while a run is going) and a dry-run tag. Run statuses join STATES, so the list and the run page use status_badge: queued dim, running info (blue, so work in progress doesn't read as a problem), success ok "Succeeded", failed crit, interrupted warn, cancelled dim. run_started.html leaves its hard-coded colours for tokens; the run page's facts move to the .kv list and its failures to var(--red). Template hex colours: 54 -> 47. Also fixes #5124's sidebar count, which CI run 8389 failed on: the stylesheet guard allows no weight above 500, and .nav-count asked for 600. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4fbe93bdf7 |
feat(alerts): a sentence per alert, a link to what it watches, a count in the sidebar (#5124)
The Alerts page's active list now says each alert in one line around the resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m", with web-1 linking to its host, monitor or container page (or the plugin's page for Traefik, UniFi and SNMP). Values take their unit from the metric name through formatting.metric_value. A silent series reads "No cpu_pct from web-1 for 20m"; an up/down metric says the word. Actions follow the routes' roles: acknowledge, edit, toggle and New rule for operators and admins, Delete for admins only, nothing for viewers. The sidebar shows how many alerts need attention beside Alerts, from an async context processor that skips htmx fragments, logged-out pages and the database-down page, and renders without the count if the query fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6caf737917 |
test: render plugin templates through render_template with the plugin's folder in front (#5122)
Calling app.update_template_context directly failed in CI (run 8386: "missing 1 required positional argument: 'context'"). Rather than reproduce Quart's render path, the helper puts the plugin's template folder ahead of the app's loader for the one render and calls quart.render_template, which supplies the request context as it does for every page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6cf04f57e1 |
feat(docker): needs attention first, stale rows say so, no sparklines on the overview (#5123)
The Docker page opens with a "Needs attention" card: every container
with something wrong, worst problem first — restarting, crash-looping,
unhealthy, OOM-killed, exited non-zero — each with what it does ("is
restarting and keeps dying · exit 1 · since 8m ago"), linked to its page.
It is ranked by the same code as the MCP tools: container_problems,
problem_since and the per-container ranking move from mcp_tools into
service.py (ranked_containers, needs_attention), and mcp_tools' _rows and
findings now read them from there.
Counts follow as stat tiles: needs attention, unhealthy, restarting (or
crash-looping), running, stopped, stale, services, hosts.
A container whose last report is older than the host agent's staleness
window (stale_after_seconds, 180 s by default) carries a Stale badge with
when it last reported, instead of a live-looking green dot.
The per-container CPU and memory history columns are gone from the
overview, and with them one history query per container per refresh; the
container's own page keeps its charts. With nothing left to range over,
the time-range control goes too, and the page now refreshes on the poll
interval. CPU and memory are graded by Settings → Thresholds rather than
50/80 and 70/90 written into the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
2a3af60e99 |
test(hosts): render the host_agent fragments whether or not the plugin is enabled (#5122)
The unit app does not register plugin blueprints, so render_template could not find _host_vitals.html or _host_metrics.html (TemplateNotFound, CI run 8384). tests/_templates.render_plugin_template renders a plugin's template through the app's own Jinja environment and request context, with the plugin's template folder ahead of the app's loader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c19f3c21f |
feat(hosts): one health header, graded by one set of thresholds (#5122)
The host page's vitals strip becomes its health header: Live or Stale, then one sentence — "Memory 93% is past its critical threshold", "Every reading is within its thresholds", or, for a stale host, "No report since 20m ago; the readings below are from then" — and last seen as |ago. CPU, memory, root disk and load per core follow as stat tiles graded by Settings → Thresholds, with their sparklines. The full-metrics page drew its bars and gauges at a hard-coded 70/90 (temperatures 70/85) while every other page used the configured cutoffs (80/90 by default), so a 75 % CPU was amber here and fine everywhere else. bar() now takes the metric kind and colours its .meter-fill through status_class; gauges, temperatures and per-core bars do the same. The host's monitor rows move onto the Monitors list's shapes: a status dot, the name linking to the monitor's page, Ping/TCP/DNS/HTTP, pill-up/ pill-down, latency and uptime through status_class, labelled icon buttons. HEX_CEILING drops to 54. New formatting.worst_reading(readings, thresholds) picks the reading furthest past its cutoff, the first of equals winning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
629c466fd9 |
test(monitors): take "12 minutes ago" from the render's own clock (#5121)
NOW is fixed when the module is imported; by the time the detail page rendered in CI seven seconds had passed, and |ago rightly said "12m 7s ago". The page was right; the test's clock was stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e595bd922a |
feat(monitors): problems first, a page per monitor, adding on its own page (#5121)
The Monitors list now opens with a count strip (down, up with how many degraded, pending, paused) and sorts down, degraded, pending, up, paused, by name within each. Degraded is up but worth a look: uptime at or below its warning floor, or the latest response past its critical cutoff, both from Settings → Thresholds; a slow-but-tolerable reply is coloured, not sorted. A monitor had no page of its own: every link went to the list. Now /monitors/<id> shows its state and since when, latency, uptime for 24h/7d/ 30d, TLS days, the last checks, recent failures with their errors, and its settings (HTTP header names, never their values). Status links to it, and adding or editing returns to it. Adding is its own page (/monitors/new, form.html shared with edit) behind an "Add monitor" button; a refused add re-renders the form with the error and what was typed instead of a bare 400 text. Row actions are labelled icon buttons, type names read Ping/TCP/DNS/HTTP (|monitor_type), and the heartbeat uses pill-up/pill-down, so HEX_CEILING drops to 56. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fda54a3c31 |
feat(status): what is wrong first — down monitors and active alerts, and since when (#5120)
#5120 "Status page: what is wrong first". - A "Needs attention" card heads the Status board: each down monitor with how long it has been down, and each active alert (firing, acknowledged, pending, no data) with its condition and how long, linking to its history. - StatusEntry.since / since_at_least: when the current up/down run began, from the recent checks (streak_start); "for at least" when every check we have agrees. - Counts as stat tiles (Down, Active alerts, Pending, Up); names wrap instead of being cut off; dots carry their label. - Uptime and TLS are coloured by status_class against Settings → Thresholds instead of literal cutoffs; heartbeat pills and DOWN/UP use tokens and status_badge rather than hex and bold. - tests/core/test_status_page.py: streak_start cases, and the rows render with the problem card before the list, each since-time, threshold tones and no hex colour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e864cd4b6c |
feat(ui): share view on steward.css; breadcrumbs everywhere they belong; focus ring; tables scroll in their card (#5095)
#5095 "Share view on the shared stylesheet; breadcrumbs, focus-visible and wide tables". - dashboard/share_view.html loads theme.css and steward.css; its own copy of the old purple palette is gone. Only the read-only frame and the x/y/w/h grid stay inline, written in tokens. HEX_CEILING 75 -> 58. - Breadcrumbs on settings/general, settings/thresholds, the dashboard share page and share-link-created. tests/test_breadcrumbs.py fails on any page extending base.html without a crumbs() trail, unless it is on the named list of section top pages, logged-out and error pages (rule 134). - :focus-visible draws --fs-focus-ring on links, buttons, summaries, inputs and anything with tabindex. - Any card holding a .table scrolls sideways (card:has(table.table)), so a wide table no longer widens the page at phone width; .card-flush scrolls too, main gets min-width:0, and .table-wrap covers a table outside a card. - tests/test_share_view.py renders the share view: shared sheets, no hex colour, escaped name, widgets loading through the share token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
477b05b9d0 |
feat(ui): status_badge, status_dot, stat_tile, empty_state, page_header, icon_button and form classes (#5094)
#5094 "Status and layout macros, and form classes". - formatting.STATES: one state -> (tone, label) map for up, down, stale, firing, pending, acknowledged, no_data, ok, warn, crit, resolved and inactive; status_info() reads it (an unmapped state is drawn dim under its own name). required() lets a macro refuse a missing argument. - _macros.html: status_badge and status_dot (a dot always carries its label, sr-only unless shown), stat_tile, empty_state, page_header (actions via a call block) and icon_button, which raises without a label. - steward.css: tone classes (badge-/dot-/stat- ok, warn, crit, info, dim), .sr-only, .page-header, .stat-tile, .empty-state, .btn-icon and the form classes .field, .field-label, .field-hint and .input. Info text is a 40% mix (--blue): the house --fs-info-fg falls to 4.18:1 on its tint over a hovered row, 40% holds 5.09:1. - First users: the Alerts list, history and maintenance pages (headers, state badges, empty states) and the alert forms' hints; the inline inp/lbl style sets on the monitor and host forms become .input / .field-label. - tests/core/test_macros.py renders each macro through the app environment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f47fc23265 |
fix(ui): delete prompts use data-confirm, so a quote in a name no longer runs script (#5099)
#5099 "A name containing a quote runs script when an admin clicks Delete". Inline handlers like onclick="return confirm('Delete {{ rule.name }}?')" are HTML-decoded before the JavaScript is parsed, so autoescaping did not stop a name such as x');fetch(…);// from running. - Every prompt (22 handlers, plus the docker prune and token revoke ones) is now data-confirm="…", read through dataset by one submit listener in base.html; nothing rendered is parsed as JavaScript. - The remaining inline handlers take integers only (|int), and the time range buttons pass this.dataset.range. - tests/test_inline_handlers.py fails on any {{ }} in an inline handler that is not |int, with a self-test; tests/_templates.py is the one template walker the template guards now share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
737d699308 |
fix(ci): integration finds its own Postgres when another repo's job shares the runner (#5104)
#5104 "CI integration lane connects to another repo's Postgres when two integration jobs share the runner". Run 8358 took task 31950's Postgres (another repo's integration job) with `--filter name=integration | head -n1` and every test failed to connect. Read this job's GITEA-ACTIONS-TASK-<n>- prefix from its own container and require exactly one Postgres under it. (Docker ORs repeated name= filters, so the old one is dropped, not kept.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bf08169176 |
feat(ui): shared |bytes |rate |duration |ago filters and status_class from Settings → Thresholds (#5093)
#5093 "Shared format filters and status_class driven by Settings → Thresholds". - steward/core/formatting.py: human_bytes (binary units), human_rate, human_duration (two largest units), ago (a <time> titled with the absolute UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown, which raises on a kind the thresholds don't name. - app.py registers them as the bytes, rate, duration and ago filters and the status_class global; threshold_style is now graded by status_class (and caps its weight at 500). - Replaces the copies: docker's _human_bytes/_human_uptime, core alerts' _duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi byte-rate arithmetic, the swarm-failures ago macro and two uptime spans. - Tests: a table per helper, and every template compiled against the app's filters, so an unknown filter fails CI rather than a page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
65411d3ce8 |
feat(ui): base stylesheet on the Steward design-system tokens; readable status colours (#5092)
#5092 "Tokens". base.html's inline palette was its own: white text on the gold button (1.81:1), a red at 3.26:1 on a card, a warning yellow at hue 42 beside the gold accent at 46, a lavender page title, serif labels at 13px and weights up to 700. - static/theme.css is design system 6's generated sheet (FabledSword + the Steward accent). static/steward.css is base.html's stylesheet moved out, with :root now aliases onto those tokens (--bg, --text-muted, --green…) so every template keeps rendering; it states no colour of its own. - Status text colours are Steward overrides recorded in the design system (tokens 182-184): success #5FB87A, warning #E08A3C, error #EC7A6C. The house -fg mixes read "up" as grey hint text (dE 8.2) and warning as the accent (dE 11.5); these are >= 4.5:1 on page, card, hover and their own 15-25% tints. --orange is now the midpoint of warning and error. - Primary button and active range button: dark text on gold (7.82:1). Danger button on the house destructive fill. Badges and alerts take the -fg colour on a tint; the gold badge uses --fs-accent-fg. - Type: Inter body, Fraunces only for page titles and h1/h2, JetBrains Mono for code; section and widget titles become the tiny caps label in the body face; weights capped at 500. - Removed: the lavender title and share-view gradient, #606080, and the undefined --card-bg, --accent-faint, --accent-bg, --bg-alt, --border-low and --row-alt (now real tokens); the dead settings/index.html. - Guards (tests/test_style_tokens.py): every var(--x) used is defined; steward.css states no colour or weight over 500; hex colours in templates may only go down (ceiling 75, from 81 before this change), each with a self-test that it can fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bd28d3b0e7 |
feat(ui): serve htmx, Chart.js, Gridstack and the house fonts from Steward itself (#5091)
#5091 "Local assets". base.html, the dashboard and the share view loaded htmx and Chart.js from unpkg/jsdelivr, Gridstack from jsdelivr and their fonts from Google. On an instance with no outbound network (rule 164) every page lost htmx, and so every button and poll; and each page load told those hosts about the visitor (rule 155). - steward/static/vendor: htmx 2.0.4, Chart.js 4.4.4, Gridstack 12.6.0 (the versions the pages already used), with their licences (0BSD, MIT, MIT). - steward/static/fonts: Inter, Fraunces and JetBrains Mono, the design system's faces (SIL OFL, licences beside them), every unicode-range subset, with fonts.css. The serif face is now Fraunces in place of EB Garamond (base) and Libertinus Serif (share view); applying Inter and the type scale is the tokens step. - Templates use url_for('static'); /static already passes every gate. - Guard (tests/test_no_remote_assets.py): no template or stylesheet loads a script, stylesheet, font or image from another origin (comments stripped, example URLs in help text allowed), with a self-test that it can fail; every url_for('static') file and every font the sheet names exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc22a35e9d |
feat(alerts): alert history page — transitions, deliveries, mutes and actions (#5066)
#5066 "Alert history page". /alerts/history lists every alert transition in a range (1h–30d, the shared range picker), newest first, all rules or one (?rule=). Each row: from → to, the value against the rule's condition ("no data" for a silent one), email and webhook sent or failed with the error, "muted · <window>" (and "notified when the window ended") taken out of the event note, other notes, and a link to the Ansible run an action started. Range and rule changes swap only the rows (HTMX). Viewer role. The Alerts page links each rule's history and the whole history. The query moves to alert_svc.history, which MCP get_alert_history now uses too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c9f5c049db |
feat(alerts): renames carry their alerts — monitor and host renames move rules, windows and history (#5065)
#5065 "Renames carry their alerts". Alert rules, maintenance-window scopes and metric history key a thing by (source, name), so a rename used to leave the rules watching a name nothing reported under, and split the history. - core/renames.carry_rename moves AlertRule.resource_name, MaintenanceWindow.scope_resource, plugin_metrics and plugin_metrics_hourly to the new name in the caller's transaction. A host rename also carries its parts: host_agent "<host>:…" and docker "<host>/…" (HOST_SOURCES). Matching is exact (a prefix compare, not LIKE or a collation range). History left under the new name by something deleted is dropped first rather than merged in. - Web monitor edit, MCP update_monitor and web host edit call it; MCP answers with `carried` (it used to return a warning naming stranded rules). - A name another monitor of the same type, or another host, already has is refused on create and rename: two would share one history. - The rename is audited (monitor.updated / host.renamed) with what moved. Tests: an integration test renames a firing monitor through the web route and finds its rule still firing and cleared by a sample on the new name, its window and raw+hourly history moved and an orphan's history dropped; a host rename moves agent and container parts but not a longer-named host's or another source's; a guard that every host_agent resource is "<host>" or "<host>:…". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
88d34a3827 |
fix(notifications): a line break in a rule name no longer stops the alert email (#5064)
The rule name goes into the Subject header, and EmailMessage refuses a header holding a line break, so the whole email failed. The web form can't enter one but MCP takes any string. The subject now folds whitespace. Found by the new From-fallback test in CI run 8349. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
35fca57397 |
feat(notifications): JSON-safe webhook template, SMTP timeout and implicit TLS, test-send buttons (#5064)
#5064 "Notifications that arrive": - render_webhook: alert.* is escaped to sit inside a JSON string (a quote or newline in a rule name no longer breaks the body); raw.* is the plain value for |tojson; a missing value reads "no data"; the template runs sandboxed and the body is checked to be JSON before it is sent. - SMTP: 15s timeout; port 465 connects with implicit TLS, otherwise the box means STARTTLS; a From address setting (falls back to the username). - A webhook error no longer carries the URL, which is a secret. - Settings → Notifications: "Send a test email" / "Send a test webhook" send a sample alert with the form's values (unsaved) and show the result and the body that went out; template help names alert.* and raw.*. - Notification audits go through audit_request; the test send is audited as settings.notification.tested, with the channel and outcome only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ec7ce4472 |
test(alerts): wait for the alert's tasks and the run, not every task (#5063)
CI 8343: the wait-for-all-tasks loop also waited on the executor's 5-minute run-output cleanup task (_schedule_cleanup) and hit its 120s deadline. Wait on alerts._background, then poll the run to a final status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49607a2441 |
feat(alerts): dispatch after commit; maintenance mutes, not skips; actions via trigger_run (#5063)
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b009a8dd30 |
test(alerts): fix the silent-finding time and keep cpu_pct first in the host_agent catalog (#5062)
CI 8325: NOW - 10 min is 11:50, not 09:50; and rule_fields' refusal lists the catalog in order, so seconds_since_seen goes last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c368e93e2c |
feat(alerts): silence is alertable — agent offline and no-data (#5062)
Agent offline is an ordinary rule now. The host_agent scheduler records host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or registration time for an agent that never reported), replacing the task that only logged stale agents. The catalog offers the metric, and picking it on a new rule fills "> 180". No data: each rule's state tracks its last sample and its usual gap, smoothed so that an outage counts as at most three gaps. A core task, alert_silence_check, runs every 60s and marks a rule no_data_since once its series has had no sample for three usual gaps (at least 5 minutes; an hour while the cadence is unknown). A never-sampled rule counts from when it started watching. The mark sits beside the state, so a firing alert that goes silent stays firing. Each silence records one event and sends one notification (rule option notify_no_data, default on; muted in maintenance windows). The next sample clears it with a "data resumed" event and notification. Creating, changing, disabling or re-enabling a rule restarts its watch. Migration 0029 adds the columns and seeds each existing rule's last sample and gap from its last day of data, so an upgrade does not flag every rule. Shown on the Alerts page (a NO DATA badge with its start time), in MCP list_alerts, list_alert_rules and get_host (state "no_data", no_data_since), in get_overview findings (alert_no_data, or a firing alert marked silent) and in the weekly report. The rule form has the notify option, and its metric reference now matches what traefik records and lists host_agent. Emails say "no data" and carry the event's note. The metric-catalog guard also reads positional MetricBatch.add arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
da0c6cad17 |
test(alerts): seed the agent registration with SQL, not a second model import (#5060)
Importing plugins.host_agent.models after the app loaded the plugin defines host_agent_registrations twice (run 8310). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0742b7a2b7 |
feat(alerts): the metric catalog is written by what emits it (#5061)
#5061 "Metric catalog belongs to what emits: plugins register their metrics; names match what is recorded". - The static METRIC_CATALOG is gone. metric_catalog() is core's own sources (the four monitor types and "steward", the app's open_fds/open_fds_pct, which were recorded but never offered) plus each loaded plugin's get_alert_metrics() export. A disabled plugin's sources leave the picker. - traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct. The old "error_rate" was never recorded, so a rule on it could not fire. - tests/core/test_metric_catalog.py walks every MetricBatch.add / record_metric call (traefik's tuple loop included) and fails on a recorded metric the catalog lacks; it shows on a synthetic writer that it can fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
58bfc7c361 |
feat(alerts): one evaluation path; host-agent ingest fires rules (#5060)
#5060 "One evaluation path: record_metrics batch API; host-agent ingest evaluates rules". - record_metrics(session, rows) writes metric rows and evaluates the rules that watch them, reading rules, states and maintenance windows once per batch. Each rule sees its key's samples oldest first, so an agent catching up after an outage counts consecutive checks as they happened. - Host-agent ingest now goes through it. Before, it wrote plugin_metrics directly and no host_agent rule could ever fire. The catalog gains the per-mount disk_used_pct ("<host>:<mount>"). - MetricBatch gathers a transaction's samples; the docker ingest, traefik, unifi, snmp, monitor and self-monitor writers record one batch each instead of a rule query per sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d0c792a5 |
build(image): ship the ldap extra so LDAP sign-in works in the image (#5036)
The Settings -> Auth LDAP block had no ldap3 to run on: every LDAP login logged "ldap3 not installed" and failed as "Invalid credentials". Operator chose shipping it over hiding the option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a6c24299a2 |
fix(alerts): acknowledgements hold until resolve; clear after N OK; rule changes reset (#5059)
#5059 "Alert state machine: acknowledgement holds until resolve; N consecutive OK to resolve; editing, disabling or deleting a rule resets it". - The state machine is now next_step(), a pure function of one sample. An acknowledged alert stays acknowledged while breached; before, the next breaching sample re-fired it. Firing and acknowledged alerts resolve after consecutive_ok_required OK checks in a row (new rule column, default 1; rule form, rules list and MCP create/update_alert_rule clear_after_checks). - Changing what a rule watches (source, resource, metric, operator, threshold) or disabling it resets its alert to inactive; a firing or acknowledged one records a resolved event whose new note says why, and nobody is notified. Web and MCP edits share alerts.service.apply_rule_change. - Migration 0028: consecutive_ok_required, consecutive_ok_count, alert_events.note, and metric_value nullable (a reset has no sample). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
faa605cead |
feat(secrets): webhook URL and git tokens sealed; run output masks its secrets (#5037)
#5037 "Secrets at rest and in run output: webhook URL and git token encrypted, askpass via env, secret values scrubbed from logs". - webhook.url joins SECRET_KEYS; existing values are sealed at startup by migrate_plaintext_secrets. Settings -> Notifications no longer echoes the URL: it names the saved host, keeps it when left blank, and has a Remove box. - Git source http_token is sealed inside ansible.sources on save and at startup, opened in get_sources, and counted by the undecryptable-secrets scan as ansible.sources[<name>].http_token. - The askpass script holds no token: git runs it with STEWARD_GIT_TOKEN in its environment and printf prints it as data. - The executor masks every secret the run is handed (become/vault passwords, bootstrap passwords, sensitive run vars, and the per-host agent tokens the host_agent plugin now passes as redact) in each line before it is broadcast, logged or stored. - Bundled "Write agent config" tasks are no_log; a test fails if any bundled task templating steward_token lacks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ddf13110e9 |
test(models): the user column set includes auth_source and external_id (#5036)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f944cdb9b0 |
feat(auth): LDAP and SSO users keep their source; directory TLS is real (#5036)
#5036 "SSO and LDAP: users keep their auth source, no merging into local accounts, is_active checked, LDAP TLS real and filter escaped". - users gain auth_source (local/ldap/oidc) and external_id (migration 0027). A provider account is found by (source, subject) only: the LDAP entry DN or the OIDC sub. A first sign-in whose username or email another account holds is refused with a message naming why; a disabled user stays out. - Local passwords sign in local accounts only. - LDAP: port 636 is LDAPS, otherwise StartTLS before any bind; the server certificate is checked against the system CAs or a CA pasted in Settings -> Auth. The login name is escaped into the filter (RFC 4515), and an empty password is refused before the directory sees it. - OIDC: the ID token must carry this sign-in's nonce and the userinfo sub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b1da073677 |
test(mcp): import only User locally; UserRole comes from the enclosing scope (#5035)
The local import shadowed UserRole for the whole function, so its earlier use read an unassigned name (ruff F823, CI run 8294). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7a3f7869fe |
feat(auth): sessions end when they should; logins are throttled; Secure behind HTTPS (#5035)
#5035 "Sessions: lifetime applied, role and is_active re-read per request, logout is a POST, failed logins throttled". - steward/core/proxy.py: trusted_value / is_https / client_ip read X-Forwarded-Proto and -For only as far as the trusted proxy hops (Settings → General, default 1, 0 = trust none). - The session cookie is Secure, and HSTS is sent, only on requests that arrived over HTTPS by that rule; plain-HTTP and LAN use are unchanged and nothing redirects. - A login lasts Settings → General's session lifetime from when it began (session.permanent + a login_at stamp checked every request — the cookie's own age slides on every response). A deleted or deactivated user is logged out on their next request; a role change applies at once. - Logout is a POST (a form in the sidebar); GET /logout is gone. - Failed logins pause an address after login.max_failures within login.lockout_minutes (Settings → General; defaults 5 in 15 min), keyed by address so nobody can lock a user out by name. Failures are audited as auth.login_failed with the address. - to_access_cfg projects the new settings into app.config at startup and on save, beside the existing projections. - Integration tests seed real users and a login_at, since every request now re-reads the session's user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8dad79f17a |
feat(ansible): runs stay inside their source; controller-side vars are admin-only (#5034)
#5034 "Playbook runs: path and inventory checks in trigger_run, connection vars admin-only, confirm enforced server-side, web actions audited". Ansible runs inside the Steward container, so an operator who could name any playbook path, set ansible_connection=local or ProxyCommand ssh args, or put {{ lookup('pipe', …) }} in a value could run commands on Steward. - steward/ansible/guards.py: check_playbook (the source must list it), check_inventory_scope (steward:* or an inventory file the source has), check_vars_for_role (ansible_* beyond host/port/user/password/become, and any Jinja in a value, are admin-only), changed_vars (an edit is held to the role only for what it changes). - trigger_run applies the path checks, so every run path — web, MCP, alerts, schedules, plugins — gets them. - Variables are checked by role on the web run form, host-page runs, schedule create/update, inventory group/target create/update, and MCP run_playbook / create_schedule / update_schedule. - A steward:confirm playbook needs the confirmation ticked on the server side for a real (non-check) web run, as MCP already required. - Audited: runs, cancel, schedule create/update/enable/disable/delete/ run-now, playbook create/update/delete, inventory group and target create/update/delete (variable names only, never values). One audit_request helper serves the web routes. - The CSRF form guard and the route guard now each test that they fail on a synthetic broken input (rule: a guard must be able to fail). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6a1bc437c9 |
feat(auth): share links open only their own widgets; unmarked routes are closed (#5033)
#5033 "Share links open only their own dashboard; every route without a role is on an explicit allow-list". Any live share token in ?s= made the caller a viewer on every viewer route — host pages, container logs, any dashboard — and the host-agent widget routes had no role check at all. - steward/dashboard/share.py: a share token opens, read-only, exactly the widget fragment URLs its own dashboard loads — same path, same configured parameters; only the time window (hours/range) may change. - require_role marks its view; public_route marks the deliberately open ones (login, setup, health, the share page, agent ingest, install.sh, agent.py) with the reason. A before_request refuses any view with neither, so a route that forgets its decorator is closed, not open. - /plugins/host_agent/widget and /widget/history now require viewer. - The history widget's range buttons keep the rest of the query (and so a share link's token) instead of rebuilding it; the uptime widget's "Full SLA" link shows to logged-in users, not to share viewers. - tests/test_route_access.py: a source scan that fails on any unmarked route in core or bundled plugins, the default-deny refusal, and the share matcher's cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d188a673a6 |
feat(plugins): installs fetch only what the catalog names, verified (#5032)
#5032 "Plugin install takes its URL and checksum from the catalog, requires the checksum, and extracts only inside the plugin folder". The install form carried the download URL and checksum as hidden fields, so any admin POST could make Steward fetch and import code from anywhere, unchecked, and a zip member like ../x.py was written wherever it pointed. - The route takes only the plugin name and reads download_url and checksum_sha256 from the configured catalogs; a name no repository lists is refused. Installs are audited (plugin.install / plugin.install_failed). - The checksum is required; the name must be a plain identifier. - extract_plugin refuses any member that resolves outside the plugin folder, caps the download at 50 MB and the unpacked size at 200 MB. - The zip unpacks into a staging folder and is swapped in whole: a refused or failed install leaves the installed copy untouched, and an update no longer leaves the old version's deleted files behind. - The catalog shows "Can't install: the catalog lists no checksum" in place of the button for an entry without one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
96126b1c83 |
feat(auth): forged browser requests are refused (#5031)
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token on every non-GET, HTMX header". A page on another site could make a logged-in browser POST to Steward with the session cookie attached. - steward/auth/csrf.py: a per-session token, checked in one before_request for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field csrf_token, constant-time compare). The bearer-authenticated host-agent ingest is exempt; /mcp never reaches Quart. - Every POST form renders {{ csrf_field() }}; base.html carries the token in a meta tag and adds it to every HTMX request; the dashboard's fetch() calls send it. - The session cookie is SameSite=Lax; login issues a fresh token. - PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set when Settings → General was saved). - tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption, and a guard that every POST form in the templates carries the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d9ac2b46a |
feat(mcp): built-in playbooks get steward_url from Steward itself
#5015 "Built-in host-agent playbooks make an MCP caller supply steward_url" - run_playbook, create_schedule and set_alert_action fill a steward-builtin playbook's steward_url when the caller leaves it out or empty, from Settings → General → Public base URL, else the scheme and host the MCP request arrived on — the same source the web actions use. - run_playbook's reply names what it filled (filled_by_steward), and get_playbook shows that value as the variable's default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
06e89308a8 |
feat(ansible): a run's scope reads as the target or group it names
#5014 "Ansible run scope is shown as a raw id" - inventory_gen.describe_scope / scope_labels turn "steward:target:<uuid>" into "target web-1", "steward:group:<uuid>" into "group Docker Swarm", and a deleted one into "target abcdef01 (deleted)". - MCP run and schedule rows carry scope_label beside the stored scope; the ansible.run.started audit detail records it too. - The web run page and schedules table show the name, with the stored scope as the tooltip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
30cb375079 |
test(docker): host-wide log search includes the removed container's line
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f72ed97034 |
feat(mcp): get_host groups a host's parts by kind and says what it cut
#5012 "get_host cuts `parts` without saying so" - The agent's sub-resources come back as mounts (fullest first), interfaces and disk_io (busiest first) and temperatures (hottest first), each bounded on its own with total, truncated and a `more` that names parts_limit and query_metrics. - Per-core CPU is a summary (count, min/avg/max, three busiest), so 38 cores no longer push every interface out of the reply. - New argument parts_limit (default 20, max 200). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
190a5d1e80 |
feat(docker): a removed container's logs and events stay reachable by name
#5010 "Logs of a removed container cannot be reached by name" - get_container and search_container_logs fall back to the names Steward still holds logs or events for when no live container matches: the exact name from logs and events, or a part of it that matches one removed container's events. - get_container answers with removed: true, last_seen_at, its history and events; search_container_logs marks container_removed and notes when it was last seen, and says when that is older than the range. - A miss now says the container may have been removed, that Swarm replaces a dead task under a new name, and where to look instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e9341c5639 |
feat(docker): overview findings say when the problem began
#5013 "Overview findings for containers and Swarm services carry since=null" - Container problems are dated from their own evidence: a crash or restart loop from its first death in the hour, an OOM kill or error exit from the death that ended it, a failing healthcheck from its last turn to unhealthy. list_containers and get_container carry problem_since / problem_age_s. - Swarm services carry trouble_since: a crash loop's first failure in the hour, a short service's oldest troubled task in each failing slot, a recovered service's first failure in 24h. list_swarm_services shows it. - get_overview findings from the docker plugin now have since and age_s, so they rank by how long they have stood. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bac2859455 |
feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3474cdff46 |
feat(mcp): connect card, MCP docs, and an end-to-end roles/audit test over HTTP
MCP-8 #4972 "Connect card, docs, and end-to-end integration test". - AI access page: a "Connect a client" card with the /mcp URL (from Settings → General → Public base URL, else the request host) and recipes for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP clients, plus the reverse-proxy note. The token-created page shows the same recipes with the new token filled in, ready to paste. - docs/core/mcp.md: connecting, the reverse-proxy requirements, the security model (token ceiling, default deny, check-mode default, secrets, audit), tools by role, response conventions, adding tools. docs/architecture.md describes the /mcp dispatch outside the blueprint pipeline; docs/reference/code-map.md gains an MCP section. - tests/integration/test_mcp.py: over HTTP, a viewer token lists no write tools and is refused one by name; an operator token reads get_overview and its maintenance-window create/end land in the audit log as "<user> via MCP" with the token id. (Revoked → 401 was already covered.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9abe0beecc |
feat(mcp): monitoring-config write tools — monitors, alert rules, acks, maintenance
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks, maintenance". The validation the forms did inline (or not at all) moves to the services, and both the routes and the tools call it, so bad input is refused the same way with the same reason: - monitors/service: monitor_fields, build_config, normalise_target. Port 1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an object of strings, a key the type does not take is named. The forms now answer 400 with the reason instead of redirecting silently or a 500. - alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for, metrics_for, rule_fields (source, resource and metric from the same choices the rule form offers), add_rule, acknowledge, window_fields. Tools: create_monitor, update_monitor (config merged over the stored one; a rename warns about rules still watching the old name), set_monitor_enabled, create_alert_rule, update_alert_rule, acknowledge_alert, create_maintenance_window (duration or end), end_maintenance_window (ends one in progress, cancels an upcoming one) — operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options — viewer. Every write is audited as "<user> via MCP". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2ebfdc8cdf |
test(docker): the plugin registers prune_docker as an admin write tool
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
680a6bf3b6 |
feat(mcp): automation tools — playbooks, runs, schedules, alert actions, docker prune
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions". Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory (variable names only), list_ansible_runs, get_ansible_run (log tail + grep), list_schedules, wait_for_run (polls, 120s cap). Operator: run_playbook (check mode unless check=false; a steward:confirm playbook applies only with confirmed=true; secret variables reach the run but are never stored, audited or returned), cancel_ansible_run, and create/update/delete_schedule (secrets refused, check by default). Admin: set_alert_action, and the docker plugin's prune_docker, which previews from the disk report unless check=false (Ansible's check mode skips the prune commands, so a check run would show nothing). Every write is audited as "<user> via MCP" with the token id; the audit tool's user filter includes those rows. prune_extra_vars moves to the docker service so the page and the tool share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
21338dbcd9 |
feat(mcp): docker and host-agent tools; TLS and crash-loop cutoffs in Settings
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". Contributed through each plugin's register_mcp_tools hook, so they exist only while the plugin is enabled. docker (viewer, read-only): - list_containers: filter by host, status, compose project, or problem (restarting, crash_looping, unhealthy, oom_killed, exited_nonzero); problems sort first and each row names what is wrong. - get_container: state, CPU/memory history across the raw/hourly seam, lifecycle events. A unique part of a swarm task name is enough. - list_swarm_services: replicas per node (agent-less placements as ghosts) and the swarm's nodes. - search_container_logs: text (% and _ literal), stream, range, context lines, paging by a full-precision `before` cursor; over-long lines keep both ends and say how much of the middle was cut. - get_docker_disk_usage: bytes per host, reclaimable first. - mcp_overview: failing containers and under-replicated swarm services join get_overview. host_agent: list_agents (reporting / stale / pending, version, OS), and an overview finding for agents registered over 30 minutes ago that never checked in. Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in code) and the crash-loop deaths-per-hour count (3), which get_overview and the docker tools now read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
41c7509366 |
feat(mcp): diagnostic tools — overview, hosts, monitors, alerts, metrics, audit
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". Twelve read-only viewer tools: - get_overview: what is wrong now, ranked critical-then-oldest. Failing monitors with their error and when the run of failures began, firing, pending and still-breaching acknowledged alerts, TLS certificates expiring within 14 days, agents that stopped reporting, hosts over the configured CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in the last 24h, and findings from enabled plugins through a new optional `mcp_overview(db)` plugin hook. Each finding names the call that shows its evidence. - list_hosts, get_host, get_host_metrics - list_monitors, get_monitor (HTTP header values and URL passwords are redacted) - list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows - query_metrics: the whole metric store, narrowed one argument at a time (sources > resources > latest values > a series summary) - list_audit_events Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s, bounded lists that say when and how they were cut, series as min/avg/max/p95/last plus a ≤60-point series, lookup by id or case-insensitive name with a did-you-mean, and a range argument that refuses an unknown key instead of reading it as 24h. New core reads in steward/core/metric_series.py (sources and resources via loose index scans, latest values via DISTINCT ON, series across the raw/hourly-rollup seam). monitors/service.py gains monitor_states, down_since and recent_failures; agent_overview_by_host now carries the fullest disk and the last sample time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
312ea95bcf |
refactor: move host, docker, alert and monitor queries into service modules
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so the queries they share with the pages now live in plain functions that take an open session: - steward/hosts/service.py: uptime, status, monitors, agent metrics, agent overview (stale threshold passed in, not read from current_app), CPU sparks. - plugins/docker/service.py: bucket, container_history, host_map, list_containers, container_events, query_logs. - steward/alerts/service.py: active_alerts (also used by core/reports), list_rules, list_maintenance_windows. - steward/monitors/service.py: monitor_rows (was routes._rows_data). Routes are repointed and render the same data. One behavior fix rides along: the container-log search now escapes % and _ so they match literally instead of acting as LIKE wildcards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
970adf41e7 |
test(mcp): stay connected until the SSE reply ends; drop cross-context reset
The test client reported http.disconnect as soon as the request body was read, and the SDK's SSE response cancels its stream on disconnect, so tools/list came back 200 with an empty body. Model a real client: wait for the response (with a 15s deadline), then disconnect. The caller fixture reset its ContextVar from sync teardown, a different context from the async test that set it; each async test already runs in its own context copy, so the reset is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3984f639e8 |
feat(mcp): /mcp endpoint with bearer auth and role-gated tools
Second step of milestone 461. An MCP client (Claude Code, Claude Desktop via mcp-remote, Open WebUI) connects to /mcp with a personal API token. - mcp>=2.2 (SDK v2 MCPServer). Mounted by wrapping app.asgi_app, so /mcp never enters Quart's pipeline; stateless streamable HTTP so a redeploy never strands a client on a session id the new process didn't issue. - Auth: Authorization: Bearer stw_… resolves to a Caller (user + effective role) in a ContextVar; 401 with WWW-Authenticate otherwise. The web pipeline's database gate runs first, so a DB restart answers 503 instead of failing inside a tool. - Registry, default-deny: register_tool(fn, role=, read_only=) is the only door; role and read_only are required. Tools above the caller's role, or from a disabled plugin, are neither listed nor callable. Undeclared arguments are refused (not silently dropped); a tool's ValueError reaches the agent as its reason; a DB failure mid-tool says so. - Plugins contribute tools via an optional register_mcp_tools(register) export, all-or-nothing, replaced on hot-reload. Documented in docs/plugins/writing-a-plugin.md. - First tool: whoami. - Tests drive app.asgi_app directly (Quart's test client never reaches /mcp); the integration test mints a real token against Postgres and runs initialize → tools/list → whoami → revoke → 401. Task #4966 "MCP-2 — /mcp transport". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ebc25b99ed |
test(api): rename one-letter role aliases (ruff E741)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1c60fb6b4d |
feat(api): personal API tokens and the AI access page
First step of milestone 461 (MCP server). A token is the credential an MCP client presents as `Authorization: Bearer stw_…` to act as its user. - api_tokens table (0026): sha256 hash only, display prefix, role ceiling, expiry, revocation, throttled last_used_at. - steward/core/api_tokens.py: mint, list, revoke, resolve. Resolution re-reads the user's current role and acts with the lower of that and the token's ceiling, so demoting or disabling a user weakens or kills their tokens with nothing to revoke by hand. - /ai-access/ (sidebar → Automation → AI access): every user manages their own tokens, can't grant above their own role, sees the token once. Session-only: a share-link viewer is refused. Create and revoke are audited. - Integration tests drive the routes against real Postgres so their transactions run for real (#884 escaped CI behind a mocked session). Task #4965 "MCP-1 — Personal API tokens". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0da88c7076 |
fix(plugins): show load-failure reasons on the page the banner links to
The admin banner counts every plugin load failure, but settings/plugins.html only rendered failure reasons inside the plugin card macro -- which iterates discovered plugins. A plugin that failed BECAUSE it has no directory was therefore counted in the banner and shown nowhere on the page the banner links to. That dead end is what the operator hit: "1 plugin failed to load" on the dashboard, nothing wrong on the details page, and no way to clean it up. Folds the reason into the "Configured but not installed" rows rather than adding a competing section, and drives that list from the UNION of stored plugin.* settings and recorded load failures instead of stored settings alone. A failure with no stored row -- a plugin enabled by a DEFAULTS entry, which is exactly what plugin.http was -- would otherwise still be counted and still be invisible. Rows carry `removable`: only a plugin with a real stored row can be cleaned up, so a default-declared one shows the reason and an explanation instead of a Remove button that would delete nothing while the default reasserts it. Rows with a reason read as an error (red, "Failed to load"); rows that merely have leftover config stay a warning. A discovered plugin that failed is excluded -- its own card already shows the reason. Replaces find_orphaned_plugin_names outright rather than adding a second overlapping helper; it had one caller. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
8c50ff242c |
fix(db): ride out a brief database outage mid-request; explain it if it persists
The operator's 500 was NOT the startup gap I attributed it to. They found it by logging in and getting Steward's own error page -- which means create_app had completed and the app was serving, so the gaierror came from a request handler acquiring a connection, not from boot. Neither prior fix covers that: the startup retry never runs, and pool_pre_ping only helps once the database is back, since its replacement connect fails too while the container is gone. Adds a before_request gate that acquires a pooled connection with a short bounded retry (~1.75s over 4 tries) so a database restart is ridden out invisibly, and renders a distinct 503 "database unavailable" page when the budget is exhausted. The request budget is deliberately far shorter than the startup one: nobody watches a container boot, but somebody is watching this page load, and a page that hangs is worse than one that says what is wrong. /health and static are exempt -- a liveness probe must stay answerable while the database is down, or a dependency outage triggers a restart loop. Credential and missing-database errors still propagate rather than being reported as "unavailable", which would send the operator chasing the wrong problem entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
59fece855d |
fix(plugins): remove the phantom http plugin; surface orphaned plugin settings
Operator saw the `http` plugin -- deleted when ping/dns/http were unified into the Monitor entity -- still reported as enabled, with no way to clear it. It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so deleting the app_settings row did nothing: the default reasserted it on the next settings load. That is precisely why no cleanup UI could have fixed it. Rule 22 says the removed subsystem should have taken its setting with it. Purged the surviving references -- the DEFAULTS key, "http" in CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the capabilities blurb still advertising HTTP/uptime as a bundled capability -- plus a migration dropping any stored plugin.http row. Untouched: `http` as a MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial, which is kept deliberately so existing DBs resolve the revision graph. For the general case, cleanup splits by provenance rather than being uniformly automatic or uniformly manual: Bundled plugins ship in the image and version atomically with core, so they cannot be transiently missing -- a plugin.* default with no bundled directory is unambiguously a bug. Guarded by a unit test that fails CI, which is the only part safe to automate. External plugins live in operator-mounted /data/plugins, where absence is ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their config would silently destroy unrecoverable credentials on a transient condition, so Settings now lists them under "Configured but not installed" with an explicit, confirmed, audit-logged Remove. The section hides entirely when empty, and the remove route refuses if the plugin is actually installed. Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering to remove a default-only key would be a lie -- and returns names only, since plugin config can hold credentials and listing orphans never needs their values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
6c9b89390a |
fix(db): survive a database restart and a not-yet-ready database
Two connection-recovery gaps, both surfaced by a Postgres restart that left the app throwing tracebacks while the DB itself was healthy. pool_pre_ping + pool_recycle on the app engine [#2626]: when the database restarts, every connection already in the pool is dead at the socket level. SQLAlchemy only discovered that by failing a real query, so the first operation after a restart errored out on whatever triggered it. pre_ping checks liveness on checkout and swaps the dead connection transparently; pool_recycle caps connection age so a socket stranded by a NAT/conntrack timeout or a Docker network rebuild is retired on a timer instead. wait_for_database() gate at startup [#2627]: create_app touches the DB synchronously (migrations, secret re-encryption, settings load) and assumed it was both resolvable and accepting connections on the first try. Neither holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2) or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting connections"). Both are transient, so retry with capped backoff behind one gate ahead of the first DB touch. Credential and missing-database errors are classified by SQLSTATE and still fail immediately -- waiting cannot fix those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT / database.connect_timeout, default 60s) since it governs reaching the DB and so cannot live in the DB-backed settings. Tests drive the retry loop off a fake clock, so backoff and timeout behaviour are deterministic rather than wall-clock dependent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
4b97bd01ea |
docs: Fabled-Git, not Forgejo, in ci-requirements
The instance has run Gitea since the migration; "Forgejo Actions services:" is Fabled-Git Actions. Prose only — no workflow or path change. Scribe issue #2272. |
||
|
|
8c2d994e4c |
docs(host-agent): document container-log collection [M79 step 6]
Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the agent internals, and a wire-format design point on docker_logs riding in the same push (with server-side toggle/exclude enforcement + the per-container ring). The end-to-end chain (store → rotate → query + ingest enforcement) is already covered by the integration tests added across steps 2–5, so no capstone test is added. AGENT_VERSION bump (1.7.0) landed in step 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
414c8efa98 |
fix(test): avoid autobegin-vs-begin collision in the log toggle/exclude test [M79 step 5]
test_persist_logs_respects_toggle_and_exclude read via s.execute(SELECT) between two `async with s.begin()` blocks; that SELECT autobegins a transaction, so the next begin() raised "A transaction is already begun". Move both reads after the final begin block (the toggle-off persist stores nothing, so end-state still reflects the exclude phase), and pass a SimpleNamespace stand-in host to sidestep ORM attribute-expiry across the commit boundaries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
8f1c8c5cf7 |
feat(docker): container log viewer + Settings controls [M79 step 5]
The user-facing half (rule 27). Per-container log viewer + admin controls, so container logs are something the operator can actually touch. Viewer (plugins/docker): - /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive text search; newest-first so the latest lines survive the poll's scroll reset; empty/loading states; a "View logs" link from the container detail page. Jinja auto-escapes log content (no markup injection). Settings (Thresholds & Retention tab): - global on/off toggle, comma-separated exclude list, retention days + max MB per container — DB-backed, no restart. The toggle + exclude are enforced authoritatively at ingest (_persist_logs drops disabled/excluded lines), since the push model has no channel to tell an agent to stop. Tests: route-defined smoke; template-parse covers the new templates; integration test for the ingest-time toggle + exclude enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
07a841d91e |
feat(docker): per-container size+age log ring rotation [M79 step 4]
Bound docker_logs growth in the periodic cleanup task (same architecture as the metrics/events retention). run_docker_retention gains logs_retention_days + logs_max_bytes_per_container: it prunes lines past the age window, then keeps only the newest ~cap bytes per (host, container) via a window-function ring (exclusive-prefix sum, so the newest line always survives even if it alone exceeds the cap). Containers rotate independently. - settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/ max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB - cleanup.py reads the two windows fresh each run (rule 25, no restart) - integration rotate test: age prune + per-container byte cap + isolation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
a8de3570fe |
feat(docker): persist pushed container logs into docker_logs [M79 step 3]
Ingest side of container logs. The host_agent ingest route extracts sample["docker_logs"] as time-series batches (append every line, each carrying its own Docker ts with recorded_at as fallback) and hands them to the docker capability alongside the existing container/swarm/disk data — still under the begin_nested SAVEPOINT so a logs failure can't sink host metrics. - ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters records (drops the _steward truncation marker, malformed + lineless records, normalises unknown stream → stdout); persist_host_docker gains a logs= param - routes.py: accumulate docker_log_batches, add to the guard + capability call - unit test for _log_rows shaping/filtering; integration push→store→query test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
45565b2c01 |
feat(docker): docker_logs table + docker_009 migration [M79 step 2]
Dedicated host-scoped table for pushed container log lines (one row per line), chained after docker_008. host_id FK CASCADE; twin indexes — (host_id, container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring docker_events. Integration schema-shape test asserts columns + both indexes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
c95194747d |
feat(host-agent): collect container logs (incremental push, byte-capped) [M79 step 1]
Agent-side of Docker container-log collection (milestone 79). Each interval the agent fetches new log lines per running container over the local Docker socket, demuxes the multiplexed stream, and folds sample["docker_logs"] into the same push as metrics — no inbound channel needed. - since-cursor per container kept in rate-state; a container's first interval seeds from a short tail, then fetches incrementally via ?since=<cursor> - _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib, TTY/unframed fallback); one row per (stream, ts, line) - per-batch byte cap with a truncation marker; logs are dropped (never buffered) across a backoff so an outage keeps metrics but not stale logs - docker_logs_enabled (default on) + docker_log_exclude per-host config keys - AGENT_VERSION 1.6.0 -> 1.7.0 Server ignores the new sample key until the ingest step lands, so this commit is CI-safe on its own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
|
a9b3b11327 |
feat(docker): admin-gated per-host prune buttons on the image/disk page
M78 step 2. The /disk page surfaced reclaimable space + stopped counts but
said "prune deferred to a later milestone" — this wires the cleanup.
Adds POST /plugins/docker/disk/<host_id>/prune (admin only) that resolves
the host's linked AnsibleTarget → steward:target:<id> scope and fires the
bundled maintenance/docker_prune.yml through the ansible.run_playbook
capability (audited runner; the collection agent stays read-only). Mirrors
the host_agent update route. `target` ∈ {containers, images, system}:
- containers → docker container prune -f
- images → docker image prune -af (prune_all_images)
- system → docker system prune -f (conservative)
The disk() route now resolves each host group's linked target + capability
availability; disk.html renders three confirm-gated buttons per host, shown
only to admins when Ansible is available and the host has a linked target,
otherwise a clear "link a target" / "runner unavailable" hint (rules 24/26/27).
Tests: the prune_target→extra_vars mapping (the -af-only-for-images rule) is
extracted to a pure helper and unit-tested; disk_prune added to the module
smoke; disk.html covered by the existing template-parse test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
|
||
|
|
e8ac99174a |
feat(ansible): parameterize docker_prune with prune_target (containers/images/system)
M78 step 1. The bundled maintenance/docker_prune.yml was a single
`docker system prune -f`; the upcoming per-host prune buttons need a
granular split. Add a `prune_target` extra-var:
- containers → docker container prune -f
- images → docker image prune -f (+ -a when prune_all_images)
- system → docker system prune -f (+ -a / --volumes) — the DEFAULT,
so existing manual/scheduled callers (which set no var)
keep today's behavior.
Reuses the existing bundled playbook rather than adding parallel files;
keeps the description/category/confirm meta so it still self-describes as
a confirm-gated maintenance run. Adds unit coverage: the playbook stays
valid YAML, its meta still discovers, and all three targets are guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
|
||
|
|
01f5805139 |
fix(ci): serialize CI per ref + tag images at build time (fd race #1093)
The publish lane raced on the runner's shared docker daemon when two dev
pushes landed seconds apart: a concurrent run evicted the freshly-built
:<sha> image mid-push, so the post-push `docker tag :<sha> :dev` failed
with "No such image" (a flake — a lone re-run went green).
Two guards, both matching CI-runner's canonical build workflows:
- Workflow-level `concurrency: ci-${{ github.ref }}` with
cancel-in-progress:false serializes runs per ref, so no two publishes
ever share the daemon. false (not true) because rule 46 requires every
push to emit its own immutable :<sha> image — a superseded run must
still finish.
- Build both the :<sha> and moving (:dev/:latest) tags in one
`docker build -t ... -t ...`, then push each. Removes the fragile
post-push `docker tag` of an image a concurrent run could have evicted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
|
||
|
|
524fd1f509 |
feat(ansible): default source/playbook dropdowns to the first item
The schedule and host run forms opened their Source/Playbook <select>s on a "— choose … —" placeholder, forcing an extra click. Default them to the first item and cascade the dependent fields so the initial state is real: - Remove the placeholder options (schedules source+playbook, host source+ playbook, and the shared _playbook_options fragment). - schedules(): pass sel_source (first source, or the edited schedule's) and that source's pre-rendered playbooks so the playbook dropdown starts populated (create mode too, no longer the all-sources union). - Cascade on load/change: the source <select> fires playbook-options then (hx-on::after-settle) re-triggers the playbook <select>, which loads that playbook's variable fields. Host form drives the whole chain from a load trigger; schedule create mode loads vars via the playbook <select>'s own load trigger, while edit mode keeps its server-rendered prefill untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
efc11c1837 |
feat(ansible): structured playbook-variable fields on the schedule form
The Ansible Schedules form only offered a free-form extra-vars textarea,
unlike the browse/host run forms which auto-populate a field per declared
playbook variable (vars:/vars_prompt:) plus the playbook description. Wire
the schedule form to the same shared `_playbook_vars.html` partial:
- Playbook <select> now loads /ansible/playbook-vars?schedule=1 on change
into a #sch-playbook-vars container (mirrors the host detail form).
- Edit mode pre-renders the saved playbook's variables server-side with
their stored values; the extra-vars textarea keeps only leftover
(non-declared) vars.
- Partial gains optional `values` (prefill) and `schedule` flags. In the
schedule context secret vars render disabled ("secrets can't be
scheduled") since they're dropped downstream, and the destructive-confirm
gate is shown but not required.
Backend already parsed var__* fields for schedules, so this is the
front-end/partial parity fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
f40063a74d |
feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi fixes) surfaces early or can't compound, instead of silently killing the app. - Self-fd watchdog (steward/core/self_monitor.py): records open_fds and open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each minute through the normal alert pipeline, so the operator can alert on them via the existing alert-rules UI. Built-in WARNING floor at 80% gives a zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op off Linux. Registered as a core ScheduledTask in app.py. - Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker that skips a tick while a task's prior run is still in flight, so a hung poll can't stack overlapping runs (which amplify per-poll resource/fd use). A skipped task isn't penalised — it retries the next tick after it completes. - fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog metric/warning/degradation paths, and a real-fd canary that hammers tcp_check and asserts /proc/self/fd doesn't grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
0c5a1573da |
fix(unifi): close the cached client before discarding it (fd leak)
The UniFi poller caches a UnifiClient holding a long-lived httpx.AsyncClient (a real socket pool). On a login failure or a mid-poll error it set `_client = None` to force re-auth, but never called close() — orphaning the connection pool until GC. Under a flapping/unreachable controller that leaks a file descriptor every failure tick: the same class of bug as the SNMP engine leak (OSError: [Errno 24] Too many open files). Add _drop_client() that aclose()s the client (best-effort) before nulling the cache, and route both discard paths through it. Regression tests cover both failure paths plus the noop/close-error contracts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
2df5fc94a3 |
fix(snmp): close SnmpEngine after each poll to stop fd leak
A fresh SnmpEngine() was created on every poll_device() call and never closed. pysnmp opens a UDP transport socket per engine and doesn't release it on GC, so each scheduler tick (default 60s, per device) leaked a file descriptor. Over hours of polling the process hit its fd ceiling and the listening socket could no longer accept connections — OSError: [Errno 24] Too many open files on socket.accept(), locking up the app. Wrap the engine in try/finally and release its transport socket via a new _close_engine() helper that probes both pysnmp API shapes (6.2.x lextudio camelCase, canonical 7.x snake_case); all close paths are best-effort so a failed close never breaks the poll loop. Regression tests cover both shapes and the never-raises contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
b1aabb7dfa |
feat(snmp): explicit host binding (host_id / steward_host) with implicit fallback
SNMP devices map onto a Steward host's detail page via _devices_for_host, which previously matched the device's `host` field (the SNMP poll target) against the Host's address/name — a coincidence of strings that breaks when the poll target differs from how the Host is recorded. Add two optional per-device config fields that decouple the binding from the poll target: • host_id — exact Steward Host UUID match (the explicit link) • steward_host — friendly bind by Host name or address (case-insensitive) An explicit binding is exclusive: a device bound to host A never implicitly matches host B by a coincidental address. No explicit binding → existing implicit `host`-string match (fully backward compatible). host_panel passes host.id and badges explicitly-bound devices. plugin.yaml documents the new fields. Tests cover explicit id/name, exclusivity, wrong uuid, and legacy fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
24df8458ea |
feat(docker): swarm-aware container views — collapse replicas, show host, surface agent-less tasks
The container list and dashboard widget listed each swarm task as its own cryptic row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent. Make both views service-centric and cluster-complete. - swarm_view.build_swarm_services(): model-free builder that merges real container rows (collected local-per-node, so host_id = the node a task runs on) with the managers' placement (DockerSwarmService.placement_json). Where placement counts exceed the local rows on a node — i.e. nodes with no agent — it synthesizes "ghost" replicas so the replica list matches the cluster. Non-swarm containers pass through grouped by host. - /rows + rows.html: a Swarm-services panel collapses each service to one block, every replica a host chip (status + cpu); ghosts render dashed "N · no agent". Non-swarm/compose containers keep the per-host table below. New Services stat. - /widget + widget.html: same service collapse with host chips; standalone containers stay grouped by host. - Unit tests for the builder (real+ghost merge, full coverage, partial agent-less, synthesized service, down state). No migration — node_id, placement_json and swarm-node hostnames are already stored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
431f804037 |
fix(docker): move dedup helper to a model-free module (CI green)
The widget-dedup test imported plugins.docker.routes, which re-imports plugins.docker.models after the app already loaded the docker plugin — "Table 'docker_containers' is already defined". Same plugin-loader gotcha the host_agent query helpers avoid. Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM models, so it's safe to import in either environment); routes.py imports it from there and the test targets the model-free module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
e289b6c49f |
fix(docker): dedup widget counts across swarm managers; show failures not stopped
The running count was inflated: swarm-aware agents on every manager list cluster-wide tasks, so the same container (identical container_id) was reported once per manager and the dashboard widgets summed them. The earlier swarm dedup only covered the Swarm topology page — the widgets read raw rows. - _dedup_by_container_id(): collapse rows sharing a non-empty container_id (globally unique, so only true duplicates merge); first occurrence wins under the running-first ordering. Applied in both the containers and resources widgets before counting/limiting. Rows without a container_id (older agents) are kept as-is. - Replaced the static "stopped" tally — not actionable — with "failed (24h)": distinct containers with a die/oom event in the last 24h (DISTINCT so the managers' duplicate events don't double it), rendered red when non-zero. - widget.html empty-state now keys off total_count. - Regression test: same container_id from two managers counts once; id-less rows all kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
0c055ed6fa |
fix(docker): reap vanished containers; clarify the resources widget
The container persister upserted current state keyed (host_id, name) but — unlike the image/swarm/disk persisters — never deleted containers that vanished from the host's latest listing. Every removed one-shot container (CI job runners, buildkit builders, codex jobs) left a permanent "stopped" row, so the dashboard counts ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was a missing reaper. - ingest.py: after the upsert loop, delete this host's containers whose name is notin the newest snapshot (the listing is authoritative — event derivation already treats absence as removal). Mirrors images/swarm/disk. The existing `if not snapshots: return` keeps swarm/disk-only samples from touching containers; an empty container snapshot legitimately means none exist. - widget_resources.html: this widget shows the busiest running containers by CPU. Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar · % value, warn/crit colors) so it's self-explanatory. - Regression test: a container present then absent across snapshots is reaped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
51682f130a |
feat(snmp): surface SNMP device readings on the matching host page
SNMP devices are config-defined by IP/hostname, not Steward Host records, so they had no presence on a host's page. Map them by address and embed a fragment (mirrors the Docker per-host fragment). - _devices_for_host(devices_cfg, address, name): case-insensitive match of a device's configured host to the Steward host's address or name (tolerates non-dict / host-less entries). - Route /plugins/snmp/host/<id>: renders the matched device(s) + latest readings, or nothing when none map (so hosts without SNMP carry no empty card). - snmp/host_panel.html: per-device card (name · address · reachability) with a readings grid (K/M scaling) and a History link to the full device page. - hosts/detail.html embeds it after the Docker fragment, gated on snmp enabled. - Unit test for the matching helper (by address, by name, no-match, blank). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
8af297670e |
feat(metrics): roll plugin_metrics up to hourly to bound storage
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
|
b0d3e83bdd |
feat(hosts): true live vitals strip; agent panel becomes management-only
Realizes the chosen host-summary layout: a thin live vitals bar at the very top, separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD). - New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) + Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing until the agent reports. - The metric computation (latest snapshot + 6h sparkline query + load/core + PSI) moves from host_panel into host_vitals. host_panel slims to management only (reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html drops the gauge row + pressure block, keeping status + lifecycle actions. - hosts/detail.html: vitals strip on top (full width, live), then a 2-col [Monitors | Agent] grid, Ansible full width below, docker fragment last. UI only. Templates parse; plugin-template parse test covers the new fragment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
a35c369dd4 |
polish(ui): sidebar nav a11y + scroll/focus refinements (nav slice 3)
- Only the nav list scrolls when long (min-height:0 + overflow on .side-nav); the brand and user/logout stay pinned top/bottom. - Keyboard focus rings (:focus-visible) on every interactive sidebar element (brand, nav links, user/logout) and the mobile ☰ toggle. - Mobile toggle is now a real toggle: aria-controls/aria-expanded kept in sync, Escape closes the off-canvas sidebar, scrim close stays in sync too. - prefers-reduced-motion disables the sidebar slide + link transitions. - Confirmed no dead top-nav CSS/markup remnants from the old top bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
28c9a4dd2f |
fix(docker): collapse Swarm view per-cluster so multi-manager doesn't duplicate
Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.
Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.
View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
|
10dfd8ffd2 |
fix(host_agent): update full-metrics charts in place to kill refresh flicker
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
|
9e4f1983f8 |
feat(host_agent): lazy-load full-metrics charts + live-poll current state
The full-metrics page rendered once server-side with the history query inline, so the charts blocked first paint and nothing refreshed without a reload (it reads the latest snapshot the server holds — the agent pushes ~every 30s). Split it into a shell + two HTMX fragments: - Shell (host_detail.html): header + shared time-range toggle + two containers; paints instantly. - Current state (/<id>/metrics → _host_metrics.html): identity + gauges + per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence. - History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js, from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so they lazy-load (never block paint), refresh slowly, and follow the range selector. Leak-safe: previous Chart instances are destroyed before re-render. - Range toggle switched from full-reload links to the shared _time_range.html (setTimeRange + rangeChange), so only the fragments refetch. - routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a _split_host_metrics helper. - tests/test_templates_parse.py now also parses plugin templates (these fragments aren't rendered in the unit lane). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
a78af23793 |
refactor(host_agent): extract metric-query helpers to a model-free module
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
|
aff0c36d37 |
perf(host_agent): aggregate metric history in SQL; DISTINCT ON for latest
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention- bounded (core cleanup prunes > data.retention_days, default 90d) and charts top out at 30d, so the cost wasn't growth — it was the read path shipping raw rows to Python. - _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120 buckets) instead of fetching every raw sample (a 30d range was hundreds of thousands of rows) and downsampling in Python. Uses the new (source_module, resource_name, recorded_at) index. - _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY recorded_at DESC — newest row per group in one index-ordered pass, replacing the GROUP-BY-max subquery self-joined back to the whole history. - Integration test validates both against Postgres. Deliberately not a materialized raw→hourly rollup: these query-side changes deliver the speed; a rollup would additionally cut storage and remains a future option if scale demands it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
6d08db0d89 |
feat(hosts): rework host summary layout to use horizontal space
The host detail page stacked three full-width cards (Monitors, Agent, Ansible) left-aligned, using only ~40% of the width with lots of vertical whitespace. Rework toward the operator-chosen "vitals on top + 2-column" layout: - The agent panel (CPU/MEM/DISK/LOAD vitals + sparklines + lifecycle) moves to the top, full width, so a host's vitals lead the page. - Monitors and Ansible sit side by side in a responsive 2-column grid (auto-fit, stacks under ~420px), filling the width. - The Docker per-host fragment moves below, full width. Note: the vitals live in the host_agent fragment (loaded across the plugin boundary) and uptime in the host-detail context, so a single literal top "strip" mixing both isn't clean — the agent card on top is the faithful, contained realization. UI-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
3a54d6d71d |
perf(metrics): index plugin_metrics for host/dashboard reads
plugin_metrics had only a PK on id, so every host-detail, full-metrics, and dashboard-widget load sequentially scanned the entire time-series table — which grows by (sources × resources × sample cadence), so the host views got slower over time (operator-reported "blocked/slow" loads). monitor_results was already indexed, so the uptime aggregation wasn't the bottleneck. Add two composite indexes matching the hot query shapes: - (source_module, resource_name, recorded_at) — history range scans, fleet/ widget queries, and the 'host:%' sub-resource prefix. - (source_module, resource_name, metric_name, recorded_at) — the latest-value- per-metric group-by/self-join. Model __table_args__ + core migration 0023. Integration lane validates the migration via `alembic upgrade head`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
7d144780df |
fix(host_agent): TB/PB byte scaling + rebalance the full-metrics layout
Two issues on the host full-metrics view: - fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of "24.8 TB". Add TB and PB tiers. - Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards stretched to the Temperatures height — a 40-core CPU made one very tall column and left Interfaces/Disks with large empty space. Split them: Interfaces + Disks sit side-by-side at natural height (align-items:start); Temperatures becomes a full-width card whose readings flow into a compact multi-column grid (wide-and-short instead of one tall column). UI-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
2545e8c6ce |
fix(docker): widen memory byte columns to BIGINT (int32 overflow on ingest)
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes / mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM (e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32 range" and the entire docker ingest batch failed — no metrics stored for any host with a large container. The I/O counters and the milestone-77 rollup/disk tables already used BigInteger; this trio (docker_001-era) was missed. - models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes, DockerContainer.mem_limit_bytes → BigInteger. - migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place int4→int8 promotion). - integration regression test: persist an ~8 GB container, assert the current-state and time-series rows round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
c8b6719b37 |
feat(ui): plugin get_nav() hook → sidebar Infrastructure links (slice 2)
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an optional get_nav() plugin export and surface it in the sidebar. - plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant _collect_plugin_nav() (missing hook = fine; raising/malformed = logged & skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in both the startup load path and the hot-reload path. - app.py: inject plugin_nav into the template context, filtered to enabled plugins so a hot-disabled plugin's link can't linger before restart. - base.html: render plugin links under the Infrastructure group, with the same request.path active-state treatment as core links. - docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view. - Tests: collector behavior (collect, missing hook, reload-replace, malformed item, raising hook, sorted output). With docker enabled, "Docker" now appears under Infrastructure → its fleet view. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
95ebdf7045 |
feat(ui): replace flat top nav with a grouped left sidebar (slice 1)
The flat top bar was a set of ungrouped peers and gave plugin data no home. Move to a persistent left sidebar with grouped sections, per the navigation redesign (operator-chosen shell). - base.html: top <nav> → left <aside class="sidebar"> + content column. Groups: Overview (Dashboard, Status), Infrastructure (Hosts; plugin links arrive in slice 2), Monitoring (Monitors, Alerts), Automation (Ansible), Admin (Settings, Audit; admin-only). Brand on top, user/logout at the bottom. - Active link highlighted by request.path prefix (aria-current). - Responsive: sidebar slides off-canvas under 900px via a ☰ toggle + scrim (inline class toggle, no new JS deps). Candle-glow preserved. - Logged-out pages (login/setup) render without the sidebar (gated on session.user_id), content area full-width and centered as before. - Add tests/test_templates_parse.py: syntax-parses every steward template so a broken tag fails the unit lane (only the login page renders there today). Plugin nav links (Docker/SNMP/UniFi/Traefik) come next in slice 2 via a get_nav hook. UI-only; no behavior/route changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
0940dc6972 |
feat(crypto): fail loudly on unpersistable secret key; flag undecryptable secrets
Follow-up to the incident where an unwritable /data made Steward silently mint a fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only discovered when an Ansible run failed. Make both failure modes loud and visible. - config._resolve_secret_key: if a new key must be generated but can't be persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data writable by uid 1000). Also raises a clear error if an existing key file can't be read. First-run on a writable volume still generates + persists normally. - core.settings: detect secrets stored as ciphertext that won't decrypt with the current key (scan_undecryptable_secrets at startup; _is_undecryptable helper). Cached in memory; set_setting discards a key on a fresh write so the banner clears without a restart. - app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject undecryptable_secrets into the template context. - base.html: admin banner naming which secrets to re-enter, each linked to its settings tab. - compose.deploy.yml: document the uid-1000 /data write requirement and the STEWARD_SECRET_KEY option for multi-node Swarm. - Tests: secret-key resolver (reuse existing file, generate when writable, raise when unpersistable) and the undecryptable-secret detection helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
4fc8c96c41 |
fix(snmp): bundle pysnmp in image and port poller to the asyncio HLAPI
The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP polling disabled" on every poll, so polling never worked. Two coupled defects: 1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp) was never bundled even though the plugin is first-party and shipped. 2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep present, polling would have thrown and silently returned nothing. The 5.x line that still has the sync API isn't safe on the image's Python 3.13. Fix: - Dockerfile: install `.[ansible,snmp]`. - poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI, with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd` + async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x `pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency bump can't silently re-break it. - scheduler.py: await poll_device directly; drop the run_in_executor wrapper and the now-unused asyncio import. - Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping, that the poller is a coroutine, and the graceful no-pysnmp path. Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target — live polling against real devices is verified after deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
|
88091936c5 |
fix(ui): unify header/breadcrumb treatment, drop redundant back buttons
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
|
626ba69934 |
test(docker): parse-check templates + smoke routes/_human_bytes (milestone 77 #943)
CI never renders docker templates through the app (the unit-lane app uses testing=True, which skips plugin loading), so a Jinja syntax error could ship green. Add unit tests that parse every docker template, smoke-import the routes module + confirm the #942 view functions exist, and cover the _human_bytes / _human_uptime presentation helpers. Closes the render-regression gap the new UI pages introduced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |