#5014 "Ansible run scope is shown as a raw id"
- inventory_gen.describe_scope / scope_labels turn "steward:target:<uuid>"
into "target web-1", "steward:group:<uuid>" into "group Docker Swarm",
and a deleted one into "target abcdef01 (deleted)".
- MCP run and schedule rows carry scope_label beside the stored scope; the
ansible.run.started audit detail records it too.
- The web run page and schedules table show the name, with the stored
scope as the tooltip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5012 "get_host cuts `parts` without saying so"
- The agent's sub-resources come back as mounts (fullest first),
interfaces and disk_io (busiest first) and temperatures (hottest first),
each bounded on its own with total, truncated and a `more` that names
parts_limit and query_metrics.
- Per-core CPU is a summary (count, min/avg/max, three busiest), so 38
cores no longer push every interface out of the reply.
- New argument parts_limit (default 20, max 200).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5010 "Logs of a removed container cannot be reached by name"
- get_container and search_container_logs fall back to the names Steward
still holds logs or events for when no live container matches: the exact
name from logs and events, or a part of it that matches one removed
container's events.
- get_container answers with removed: true, last_seen_at, its history and
events; search_container_logs marks container_removed and notes when it
was last seen, and says when that is older than the range.
- A miss now says the container may have been removed, that Swarm replaces a
dead task under a new name, and where to look instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5013 "Overview findings for containers and Swarm services carry since=null"
- Container problems are dated from their own evidence: a crash or restart
loop from its first death in the hour, an OOM kill or error exit from the
death that ended it, a failing healthcheck from its last turn to unhealthy.
list_containers and get_container carry problem_since / problem_age_s.
- Swarm services carry trouble_since: a crash loop's first failure in the
hour, a short service's oldest troubled task in each failing slot, a
recovered service's first failure in 24h. list_swarm_services shows it.
- get_overview findings from the docker plugin now have since and age_s,
so they rank by how long they have stood.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-8 #4972 "Connect card, docs, and end-to-end integration test".
- AI access page: a "Connect a client" card with the /mcp URL (from
Settings → General → Public base URL, else the request host) and recipes
for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP
clients, plus the reverse-proxy note. The token-created page shows the
same recipes with the new token filled in, ready to paste.
- docs/core/mcp.md: connecting, the reverse-proxy requirements, the
security model (token ceiling, default deny, check-mode default, secrets,
audit), tools by role, response conventions, adding tools.
docs/architecture.md describes the /mcp dispatch outside the blueprint
pipeline; docs/reference/code-map.md gains an MCP section.
- tests/integration/test_mcp.py: over HTTP, a viewer token lists no write
tools and is refused one by name; an operator token reads get_overview
and its maintenance-window create/end land in the audit log as
"<user> via MCP" with the token id. (Revoked → 401 was already covered.)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".
The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
object of strings, a key the type does not take is named. The forms now
answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
metrics_for, rule_fields (source, resource and metric from the same
choices the rule form offers), add_rule, acknowledge, window_fields.
Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions".
Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory
(variable names only), list_ansible_runs, get_ansible_run (log tail + grep),
list_schedules, wait_for_run (polls, 120s cap).
Operator: run_playbook (check mode unless check=false; a steward:confirm
playbook applies only with confirmed=true; secret variables reach the run
but are never stored, audited or returned), cancel_ansible_run, and
create/update/delete_schedule (secrets refused, check by default).
Admin: set_alert_action, and the docker plugin's prune_docker, which
previews from the disk report unless check=false (Ansible's check mode
skips the prune commands, so a check run would show nothing).
Every write is audited as "<user> via MCP" with the token id; the audit
tool's user filter includes those rows. prune_extra_vars moves to the
docker service so the page and the tool share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Twelve read-only viewer tools:
- get_overview: what is wrong now, ranked critical-then-oldest. Failing
monitors with their error and when the run of failures began, firing,
pending and still-breaching acknowledged alerts, TLS certificates expiring
within 14 days, agents that stopped reporting, hosts over the configured
CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in
the last 24h, and findings from enabled plugins through a new optional
`mcp_overview(db)` plugin hook. Each finding names the call that shows
its evidence.
- list_hosts, get_host, get_host_metrics
- list_monitors, get_monitor (HTTP header values and URL passwords are
redacted)
- list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows
- query_metrics: the whole metric store, narrowed one argument at a time
(sources > resources > latest values > a series summary)
- list_audit_events
Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s,
bounded lists that say when and how they were cut, series as
min/avg/max/p95/last plus a ≤60-point series, lookup by id or
case-insensitive name with a did-you-mean, and a range argument that
refuses an unknown key instead of reading it as 24h.
New core reads in steward/core/metric_series.py (sources and resources
via loose index scans, latest values via DISTINCT ON, series across the
raw/hourly-rollup seam). monitors/service.py gains monitor_states,
down_since and recent_failures; agent_overview_by_host now carries the
fullest disk and the last sample time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:
- steward/hosts/service.py: uptime, status, monitors, agent metrics,
agent overview (stale threshold passed in, not read from current_app),
CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).
Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test client reported http.disconnect as soon as the request body was
read, and the SDK's SSE response cancels its stream on disconnect, so
tools/list came back 200 with an empty body. Model a real client: wait
for the response (with a 15s deadline), then disconnect.
The caller fixture reset its ContextVar from sync teardown, a different
context from the async test that set it; each async test already runs in
its own context copy, so the reset is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second step of milestone 461. An MCP client (Claude Code, Claude Desktop
via mcp-remote, Open WebUI) connects to /mcp with a personal API token.
- mcp>=2.2 (SDK v2 MCPServer). Mounted by wrapping app.asgi_app, so /mcp
never enters Quart's pipeline; stateless streamable HTTP so a redeploy
never strands a client on a session id the new process didn't issue.
- Auth: Authorization: Bearer stw_… resolves to a Caller (user +
effective role) in a ContextVar; 401 with WWW-Authenticate otherwise.
The web pipeline's database gate runs first, so a DB restart answers
503 instead of failing inside a tool.
- Registry, default-deny: register_tool(fn, role=, read_only=) is the only
door; role and read_only are required. Tools above the caller's role,
or from a disabled plugin, are neither listed nor callable. Undeclared
arguments are refused (not silently dropped); a tool's ValueError
reaches the agent as its reason; a DB failure mid-tool says so.
- Plugins contribute tools via an optional register_mcp_tools(register)
export, all-or-nothing, replaced on hot-reload. Documented in
docs/plugins/writing-a-plugin.md.
- First tool: whoami.
- Tests drive app.asgi_app directly (Quart's test client never reaches
/mcp); the integration test mints a real token against Postgres and
runs initialize → tools/list → whoami → revoke → 401.
Task #4966 "MCP-2 — /mcp transport".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First step of milestone 461 (MCP server). A token is the credential an
MCP client presents as `Authorization: Bearer stw_…` to act as its user.
- api_tokens table (0026): sha256 hash only, display prefix, role
ceiling, expiry, revocation, throttled last_used_at.
- steward/core/api_tokens.py: mint, list, revoke, resolve. Resolution
re-reads the user's current role and acts with the lower of that and
the token's ceiling, so demoting or disabling a user weakens or kills
their tokens with nothing to revoke by hand.
- /ai-access/ (sidebar → Automation → AI access): every user manages
their own tokens, can't grant above their own role, sees the token
once. Session-only: a share-link viewer is refused. Create and revoke
are audited.
- Integration tests drive the routes against real Postgres so their
transactions run for real (#884 escaped CI behind a mocked session).
Task #4965 "MCP-1 — Personal API tokens".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admin banner counts every plugin load failure, but settings/plugins.html
only rendered failure reasons inside the plugin card macro -- which iterates
discovered plugins. A plugin that failed BECAUSE it has no directory was
therefore counted in the banner and shown nowhere on the page the banner links
to. That dead end is what the operator hit: "1 plugin failed to load" on the
dashboard, nothing wrong on the details page, and no way to clean it up.
Folds the reason into the "Configured but not installed" rows rather than
adding a competing section, and drives that list from the UNION of stored
plugin.* settings and recorded load failures instead of stored settings alone.
A failure with no stored row -- a plugin enabled by a DEFAULTS entry, which is
exactly what plugin.http was -- would otherwise still be counted and still be
invisible.
Rows carry `removable`: only a plugin with a real stored row can be cleaned up,
so a default-declared one shows the reason and an explanation instead of a
Remove button that would delete nothing while the default reasserts it. Rows
with a reason read as an error (red, "Failed to load"); rows that merely have
leftover config stay a warning. A discovered plugin that failed is excluded --
its own card already shows the reason.
Replaces find_orphaned_plugin_names outright rather than adding a second
overlapping helper; it had one caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator's 500 was NOT the startup gap I attributed it to. They found it by
logging in and getting Steward's own error page -- which means create_app had
completed and the app was serving, so the gaierror came from a request handler
acquiring a connection, not from boot. Neither prior fix covers that: the
startup retry never runs, and pool_pre_ping only helps once the database is
back, since its replacement connect fails too while the container is gone.
Adds a before_request gate that acquires a pooled connection with a short
bounded retry (~1.75s over 4 tries) so a database restart is ridden out
invisibly, and renders a distinct 503 "database unavailable" page when the
budget is exhausted. The request budget is deliberately far shorter than the
startup one: nobody watches a container boot, but somebody is watching this
page load, and a page that hangs is worse than one that says what is wrong.
/health and static are exempt -- a liveness probe must stay answerable while
the database is down, or a dependency outage triggers a restart loop.
Credential and missing-database errors still propagate rather than being
reported as "unavailable", which would send the operator chasing the wrong
problem entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator saw the `http` plugin -- deleted when ping/dns/http were unified into
the Monitor entity -- still reported as enabled, with no way to clear it.
It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so
deleting the app_settings row did nothing: the default reasserted it on the
next settings load. That is precisely why no cleanup UI could have fixed it.
Rule 22 says the removed subsystem should have taken its setting with it.
Purged the surviving references -- the DEFAULTS key, "http" in
CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the
capabilities blurb still advertising HTTP/uptime as a bundled capability --
plus a migration dropping any stored plugin.http row. Untouched: `http` as a
MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial,
which is kept deliberately so existing DBs resolve the revision graph.
For the general case, cleanup splits by provenance rather than being uniformly
automatic or uniformly manual:
Bundled plugins ship in the image and version atomically with core, so they
cannot be transiently missing -- a plugin.* default with no bundled directory
is unambiguously a bug. Guarded by a unit test that fails CI, which is the
only part safe to automate.
External plugins live in operator-mounted /data/plugins, where absence is
ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their
config would silently destroy unrecoverable credentials on a transient
condition, so Settings now lists them under "Configured but not installed"
with an explicit, confirmed, audit-logged Remove. The section hides entirely
when empty, and the remove route refuses if the plugin is actually installed.
Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering
to remove a default-only key would be a lie -- and returns names only, since
plugin config can hold credentials and listing orphans never needs their values.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two connection-recovery gaps, both surfaced by a Postgres restart that left
the app throwing tracebacks while the DB itself was healthy.
pool_pre_ping + pool_recycle on the app engine [#2626]: when the database
restarts, every connection already in the pool is dead at the socket level.
SQLAlchemy only discovered that by failing a real query, so the first
operation after a restart errored out on whatever triggered it. pre_ping
checks liveness on checkout and swaps the dead connection transparently;
pool_recycle caps connection age so a socket stranded by a NAT/conntrack
timeout or a Docker network rebuild is retired on a timer instead.
wait_for_database() gate at startup [#2627]: create_app touches the DB
synchronously (migrations, secret re-encryption, settings load) and assumed
it was both resolvable and accepting connections on the first try. Neither
holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2)
or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting
connections"). Both are transient, so retry with capped backoff behind one
gate ahead of the first DB touch. Credential and missing-database errors
are classified by SQLSTATE and still fail immediately -- waiting cannot fix
those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT /
database.connect_timeout, default 60s) since it governs reaching the DB and
so cannot live in the DB-backed settings.
Tests drive the retry loop off a fake clock, so backoff and timeout
behaviour are deterministic rather than wall-clock dependent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The instance has run Gitea since the migration; "Forgejo Actions services:"
is Fabled-Git Actions. Prose only — no workflow or path change.
Scribe issue #2272.
Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config
keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the
agent internals, and a wire-format design point on docker_logs riding in the same
push (with server-side toggle/exclude enforcement + the per-container ring).
The end-to-end chain (store → rotate → query + ingest enforcement) is already
covered by the integration tests added across steps 2–5, so no capstone test is
added. AGENT_VERSION bump (1.7.0) landed in step 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
test_persist_logs_respects_toggle_and_exclude read via s.execute(SELECT) between
two `async with s.begin()` blocks; that SELECT autobegins a transaction, so the
next begin() raised "A transaction is already begun". Move both reads after the
final begin block (the toggle-off persist stores nothing, so end-state still
reflects the exclude phase), and pass a SimpleNamespace stand-in host to sidestep
ORM attribute-expiry across the commit boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.
Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
text search; newest-first so the latest lines survive the poll's scroll reset;
empty/loading states; a "View logs" link from the container detail page.
Jinja auto-escapes log content (no markup injection).
Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
per container — DB-backed, no restart. The toggle + exclude are enforced
authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
the push model has no channel to tell an agent to stop.
Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.
- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.
- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
records (drops the _steward truncation marker, malformed + lineless records,
normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Dedicated host-scoped table for pushed container log lines (one row per line),
chained after docker_008. host_id FK CASCADE; twin indexes — (host_id,
container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring
docker_events. Integration schema-shape test asserts columns + both indexes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Agent-side of Docker container-log collection (milestone 79). Each interval the
agent fetches new log lines per running container over the local Docker socket,
demuxes the multiplexed stream, and folds sample["docker_logs"] into the same
push as metrics — no inbound channel needed.
- since-cursor per container kept in rate-state; a container's first interval
seeds from a short tail, then fetches incrementally via ?since=<cursor>
- _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib,
TTY/unframed fallback); one row per (stream, ts, line)
- per-batch byte cap with a truncation marker; logs are dropped (never buffered)
across a backoff so an outage keeps metrics but not stale logs
- docker_logs_enabled (default on) + docker_log_exclude per-host config keys
- AGENT_VERSION 1.6.0 -> 1.7.0
Server ignores the new sample key until the ingest step lands, so this commit is
CI-safe on its own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
M78 step 2. The /disk page surfaced reclaimable space + stopped counts but
said "prune deferred to a later milestone" — this wires the cleanup.
Adds POST /plugins/docker/disk/<host_id>/prune (admin only) that resolves
the host's linked AnsibleTarget → steward:target:<id> scope and fires the
bundled maintenance/docker_prune.yml through the ansible.run_playbook
capability (audited runner; the collection agent stays read-only). Mirrors
the host_agent update route. `target` ∈ {containers, images, system}:
- containers → docker container prune -f
- images → docker image prune -af (prune_all_images)
- system → docker system prune -f (conservative)
The disk() route now resolves each host group's linked target + capability
availability; disk.html renders three confirm-gated buttons per host, shown
only to admins when Ansible is available and the host has a linked target,
otherwise a clear "link a target" / "runner unavailable" hint (rules 24/26/27).
Tests: the prune_target→extra_vars mapping (the -af-only-for-images rule) is
extracted to a pure helper and unit-tested; disk_prune added to the module
smoke; disk.html covered by the existing template-parse test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
M78 step 1. The bundled maintenance/docker_prune.yml was a single
`docker system prune -f`; the upcoming per-host prune buttons need a
granular split. Add a `prune_target` extra-var:
- containers → docker container prune -f
- images → docker image prune -f (+ -a when prune_all_images)
- system → docker system prune -f (+ -a / --volumes) — the DEFAULT,
so existing manual/scheduled callers (which set no var)
keep today's behavior.
Reuses the existing bundled playbook rather than adding parallel files;
keeps the description/category/confirm meta so it still self-describes as
a confirm-gated maintenance run. Adds unit coverage: the playbook stays
valid YAML, its meta still discovers, and all three targets are guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The publish lane raced on the runner's shared docker daemon when two dev
pushes landed seconds apart: a concurrent run evicted the freshly-built
:<sha> image mid-push, so the post-push `docker tag :<sha> :dev` failed
with "No such image" (a flake — a lone re-run went green).
Two guards, both matching CI-runner's canonical build workflows:
- Workflow-level `concurrency: ci-${{ github.ref }}` with
cancel-in-progress:false serializes runs per ref, so no two publishes
ever share the daemon. false (not true) because rule 46 requires every
push to emit its own immutable :<sha> image — a superseded run must
still finish.
- Build both the :<sha> and moving (:dev/:latest) tags in one
`docker build -t ... -t ...`, then push each. Removes the fragile
post-push `docker tag` of an image a concurrent run could have evicted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The schedule and host run forms opened their Source/Playbook <select>s on a
"— choose … —" placeholder, forcing an extra click. Default them to the first
item and cascade the dependent fields so the initial state is real:
- Remove the placeholder options (schedules source+playbook, host source+
playbook, and the shared _playbook_options fragment).
- schedules(): pass sel_source (first source, or the edited schedule's) and
that source's pre-rendered playbooks so the playbook dropdown starts populated
(create mode too, no longer the all-sources union).
- Cascade on load/change: the source <select> fires playbook-options then
(hx-on::after-settle) re-triggers the playbook <select>, which loads that
playbook's variable fields. Host form drives the whole chain from a load
trigger; schedule create mode loads vars via the playbook <select>'s own load
trigger, while edit mode keeps its server-rendered prefill untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ansible Schedules form only offered a free-form extra-vars textarea,
unlike the browse/host run forms which auto-populate a field per declared
playbook variable (vars:/vars_prompt:) plus the playbook description. Wire
the schedule form to the same shared `_playbook_vars.html` partial:
- Playbook <select> now loads /ansible/playbook-vars?schedule=1 on change
into a #sch-playbook-vars container (mirrors the host detail form).
- Edit mode pre-renders the saved playbook's variables server-side with
their stored values; the extra-vars textarea keeps only leftover
(non-declared) vars.
- Partial gains optional `values` (prefill) and `schedule` flags. In the
schedule context secret vars render disabled ("secrets can't be
scheduled") since they're dropped downstream, and the destructive-confirm
gate is shown but not required.
Backend already parsed var__* fields for schedules, so this is the
front-end/partial parity fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.
- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
minute through the normal alert pipeline, so the operator can alert on them
via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
off Linux. Registered as a core ScheduledTask in app.py.
- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
that skips a tick while a task's prior run is still in flight, so a hung poll
can't stack overlapping runs (which amplify per-poll resource/fd use). A
skipped task isn't penalised — it retries the next tick after it completes.
- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
and asserts /proc/self/fd doesn't grow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UniFi poller caches a UnifiClient holding a long-lived httpx.AsyncClient
(a real socket pool). On a login failure or a mid-poll error it set
`_client = None` to force re-auth, but never called close() — orphaning the
connection pool until GC. Under a flapping/unreachable controller that leaks
a file descriptor every failure tick: the same class of bug as the SNMP
engine leak (OSError: [Errno 24] Too many open files).
Add _drop_client() that aclose()s the client (best-effort) before nulling
the cache, and route both discard paths through it. Regression tests cover
both failure paths plus the noop/close-error contracts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh SnmpEngine() was created on every poll_device() call and never
closed. pysnmp opens a UDP transport socket per engine and doesn't release
it on GC, so each scheduler tick (default 60s, per device) leaked a file
descriptor. Over hours of polling the process hit its fd ceiling and the
listening socket could no longer accept connections — OSError: [Errno 24]
Too many open files on socket.accept(), locking up the app.
Wrap the engine in try/finally and release its transport socket via a new
_close_engine() helper that probes both pysnmp API shapes (6.2.x lextudio
camelCase, canonical 7.x snake_case); all close paths are best-effort so a
failed close never breaks the poll loop. Regression tests cover both shapes
and the never-raises contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SNMP devices map onto a Steward host's detail page via _devices_for_host,
which previously matched the device's `host` field (the SNMP poll target)
against the Host's address/name — a coincidence of strings that breaks when
the poll target differs from how the Host is recorded.
Add two optional per-device config fields that decouple the binding from the
poll target:
• host_id — exact Steward Host UUID match (the explicit link)
• steward_host — friendly bind by Host name or address (case-insensitive)
An explicit binding is exclusive: a device bound to host A never implicitly
matches host B by a coincidental address. No explicit binding → existing
implicit `host`-string match (fully backward compatible).
host_panel passes host.id and badges explicitly-bound devices. plugin.yaml
documents the new fields. Tests cover explicit id/name, exclusivity, wrong
uuid, and legacy fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The container list and dashboard widget listed each swarm task as its own cryptic
row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent.
Make both views service-centric and cluster-complete.
- swarm_view.build_swarm_services(): model-free builder that merges real container
rows (collected local-per-node, so host_id = the node a task runs on) with the
managers' placement (DockerSwarmService.placement_json). Where placement counts
exceed the local rows on a node — i.e. nodes with no agent — it synthesizes
"ghost" replicas so the replica list matches the cluster. Non-swarm containers
pass through grouped by host.
- /rows + rows.html: a Swarm-services panel collapses each service to one block,
every replica a host chip (status + cpu); ghosts render dashed "N · no agent".
Non-swarm/compose containers keep the per-host table below. New Services stat.
- /widget + widget.html: same service collapse with host chips; standalone
containers stay grouped by host.
- Unit tests for the builder (real+ghost merge, full coverage, partial agent-less,
synthesized service, down state).
No migration — node_id, placement_json and swarm-node hostnames are already stored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.
Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.
- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
(globally unique, so only true duplicates merge); first occurrence wins under
the running-first ordering. Applied in both the containers and resources
widgets before counting/limiting. Rows without a container_id (older agents)
are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
distinct containers with a die/oom event in the last 24h (DISTINCT so the
managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
all kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.
- ingest.py: after the upsert loop, delete this host's containers whose name is
notin the newest snapshot (the listing is authoritative — event derivation
already treats absence as removal). Mirrors images/swarm/disk. The existing
`if not snapshots: return` keeps swarm/disk-only samples from touching
containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
% value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
SNMP devices are config-defined by IP/hostname, not Steward Host records, so
they had no presence on a host's page. Map them by address and embed a fragment
(mirrors the Docker per-host fragment).
- _devices_for_host(devices_cfg, address, name): case-insensitive match of a
device's configured host to the Steward host's address or name (tolerates
non-dict / host-less entries).
- Route /plugins/snmp/host/<id>: renders the matched device(s) + latest readings,
or nothing when none map (so hosts without SNMP carry no empty card).
- snmp/host_panel.html: per-device card (name · address · reachability) with a
readings grid (K/M scaling) and a History link to the full device page.
- hosts/detail.html embeds it after the Docker fragment, gated on snmp enabled.
- Unit test for the matching helper (by address, by name, no-match, blank).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Realizes the chosen host-summary layout: a thin live vitals bar at the very top,
separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD).
- New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact
CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) +
Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing
until the agent reports.
- The metric computation (latest snapshot + 6h sparkline query + load/core + PSI)
moves from host_panel into host_vitals. host_panel slims to management only
(reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html
drops the gauge row + pressure block, keeping status + lifecycle actions.
- hosts/detail.html: vitals strip on top (full width, live), then a 2-col
[Monitors | Agent] grid, Ansible full width below, docker fragment last.
UI only. Templates parse; plugin-template parse test covers the new fragment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
- Only the nav list scrolls when long (min-height:0 + overflow on .side-nav);
the brand and user/logout stay pinned top/bottom.
- Keyboard focus rings (:focus-visible) on every interactive sidebar element
(brand, nav links, user/logout) and the mobile ☰ toggle.
- Mobile toggle is now a real toggle: aria-controls/aria-expanded kept in sync,
Escape closes the off-canvas sidebar, scrim close stays in sync too.
- prefers-reduced-motion disables the sidebar slide + link transitions.
- Confirmed no dead top-nav CSS/markup remnants from the old top bar.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.
Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.
View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The full-metrics page rendered once server-side with the history query inline,
so the charts blocked first paint and nothing refreshed without a reload (it
reads the latest snapshot the server holds — the agent pushes ~every 30s).
Split it into a shell + two HTMX fragments:
- Shell (host_detail.html): header + shared time-range toggle + two containers;
paints instantly.
- Current state (/<id>/metrics → _host_metrics.html): identity + gauges +
per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest
query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence.
- History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js,
from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so
they lazy-load (never block paint), refresh slowly, and follow the range
selector. Leak-safe: previous Chart instances are destroyed before re-render.
- Range toggle switched from full-reload links to the shared _time_range.html
(setTimeRange + rangeChange), so only the fragments refetch.
- routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a
_split_host_metrics helper.
- tests/test_templates_parse.py now also parses plugin templates (these
fragments aren't rendered in the unit lane).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention-
bounded (core cleanup prunes > data.retention_days, default 90d) and charts top
out at 30d, so the cost wasn't growth — it was the read path shipping raw rows
to Python.
- _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120
buckets) instead of fetching every raw sample (a 30d range was hundreds of
thousands of rows) and downsampling in Python. Uses the new
(source_module, resource_name, recorded_at) index.
- _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY
recorded_at DESC — newest row per group in one index-ordered pass, replacing
the GROUP-BY-max subquery self-joined back to the whole history.
- Integration test validates both against Postgres.
Deliberately not a materialized raw→hourly rollup: these query-side changes
deliver the speed; a rollup would additionally cut storage and remains a future
option if scale demands it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The host detail page stacked three full-width cards (Monitors, Agent, Ansible)
left-aligned, using only ~40% of the width with lots of vertical whitespace.
Rework toward the operator-chosen "vitals on top + 2-column" layout:
- The agent panel (CPU/MEM/DISK/LOAD vitals + sparklines + lifecycle) moves to
the top, full width, so a host's vitals lead the page.
- Monitors and Ansible sit side by side in a responsive 2-column grid
(auto-fit, stacks under ~420px), filling the width.
- The Docker per-host fragment moves below, full width.
Note: the vitals live in the host_agent fragment (loaded across the plugin
boundary) and uptime in the host-detail context, so a single literal top "strip"
mixing both isn't clean — the agent card on top is the faithful, contained
realization. UI-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
plugin_metrics had only a PK on id, so every host-detail, full-metrics, and
dashboard-widget load sequentially scanned the entire time-series table — which
grows by (sources × resources × sample cadence), so the host views got slower
over time (operator-reported "blocked/slow" loads). monitor_results was already
indexed, so the uptime aggregation wasn't the bottleneck.
Add two composite indexes matching the hot query shapes:
- (source_module, resource_name, recorded_at) — history range scans, fleet/
widget queries, and the 'host:%' sub-resource prefix.
- (source_module, resource_name, metric_name, recorded_at) — the latest-value-
per-metric group-by/self-join.
Model __table_args__ + core migration 0023. Integration lane validates the
migration via `alembic upgrade head`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two issues on the host full-metrics view:
- fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of
"24.8 TB". Add TB and PB tiers.
- Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards
stretched to the Temperatures height — a 40-core CPU made one very tall column
and left Interfaces/Disks with large empty space. Split them: Interfaces +
Disks sit side-by-side at natural height (align-items:start); Temperatures
becomes a full-width card whose readings flow into a compact multi-column grid
(wide-and-short instead of one tall column).
UI-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes /
mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM
(e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32
range" and the entire docker ingest batch failed — no metrics stored for any
host with a large container.
The I/O counters and the milestone-77 rollup/disk tables already used
BigInteger; this trio (docker_001-era) was missed.
- models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes,
DockerContainer.mem_limit_bytes → BigInteger.
- migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place
int4→int8 promotion).
- integration regression test: persist an ~8 GB container, assert the
current-state and time-series rows round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.
- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
_collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
item, raising hook, sorted output).
With docker enabled, "Docker" now appears under Infrastructure → its fleet view.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The flat top bar was a set of ungrouped peers and gave plugin data no home.
Move to a persistent left sidebar with grouped sections, per the navigation
redesign (operator-chosen shell).
- base.html: top <nav> → left <aside class="sidebar"> + content column.
Groups: Overview (Dashboard, Status), Infrastructure (Hosts; plugin links
arrive in slice 2), Monitoring (Monitors, Alerts), Automation (Ansible),
Admin (Settings, Audit; admin-only). Brand on top, user/logout at the bottom.
- Active link highlighted by request.path prefix (aria-current).
- Responsive: sidebar slides off-canvas under 900px via a ☰ toggle + scrim
(inline class toggle, no new JS deps). Candle-glow preserved.
- Logged-out pages (login/setup) render without the sidebar (gated on
session.user_id), content area full-width and centered as before.
- Add tests/test_templates_parse.py: syntax-parses every steward template so a
broken tag fails the unit lane (only the login page renders there today).
Plugin nav links (Docker/SNMP/UniFi/Traefik) come next in slice 2 via a get_nav
hook. UI-only; no behavior/route changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-up to the incident where an unwritable /data made Steward silently mint a
fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only
discovered when an Ansible run failed. Make both failure modes loud and visible.
- config._resolve_secret_key: if a new key must be generated but can't be
persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and
refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data
writable by uid 1000). Also raises a clear error if an existing key file can't
be read. First-run on a writable volume still generates + persists normally.
- core.settings: detect secrets stored as ciphertext that won't decrypt with the
current key (scan_undecryptable_secrets at startup; _is_undecryptable helper).
Cached in memory; set_setting discards a key on a fresh write so the banner
clears without a restart.
- app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject
undecryptable_secrets into the template context.
- base.html: admin banner naming which secrets to re-enter, each linked to its
settings tab.
- compose.deploy.yml: document the uid-1000 /data write requirement and the
STEWARD_SECRET_KEY option for multi-node Swarm.
- Tests: secret-key resolver (reuse existing file, generate when writable, raise
when unpersistable) and the undecryptable-secret detection helper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP
polling disabled" on every poll, so polling never worked. Two coupled defects:
1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp)
was never bundled even though the plugin is first-party and shipped.
2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which
pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep
present, polling would have thrown and silently returned nothing. The 5.x
line that still has the sync API isn't safe on the image's Python 3.13.
Fix:
- Dockerfile: install `.[ansible,snmp]`.
- poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI,
with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd`
+ async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x
`pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency
bump can't silently re-break it.
- scheduler.py: await poll_device directly; drop the run_in_executor wrapper
and the now-unused asyncio import.
- Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping,
that the poller is a coroutine, and the graceful no-pysnmp path.
Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target —
live polling against real devices is verified after deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
CI never renders docker templates through the app (the unit-lane app uses
testing=True, which skips plugin loading), so a Jinja syntax error could ship
green. Add unit tests that parse every docker template, smoke-import the routes
module + confirm the #942 view functions exist, and cover the _human_bytes /
_human_uptime presentation helpers. Closes the render-regression gap the new UI
pages introduced.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Container list now sub-groups each host's containers by compose project (or
swarm service) with a small subheading, preserving the running-first order;
hosts with no such labels render flat as before. The dashboard status widget
links each container to its detail page and surfaces enriched state inline —
health dot (healthy/unhealthy), restart count, and last non-zero exit code for
stopped containers. Completes the #942 UI surface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two new read-only sub-pages, linked from the Docker index header only when the
data exists (non-swarm / Docker-less installs aren't offered empty pages):
- /plugins/docker/swarm — services with replica health (running/desired,
colour-coded green/amber/red), Swarm nodes (role/availability/status, leader
badge), and task→node placement with node ids resolved to hostnames. Grouped
by reporting manager host. Empty state explains manager-only collection.
- /plugins/docker/disk — per-host reclaimable space, image/layer/container/
volume/build-cache sizes, stopped-container count, and a per-image table
(size, shared, ref count, reclaimable badge). Notes prune actions are deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
New /plugins/docker/container/<host_id>/<name> detail page (v1 quality):
status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM),
net + block I/O (humanised), image/compose/swarm-service/node/ports, a
range-toggled CPU/mem history graph (HTMX fragment reusing the time-range
selector), and a lifecycle timeline rendered from docker_events (glyph+colour
per event kind). Not-found and empty-history/empty-timeline states included.
Container names in the main list and the per-host hub panel now link to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
test_disk_usage_persisted opened a second session.begin() after interleaved
SELECTs, which autobegin a transaction → "A transaction is already begun".
Roll back the read transaction before the re-report write. Restores the
integration lane green for the /system/df slice (a840d6f).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.
New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:
* services roll desired-vs-running replicas up from the task list (replica
health isn't on the service object), handle replicated + global mode, strip
the @sha256 image digest, and carry cross-node task→node placement.
* nodes normalise role / availability / status + the manager leader flag.
build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds the milestone-77 storage that doesn't fit on the per-container row:
* docker_events — lifecycle (start/stop/die/oom/health_change), to be
derived by diffing consecutive host snapshots; host-scoped, indexed for
timeline lookups (host_id, container_name, at) and retention pruning (at).
* docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
topology (desired-vs-running replicas, node role/availability/status).
Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).
Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a live range switch to the history widget that re-requests the fragment
without entering edit mode. It rewrites the parent cell's hx-get so the choice
survives polling (htmx re-reads the attribute each poll), then fetches at once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent already collects + ingests net throughput, disk I/O, and temps;
add them to the fleet-glance rows (fmt_bps + two-line io cells + temp with
70/85C threshold color), each shown only when the host reports it so VMs/
containers without sensors don't show blank cells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an inverted uptime_style macro (low-is-bad mirror of metric_style) and
color the inline ping latency (warn 100ms / crit 250ms) and 24h uptime values
in the Hosts-Overview widget, which were the remaining uncolored metrics.
(Uptime/SLA + Ping widgets already colored degraded values.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator chose a single canonical summary. Repurpose the status_overview
widget into 'Overview' (host count + monitor up/down/pending + Alerts link;
key kept so existing dashboards upgrade in place) and remove the redundant
fixed top strip from the dashboard view. Drops _get_summary_stats and its
now-unused PingResult/DnsResult imports (rule 22).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Hosts-Overview: move the CPU sparkline inline next to the cpu % value it
represents (was floated far right on the name line, reading as unrelated).
- Host Agent history widget: caption the chart with the host it represents
(the panel title is generic) — links to the host hub.
- History widget: snap the y-axis to a stable 0..next-10%-band ceiling instead
of auto-scaling to the exact peak, so the 'zoom' no longer jumps each poll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restore the host_agent fleet-glance widget to one horizontal row per host
(name left, metric cells + sparklines right) instead of the multi-column
block grid, and zebra-stripe odd rows so adjacent hosts read as distinct.
Names still wrap rather than hard-truncate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent reports every few seconds, so multi-hour history series were hundreds–
thousands of points — a dense, noisy line (esp. CPU). Bucket-average server-side
to ~120 points (keeps the shape, drops the noise) for both the history-graph
widget and the host-detail charts. Also fix the chart tooltip title showing the
raw epoch-ms (e.g. 1,781,720,471,459) — format it as HH:MM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address graphical issues raised from the dashboard screenshot:
- No more truncated host names. Hosts-Overview and Host-Agent-Resources put the
host name on its own line (full, wraps if needed) with its data grouped beneath
it; ping/dns (.ping-name) and uptime widget names wrap instead of ellipsis.
Prefer vertical overflow over cramming/truncating a row.
- History graph fills the panel: drop the fixed 0–100 y-axis ceiling (beginAtZero
+ 8% grace) so the lines use the vertical space instead of hugging the bottom;
axis labels still show real %.
- Container-query breakpoints: the widget body is now a query container, so
fragments restyle to their OWN panel width. Host-list widgets flow into 2 cols
≥520px and 3 cols ≥900px (.host-blocks) — use the width, remove vertical
deadspace — and collapse to one column when narrow. Mirrored into the share view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator ask: show the graphs next to their fields in the fleet widget, like the
host page's AGENT panel. Each row now renders cpu/mem/disk/load as a value with a
trend sparkline beneath it (1h window), instead of bare numbers.
- _fleet_rows fetches a per-host recent series (cpu/mem/load host-level, disk from
the root mount) in one 1h query and attaches a sparkline per metric to each row.
- widget_table.html lays out a metric cell (label + threshold-coloured value +
sparkline) per field, mirroring panel.html. Threshold colour is computed in the
loop and passed into the cell macro (keeps Jinja macro scope clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase D — the dashboard view IS the edit surface (operator ask):
- /d/<id> renders the grid via Gridstack in STATIC mode — positioned exactly like
before, live HTMX widget bodies keep polling. An "Edit" button flips the same
grid interactive (grid.setStatic(false)) in place, so you drag/resize over real
data. "Done" flips it back. Layout autosaves on change.
- The widget picker is now a bottom drawer (position:fixed overlay) revealed by
body.dash-editing — so the dashboard width is identical entering/leaving edit.
- Add: POST returns a single grid item; JS inserts it + grid.makeWidget +
htmx.process so it loads live data. Remove: POST 204 + grid.removeWidget.
Per-panel drag handle + remove ✕ are in the DOM for editors, shown only while
editing.
- Gridstack loads for everyone (viewers get the static grid); edit wiring is
gated on can_edit. Mobile collapses to one column.
- Removed the separate /d/<id>/edit route + edit.html + _edit_panels.html
(rule 22). Dashboard-list Edit and new-dashboard create now deep-link
/d/<id>?edit=1 which opens edit mode on load.
Browser-only behaviour — CI can't exercise it; needs an operator visual check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase C — bring the host-view graphs onto the dashboard:
- host_resource_history widget reworked into a real host-view chart: epoch-ms
linear axis (no Chart.js date adapter), themed like the host-detail charts,
maintainAspectRatio:false so it fills the resized panel, unique canvas per
widget instance (wid), and empty states ("pick a host" / "no metrics yet").
Was previously unusable — it had a broken time axis and no way to choose a host.
- Add a "host" param type: the edit form renders a live dropdown of hosts
(dashboard routes now pass the host list to the editor); the chosen host_id is
stored in config and fed to the widget.
- Hosts-overview widget gains a per-row CPU sparkline (last hour) via the shared
sparkline_svg helper — the host-view at-a-glance trend, on the main widget.
Charts/sparklines render only in the browser, so CI can't exercise them — needs
an operator visual check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase B — Grafana-style drag-resize grid for dashboard widgets:
- DashboardWidget: replace `position` with a 12-col grid placement
(grid_x/grid_y/grid_w/grid_h). Migration 0021 backfills the old position
order into a 3-up grid (4 cols x 4 cells each) and drops position.
- View + share render a static CSS grid from x/y/w/h: fixed cell height
(h * 70px) with the body scrolling, so the arranged layout is what's shown;
collapses to a single column under 820px.
- Edit view: Gridstack.js 12.6.0 (vanilla, CDN, pinned) — drag the title bar
to move, drag a corner/edge to resize; every change autosaves to a new
/d/<id>/edit/layout endpoint. Replaces the SortableJS position reorder.
- add_widget appends at the bottom-left; remove no longer renumbers.
_get_widgets now orders by grid position (drives DOM + mobile fallback order).
Note: Gridstack drag-resize is browser-only, so CI can't exercise it — needs an
operator visual check of the edit experience.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase A (clarity wins, no schema change):
- Add shared metric_style() macro in _macros.html: colours a numeric metric
amber (>=warn) / red (>=crit) on the value ITSELF, not just the status dot.
Defaults 80/90 for percentage gauges; load /core uses warn 80 / crit 100.
Applied to CPU/mem/disk across host_agent panel + fleet widget + the unified
hosts-overview widget.
- Link every host reference to the host hub (/hosts/<id>) in ping, dns,
hosts-overview, host_agent fleet, and uptime widgets; status widget entries
link to their own detail_url. All guarded on session.user_id so the public
share view degrades to plain text (the bare href carries no share token).
- Fix truncation: title= tooltips on all names that ellipsis, plus a hover
underline affordance on the now-clickable ping-name links.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New maintenance/system_update.yml: cross-distro OS package upgrade (apt +
dnf/yum) with two run-form flags:
- restart_services: restart every service that needs it after the upgrade
(Debian via needrestart -r a, installed if missing; RHEL via
needs-restarting -s → systemctl try-restart).
- reboot_if_required: reboot the host only when a reboot is actually pending
(Debian /var/run/reboot-required; RHEL needs-restarting -r). Never reboots
otherwise.
Tagged steward:category=maintenance and steward:confirm=true (significant /
reboot-capable), so it shows in the run UI with the confirm gate and both flags
as fill-in fields. No code changes — auto-discovered from the builtin source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Host agent panel (reporting state) gains a "Re-provision" collapsible (admin +
linked target + managed key): bootstrap user/password → provision.yml, which
reinstalls the steward account + managed key + agent. This is the missing path
after regenerating the managed key — Update alone can't fix a broken key.
- Remove the Ansible sections (run-playbook + target link) from the host EDIT
page — they were the stale free-text version and the wrong place. They live on
the host detail hub (dropdown + discovered variables). Edit page now links to
the host page; edit_host route simplified (no ansible fetch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a session has expired, require_role now bounces to /login?next=<path> and
sends the user back there after re-login.
- middleware: safe_next_url() (same-site relative path/query only; rejects
off-site, protocol-relative, javascript:, and the auth pages — no open
redirect). _login_redirect() builds the next param; for HTMX requests it uses
HX-Current-URL (the page, not the fragment) and HX-Redirect so the whole
browser navigates instead of swapping the login page into a fragment.
- login GET/POST carry `next` (hidden field), validated, used on success for
local + LDAP; OIDC stashes it in session across the IdP round-trip.
- login.html: hidden next field + next on the SSO link.
- tests for safe_next_url.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When ansible.ssh_private_key can't be decrypted (app secret key changed),
decrypt_secret returns the ciphertext unchanged; the executor was writing that
enc:v1: blob as the SSH key file → cryptic "Load key ...: error in libcrypto"
→ Permission denied. Now:
- build_credentials skips a still-encrypted key value (never writes ciphertext
as a key file).
- start_run broadcasts a plain-language run error: "The managed SSH key could
not be decrypted (the app secret key changed). Regenerate it in Settings →
Ansible and re-provision the host(s)."
The run still fails (no usable key), but the reason is now obvious instead of a
libcrypto error. Operator remedy: regenerate the managed key + re-provision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run UX: when start_run throws before/around launch (e.g. ENOSPC creating the
temp dir — the box is out of disk), the run was marked "failed" with empty
output. Now the exception is broadcast + written to the run output/results so
the run view shows e.g. "[run error] OSError: [Errno 28] No space left on
device" instead of a blank failure.
Widget audit follow-ups (no broken links were found; these are consistency):
- host_resource_history widget now charts root (/) disk, consistent with the
host panel (was the opaque "disk worst").
- host_resources widget: tooltip on the health dot explaining it warns on the
worst mount while the number shows root.
- status_overview widget detail_url /status → /status/ (avoid redirect).
- Normalize ad-hoc widget empty-states to the shared .empty style (wording,
which distinguishes "configured" vs "data yet", preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the dashboard in line with the unified host IA + improved displays.
- New core "Hosts — Overview" widget (/hosts/overview/widget): one row per host
combining monitor status (ping dot + latency, uptime 24h) with the agent
glance (CPU / memory / disk root + stale flag), each row linking to the host
hub. Reads agent data from the generic PluginMetric table via a core-safe
_agent_overview_by_host helper (no host_agent import); freshness vs the
plugin's stale window. The granular Ping/DNS/Uptime/Agent widgets stay.
- Group the add-widget picker into Core monitors / Monitoring capabilities /
Integrations (a `group` field on every WIDGET_REGISTRY entry + section
headings in _edit_panels.html), matching the Settings → Plugins taxonomy.
- Fix the agent fleet widget rows to link to the host hub (/hosts/<id>) instead
of the old /plugins/host_agent/<id>/ page.
Scribe #903.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the playbook metadata convention with a namespaced `# steward:<key>:`
comment block:
- steward:category — free-text grouping label, shown as a badge in the browse
list and on the run form.
- steward:confirm — true/yes/1/on marks a playbook destructive; the run form
then requires a confirmation tick (required checkbox in the shared vars
fragment) before it can launch.
sources.discover_playbook_meta() parses description + category + confirm (first
match per key; `# description:` still primary, `# steward:description:` alias).
discover_playbook_description() now delegates to it. The browse list reads
per-playbook meta to show category badges + descriptions; the run-form and
playbook-vars fragments render the badge + confirm gate.
Bundled playbooks tagged: docker_prune → category maintenance + confirm true;
provision/install/update → category host-agent.
Docs: docs/reference/playbook-authoring.md updated (keys now implemented) and a
quick reference added next to the code at steward/ansible/PLAYBOOK_CONVENTIONS.md.
Tests added for category/confirm/alias parsing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A self-contained guide (docs/reference/playbook-authoring.md) to the contract
between a playbook and Steward's run UI — the `# description:` comment,
vars/vars_prompt → fill-in fields, secret naming, hosts: all targeting,
managed credentials, and idempotency. Includes a "what Steward reads" summary
table, the metadata/extensibility note (only `# description:` today; reserved
`# steward:<key>:` namespace for future keys), and an annotated example.
Meant to be fed to another session authoring Steward-friendly playbooks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Playbooks can ship a human description Steward reads and shows when one is
selected. Convention: a `# description: <text>` magic comment (Ansible rejects
unknown play keys, so a comment is the portable place — works for third-party
playbooks too); falls back to the first play's name:. sources
.discover_playbook_description().
Surfaced at the top of the shared _playbook_vars.html partial, which loads on
playbook selection in the host run form, schedules form, and browse run form.
All four bundled playbooks (provision/install/update/docker_prune) now carry a
description line. Unit tests added.
Scribe #900.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Each at-a-glance metric (CPU, Memory, Disk /, Load) now shows a 6h sparkline
(reused core.status.sparkline_svg + a recent-series query; disk uses the root
mount sub-resource) so trend/consistency is visible, not just the instant.
- Load is now normalized: "Load /core" = 1m load ÷ CPU cores as % (100% = run
queue matches capacity), comparable across different hardware. Cores derived
from the per-core CPU metrics already collected — no agent change. Raw load
in the tooltip.
- Added a "Pressure 10s" line: PSI cpu/mem/io (some, avg10), the
hardware-independent saturation signal already collected by the agent.
Scribe #898.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Disk (worst)" number was opaque at a glance. Show the root filesystem (/)
usage instead — what people actually care about — on the host panel and the
dashboard widget. "Worst" is kept only for the widget's health dot (so a full
/var or /data still warns) and on the full-metrics page (per-mount + worst
trend). Added hover tooltips defining CPU / Memory / Disk / Load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh install had no prompt for general.public_base_url, so first-run agent
installs/share links silently used the request Host header. Add a "Steward URL"
field to the first-run /setup page (create-admin), pre-filled with the current
address, with help text. setup_post saves general.public_base_url and applies
it to app.config immediately (no restart). Editable later in Settings → General.
Scribe #896.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The assertion failed ("Pass steward_url/token/pubkey") because those were
injected as inventory HOST vars, but the playbooks declared them in the play
`vars:` block — and play vars OUTRANK inventory host vars, so the empty
defaults won and the injected values never reached the play.
- Pass globals (steward_url, steward_pubkey, steward_user, agent_interval) as
extra-vars via the JSON -e @file (highest precedence, space-safe). Keep only
the per-host steward_token as an inventory host var.
- provision.yml / install.yml: drop steward_token from `vars:` so the host var
isn't shadowed; assertions use `| default('')` for the un-defaulted token.
This was the next layer under the inventory-format fix — first the inventory
wouldn't parse, now the injected vars actually apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
