Commit Graph
276 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 b1da073677 test(mcp): import only User locally; UserRole comes from the enclosing scope (#5035)
CI / lint (push) Successful in 1s
CI / integration (push) Successful in 43s
CI / unit (push) Successful in 1m2s
CI / publish (push) Successful in 1m5s
The local import shadowed UserRole for the whole function, so its earlier
use read an unassigned name (ruff F823, CI run 8294).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 11:03:16 -04:00
bvandeusenandClaude Opus 5.5 7a3f7869fe feat(auth): sessions end when they should; logins are throttled; Secure behind HTTPS (#5035)
CI / lint (push) Failing after 2s
CI / integration (push) Failing after 47s
CI / unit (push) Successful in 1m5s
CI / publish (push) Skipped
#5035 "Sessions: lifetime applied, role and is_active re-read per request,
logout is a POST, failed logins throttled".

- steward/core/proxy.py: trusted_value / is_https / client_ip read
  X-Forwarded-Proto and -For only as far as the trusted proxy hops
  (Settings → General, default 1, 0 = trust none).
- The session cookie is Secure, and HSTS is sent, only on requests that
  arrived over HTTPS by that rule; plain-HTTP and LAN use are unchanged
  and nothing redirects.
- A login lasts Settings → General's session lifetime from when it began
  (session.permanent + a login_at stamp checked every request — the
  cookie's own age slides on every response). A deleted or deactivated
  user is logged out on their next request; a role change applies at once.
- Logout is a POST (a form in the sidebar); GET /logout is gone.
- Failed logins pause an address after login.max_failures within
  login.lockout_minutes (Settings → General; defaults 5 in 15 min),
  keyed by address so nobody can lock a user out by name. Failures are
  audited as auth.login_failed with the address.
- to_access_cfg projects the new settings into app.config at startup and
  on save, beside the existing projections.
- Integration tests seed real users and a login_at, since every request
  now re-reads the session's user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 11:02:27 -04:00
bvandeusenandClaude Opus 5.5 8dad79f17a feat(ansible): runs stay inside their source; controller-side vars are admin-only (#5034)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m12s
CI / integration (push) Successful in 59s
CI / publish (push) Successful in 1m3s
#5034 "Playbook runs: path and inventory checks in trigger_run, connection
vars admin-only, confirm enforced server-side, web actions audited".
Ansible runs inside the Steward container, so an operator who could name
any playbook path, set ansible_connection=local or ProxyCommand ssh args,
or put {{ lookup('pipe', …) }} in a value could run commands on Steward.

- steward/ansible/guards.py: check_playbook (the source must list it),
  check_inventory_scope (steward:* or an inventory file the source has),
  check_vars_for_role (ansible_* beyond host/port/user/password/become,
  and any Jinja in a value, are admin-only), changed_vars (an edit is
  held to the role only for what it changes).
- trigger_run applies the path checks, so every run path — web, MCP,
  alerts, schedules, plugins — gets them.
- Variables are checked by role on the web run form, host-page runs,
  schedule create/update, inventory group/target create/update, and MCP
  run_playbook / create_schedule / update_schedule.
- A steward:confirm playbook needs the confirmation ticked on the server
  side for a real (non-check) web run, as MCP already required.
- Audited: runs, cancel, schedule create/update/enable/disable/delete/
  run-now, playbook create/update/delete, inventory group and target
  create/update/delete (variable names only, never values). One
  audit_request helper serves the web routes.
- The CSRF form guard and the route guard now each test that they fail
  on a synthetic broken input (rule: a guard must be able to fail).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:57:34 -04:00
bvandeusenandClaude Opus 5.5 6a1bc437c9 feat(auth): share links open only their own widgets; unmarked routes are closed (#5033)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 45s
CI / unit (push) Successful in 1m4s
CI / publish (push) Successful in 53s
#5033 "Share links open only their own dashboard; every route without a
role is on an explicit allow-list". Any live share token in ?s= made the
caller a viewer on every viewer route — host pages, container logs, any
dashboard — and the host-agent widget routes had no role check at all.

- steward/dashboard/share.py: a share token opens, read-only, exactly the
  widget fragment URLs its own dashboard loads — same path, same
  configured parameters; only the time window (hours/range) may change.
- require_role marks its view; public_route marks the deliberately open
  ones (login, setup, health, the share page, agent ingest, install.sh,
  agent.py) with the reason. A before_request refuses any view with
  neither, so a route that forgets its decorator is closed, not open.
- /plugins/host_agent/widget and /widget/history now require viewer.
- The history widget's range buttons keep the rest of the query (and so
  a share link's token) instead of rebuilding it; the uptime widget's
  "Full SLA" link shows to logged-in users, not to share viewers.
- tests/test_route_access.py: a source scan that fails on any unmarked
  route in core or bundled plugins, the default-deny refusal, and the
  share matcher's cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:51:29 -04:00
bvandeusenandClaude Opus 5.5 d188a673a6 feat(plugins): installs fetch only what the catalog names, verified (#5032)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 44s
CI / unit (push) Successful in 1m5s
CI / publish (push) Successful in 58s
#5032 "Plugin install takes its URL and checksum from the catalog, requires
the checksum, and extracts only inside the plugin folder". The install form
carried the download URL and checksum as hidden fields, so any admin POST
could make Steward fetch and import code from anywhere, unchecked, and a zip
member like ../x.py was written wherever it pointed.

- The route takes only the plugin name and reads download_url and
  checksum_sha256 from the configured catalogs; a name no repository lists
  is refused. Installs are audited (plugin.install / plugin.install_failed).
- The checksum is required; the name must be a plain identifier.
- extract_plugin refuses any member that resolves outside the plugin
  folder, caps the download at 50 MB and the unpacked size at 200 MB.
- The zip unpacks into a staging folder and is swapped in whole: a refused
  or failed install leaves the installed copy untouched, and an update no
  longer leaves the old version's deleted files behind.
- The catalog shows "Can't install: the catalog lists no checksum" in place
  of the button for an entry without one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:47:43 -04:00
bvandeusenandClaude Opus 5.5 96126b1c83 feat(auth): forged browser requests are refused (#5031)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 49s
CI / unit (push) Successful in 1m3s
CI / publish (push) Successful in 1m18s
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token
on every non-GET, HTMX header". A page on another site could make a
logged-in browser POST to Steward with the session cookie attached.

- steward/auth/csrf.py: a per-session token, checked in one before_request
  for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field
  csrf_token, constant-time compare). The bearer-authenticated host-agent
  ingest is exempt; /mcp never reaches Quart.
- Every POST form renders {{ csrf_field() }}; base.html carries the token
  in a meta tag and adds it to every HTMX request; the dashboard's fetch()
  calls send it.
- The session cookie is SameSite=Lax; login issues a fresh token.
- PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set
  when Settings → General was saved).
- tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption,
  and a guard that every POST form in the templates carries the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:44:19 -04:00
bvandeusenandClaude Opus 5.5 1d9ac2b46a feat(mcp): built-in playbooks get steward_url from Steward itself
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 48s
CI / unit (push) Successful in 1m11s
CI / publish (push) Successful in 1m23s
#5015 "Built-in host-agent playbooks make an MCP caller supply steward_url"

- run_playbook, create_schedule and set_alert_action fill a steward-builtin
  playbook's steward_url when the caller leaves it out or empty, from
  Settings → General → Public base URL, else the scheme and host the MCP
  request arrived on — the same source the web actions use.
- run_playbook's reply names what it filled (filled_by_steward), and
  get_playbook shows that value as the variable's default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:30:23 -04:00
bvandeusenandClaude Opus 5.5 06e89308a8 feat(ansible): a run's scope reads as the target or group it names
#5014 "Ansible run scope is shown as a raw id"

- inventory_gen.describe_scope / scope_labels turn "steward:target:<uuid>"
  into "target web-1", "steward:group:<uuid>" into "group Docker Swarm",
  and a deleted one into "target abcdef01 (deleted)".
- MCP run and schedule rows carry scope_label beside the stored scope; the
  ansible.run.started audit detail records it too.
- The web run page and schedules table show the name, with the stored
  scope as the tooltip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:29:06 -04:00
bvandeusenandClaude Opus 5.5 30cb375079 test(docker): host-wide log search includes the removed container's line
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 49s
CI / unit (push) Successful in 1m5s
CI / publish (push) Successful in 1m23s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:27:34 -04:00
bvandeusenandClaude Opus 5.5 f72ed97034 feat(mcp): get_host groups a host's parts by kind and says what it cut
#5012 "get_host cuts `parts` without saying so"

- The agent's sub-resources come back as mounts (fullest first),
  interfaces and disk_io (busiest first) and temperatures (hottest first),
  each bounded on its own with total, truncated and a `more` that names
  parts_limit and query_metrics.
- Per-core CPU is a summary (count, min/avg/max, three busiest), so 38
  cores no longer push every interface out of the reply.
- New argument parts_limit (default 20, max 200).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:27:20 -04:00
bvandeusenandClaude Opus 5.5 190a5d1e80 feat(docker): a removed container's logs and events stay reachable by name
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 50s
CI / unit (push) Successful in 1m5s
CI / publish (push) Skipped
#5010 "Logs of a removed container cannot be reached by name"

- get_container and search_container_logs fall back to the names Steward
  still holds logs or events for when no live container matches: the exact
  name from logs and events, or a part of it that matches one removed
  container's events.
- get_container answers with removed: true, last_seen_at, its history and
  events; search_container_logs marks container_removed and notes when it
  was last seen, and says when that is older than the range.
- A miss now says the container may have been removed, that Swarm replaces a
  dead task under a new name, and where to look instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:26:04 -04:00
bvandeusenandClaude Opus 5.5 e9341c5639 feat(docker): overview findings say when the problem began
#5013 "Overview findings for containers and Swarm services carry since=null"

- Container problems are dated from their own evidence: a crash or restart
  loop from its first death in the hour, an OOM kill or error exit from the
  death that ended it, a failing healthcheck from its last turn to unhealthy.
  list_containers and get_container carry problem_since / problem_age_s.
- Swarm services carry trouble_since: a crash loop's first failure in the
  hour, a short service's oldest troubled task in each failing slot, a
  recovered service's first failure in 24h. list_swarm_services shows it.
- get_overview findings from the docker plugin now have since and age_s,
  so they rank by how long they have stood.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:24:10 -04:00
bvandeusenandClaude Opus 5.5 bac2859455 feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 52s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 1m41s
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"

Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
  desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
  docker_010_swarm_tasks).

Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
  `running`. Without the task on record, a newer or running task in the same
  slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
  - failures_1h and failures_24h (failed or rejected tasks);
  - crash_looping (Settings → Thresholds count);
  - last_error, in Swarm's own words.

MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
  (warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.

Web:
- The containers page swarm cards and the Swarm topology page show crash
  loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
  longer counts a rolling update's killed task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 09:46:07 -04:00
bvandeusenandClaude Opus 5.5 3474cdff46 feat(mcp): connect card, MCP docs, and an end-to-end roles/audit test over HTTP
CI / lint (push) Successful in 1s
CI / integration (push) Successful in 44s
CI / unit (push) Successful in 1m5s
CI / publish (push) Successful in 1m15s
MCP-8 #4972 "Connect card, docs, and end-to-end integration test".

- AI access page: a "Connect a client" card with the /mcp URL (from
  Settings → General → Public base URL, else the request host) and recipes
  for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP
  clients, plus the reverse-proxy note. The token-created page shows the
  same recipes with the new token filled in, ready to paste.
- docs/core/mcp.md: connecting, the reverse-proxy requirements, the
  security model (token ceiling, default deny, check-mode default, secrets,
  audit), tools by role, response conventions, adding tools.
  docs/architecture.md describes the /mcp dispatch outside the blueprint
  pipeline; docs/reference/code-map.md gains an MCP section.
- tests/integration/test_mcp.py: over HTTP, a viewer token lists no write
  tools and is refused one by name; an operator token reads get_overview
  and its maintenance-window create/end land in the audit log as
  "<user> via MCP" with the token id. (Revoked → 401 was already covered.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:25:41 -04:00
bvandeusenandClaude Opus 5.5 9abe0beecc feat(mcp): monitoring-config write tools — monitors, alert rules, acks, maintenance
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 41s
CI / unit (push) Successful in 1m0s
CI / publish (push) Successful in 58s
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".

The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
  1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
  object of strings, a key the type does not take is named. The forms now
  answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
  metrics_for, rule_fields (source, resource and metric from the same
  choices the rule form offers), add_rule, acknowledge, window_fields.

Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:19:54 -04:00
bvandeusenandClaude Opus 5.5 2ebfdc8cdf test(docker): the plugin registers prune_docker as an admin write tool
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 43s
CI / unit (push) Successful in 1m0s
CI / publish (push) Successful in 58s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:09:57 -04:00
bvandeusenandClaude Opus 5.5 680a6bf3b6 feat(mcp): automation tools — playbooks, runs, schedules, alert actions, docker prune
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 57s
CI / unit (push) Failing after 1m8s
CI / publish (push) Skipped
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions".

Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory
(variable names only), list_ansible_runs, get_ansible_run (log tail + grep),
list_schedules, wait_for_run (polls, 120s cap).

Operator: run_playbook (check mode unless check=false; a steward:confirm
playbook applies only with confirmed=true; secret variables reach the run
but are never stored, audited or returned), cancel_ansible_run, and
create/update/delete_schedule (secrets refused, check by default).

Admin: set_alert_action, and the docker plugin's prune_docker, which
previews from the disk report unless check=false (Ansible's check mode
skips the prune commands, so a check run would show nothing).

Every write is audited as "<user> via MCP" with the token id; the audit
tool's user filter includes those rows. prune_extra_vars moves to the
docker service so the page and the tool share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:06:15 -04:00
bvandeusenandClaude Opus 5.5 21338dbcd9 feat(mcp): docker and host-agent tools; TLS and crash-loop cutoffs in Settings
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 37s
CI / unit (push) Successful in 58s
CI / publish (push) Successful in 54s
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.

docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
  (restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
  problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
  lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
  and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
  lines, paging by a full-precision `before` cursor; over-long lines keep
  both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
  join get_overview.

host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.

Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:47:02 -04:00
bvandeusenandClaude Opus 5.5 41c7509366 feat(mcp): diagnostic tools — overview, hosts, monitors, alerts, metrics, audit
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 36s
CI / unit (push) Successful in 58s
CI / publish (push) Successful in 1m2s
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Twelve read-only viewer tools:

- get_overview: what is wrong now, ranked critical-then-oldest. Failing
  monitors with their error and when the run of failures began, firing,
  pending and still-breaching acknowledged alerts, TLS certificates expiring
  within 14 days, agents that stopped reporting, hosts over the configured
  CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in
  the last 24h, and findings from enabled plugins through a new optional
  `mcp_overview(db)` plugin hook. Each finding names the call that shows
  its evidence.
- list_hosts, get_host, get_host_metrics
- list_monitors, get_monitor (HTTP header values and URL passwords are
  redacted)
- list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows
- query_metrics: the whole metric store, narrowed one argument at a time
  (sources > resources > latest values > a series summary)
- list_audit_events

Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s,
bounded lists that say when and how they were cut, series as
min/avg/max/p95/last plus a ≤60-point series, lookup by id or
case-insensitive name with a did-you-mean, and a range argument that
refuses an unknown key instead of reading it as 24h.

New core reads in steward/core/metric_series.py (sources and resources
via loose index scans, latest values via DISTINCT ON, series across the
raw/hourly-rollup seam). monitors/service.py gains monitor_states,
down_since and recent_failures; agent_overview_by_host now carries the
fullest disk and the last sample time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:39:41 -04:00
bvandeusenandClaude Opus 5.5 312ea95bcf refactor: move host, docker, alert and monitor queries into service modules
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 35s
CI / unit (push) Successful in 59s
CI / publish (push) Successful in 56s
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:

- steward/hosts/service.py: uptime, status, monitors, agent metrics,
  agent overview (stale threshold passed in, not read from current_app),
  CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
  list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
  list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).

Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:29:21 -04:00
bvandeusenandClaude Opus 5.5 970adf41e7 test(mcp): stay connected until the SSE reply ends; drop cross-context reset
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 37s
CI / unit (push) Successful in 1m0s
CI / publish (push) Successful in 57s
The test client reported http.disconnect as soon as the request body was
read, and the SDK's SSE response cancels its stream on disconnect, so
tools/list came back 200 with an empty body. Model a real client: wait
for the response (with a 15s deadline), then disconnect.

The caller fixture reset its ContextVar from sync teardown, a different
context from the async test that set it; each async test already runs in
its own context copy, so the reset is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:25:23 -04:00
bvandeusenandClaude Opus 5.5 3984f639e8 feat(mcp): /mcp endpoint with bearer auth and role-gated tools
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 35s
CI / unit (push) Failing after 1m0s
CI / publish (push) Skipped
Second step of milestone 461. An MCP client (Claude Code, Claude Desktop
via mcp-remote, Open WebUI) connects to /mcp with a personal API token.

- mcp>=2.2 (SDK v2 MCPServer). Mounted by wrapping app.asgi_app, so /mcp
  never enters Quart's pipeline; stateless streamable HTTP so a redeploy
  never strands a client on a session id the new process didn't issue.
- Auth: Authorization: Bearer stw_… resolves to a Caller (user +
  effective role) in a ContextVar; 401 with WWW-Authenticate otherwise.
  The web pipeline's database gate runs first, so a DB restart answers
  503 instead of failing inside a tool.
- Registry, default-deny: register_tool(fn, role=, read_only=) is the only
  door; role and read_only are required. Tools above the caller's role,
  or from a disabled plugin, are neither listed nor callable. Undeclared
  arguments are refused (not silently dropped); a tool's ValueError
  reaches the agent as its reason; a DB failure mid-tool says so.
- Plugins contribute tools via an optional register_mcp_tools(register)
  export, all-or-nothing, replaced on hot-reload. Documented in
  docs/plugins/writing-a-plugin.md.
- First tool: whoami.
- Tests drive app.asgi_app directly (Quart's test client never reaches
  /mcp); the integration test mints a real token against Postgres and
  runs initialize → tools/list → whoami → revoke → 401.

Task #4966 "MCP-2 — /mcp transport".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:24:00 -04:00
bvandeusenandClaude Opus 5.5 ebc25b99ed test(api): rename one-letter role aliases (ruff E741)
CI / integration (push) Successful in 35s
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 58s
CI / publish (push) Successful in 1m2s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:18:33 -04:00
bvandeusenandClaude Opus 5.5 1c60fb6b4d feat(api): personal API tokens and the AI access page
CI / lint (push) Failing after 3s
CI / integration (push) Successful in 37s
CI / unit (push) Successful in 58s
CI / publish (push) Skipped
First step of milestone 461 (MCP server). A token is the credential an
MCP client presents as `Authorization: Bearer stw_…` to act as its user.

- api_tokens table (0026): sha256 hash only, display prefix, role
  ceiling, expiry, revocation, throttled last_used_at.
- steward/core/api_tokens.py: mint, list, revoke, resolve. Resolution
  re-reads the user's current role and acts with the lower of that and
  the token's ceiling, so demoting or disabling a user weakens or kills
  their tokens with nothing to revoke by hand.
- /ai-access/ (sidebar → Automation → AI access): every user manages
  their own tokens, can't grant above their own role, sees the token
  once. Session-only: a share-link viewer is refused. Create and revoke
  are audited.
- Integration tests drive the routes against real Postgres so their
  transactions run for real (#884 escaped CI behind a mocked session).

Task #4965 "MCP-1 — Personal API tokens".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:15:23 -04:00
bvandeusenandClaude Opus 5 0da88c7076 fix(plugins): show load-failure reasons on the page the banner links to
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 45s
CI / integration (push) Successful in 2m24s
CI / publish (push) Successful in 1m14s
The admin banner counts every plugin load failure, but settings/plugins.html
only rendered failure reasons inside the plugin card macro -- which iterates
discovered plugins. A plugin that failed BECAUSE it has no directory was
therefore counted in the banner and shown nowhere on the page the banner links
to. That dead end is what the operator hit: "1 plugin failed to load" on the
dashboard, nothing wrong on the details page, and no way to clean it up.

Folds the reason into the "Configured but not installed" rows rather than
adding a competing section, and drives that list from the UNION of stored
plugin.* settings and recorded load failures instead of stored settings alone.
A failure with no stored row -- a plugin enabled by a DEFAULTS entry, which is
exactly what plugin.http was -- would otherwise still be counted and still be
invisible.

Rows carry `removable`: only a plugin with a real stored row can be cleaned up,
so a default-declared one shows the reason and an explanation instead of a
Remove button that would delete nothing while the default reasserts it. Rows
with a reason read as an error (red, "Failed to load"); rows that merely have
leftover config stay a warning. A discovered plugin that failed is excluded --
its own card already shows the reason.

Replaces find_orphaned_plugin_names outright rather than adding a second
overlapping helper; it had one caller.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-13 08:36:35 -04:00
bvandeusenandClaude Opus 5 8c50ff242c fix(db): ride out a brief database outage mid-request; explain it if it persists
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 40s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 1m2s
The operator's 500 was NOT the startup gap I attributed it to. They found it by
logging in and getting Steward's own error page -- which means create_app had
completed and the app was serving, so the gaierror came from a request handler
acquiring a connection, not from boot. Neither prior fix covers that: the
startup retry never runs, and pool_pre_ping only helps once the database is
back, since its replacement connect fails too while the container is gone.

Adds a before_request gate that acquires a pooled connection with a short
bounded retry (~1.75s over 4 tries) so a database restart is ridden out
invisibly, and renders a distinct 503 "database unavailable" page when the
budget is exhausted. The request budget is deliberately far shorter than the
startup one: nobody watches a container boot, but somebody is watching this
page load, and a page that hangs is worse than one that says what is wrong.

/health and static are exempt -- a liveness probe must stay answerable while
the database is down, or a dependency outage triggers a restart loop.

Credential and missing-database errors still propagate rather than being
reported as "unavailable", which would send the operator chasing the wrong
problem entirely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-13 08:23:01 -04:00
bvandeusenandClaude Opus 5 59fece855d fix(plugins): remove the phantom http plugin; surface orphaned plugin settings
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m22s
CI / publish (push) Successful in 1m16s
Operator saw the `http` plugin -- deleted when ping/dns/http were unified into
the Monitor entity -- still reported as enabled, with no way to clear it.

It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so
deleting the app_settings row did nothing: the default reasserted it on the
next settings load. That is precisely why no cleanup UI could have fixed it.
Rule 22 says the removed subsystem should have taken its setting with it.

Purged the surviving references -- the DEFAULTS key, "http" in
CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the
capabilities blurb still advertising HTTP/uptime as a bundled capability --
plus a migration dropping any stored plugin.http row. Untouched: `http` as a
MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial,
which is kept deliberately so existing DBs resolve the revision graph.

For the general case, cleanup splits by provenance rather than being uniformly
automatic or uniformly manual:

  Bundled plugins ship in the image and version atomically with core, so they
  cannot be transiently missing -- a plugin.* default with no bundled directory
  is unambiguously a bug. Guarded by a unit test that fails CI, which is the
  only part safe to automate.

  External plugins live in operator-mounted /data/plugins, where absence is
  ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their
  config would silently destroy unrecoverable credentials on a transient
  condition, so Settings now lists them under "Configured but not installed"
  with an explicit, confirmed, audit-logged Remove. The section hides entirely
  when empty, and the remove route refuses if the plugin is actually installed.

Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering
to remove a default-only key would be a lie -- and returns names only, since
plugin config can hold credentials and listing orphans never needs their values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 23:41:06 -04:00
bvandeusenandClaude Opus 5 6c9b89390a fix(db): survive a database restart and a not-yet-ready database
CI / lint (push) Successful in 4s
CI / unit (push) Successful in 50s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 1m2s
Two connection-recovery gaps, both surfaced by a Postgres restart that left
the app throwing tracebacks while the DB itself was healthy.

pool_pre_ping + pool_recycle on the app engine [#2626]: when the database
restarts, every connection already in the pool is dead at the socket level.
SQLAlchemy only discovered that by failing a real query, so the first
operation after a restart errored out on whatever triggered it. pre_ping
checks liveness on checkout and swaps the dead connection transparently;
pool_recycle caps connection age so a socket stranded by a NAT/conntrack
timeout or a Docker network rebuild is retired on a timer instead.

wait_for_database() gate at startup [#2627]: create_app touches the DB
synchronously (migrations, secret re-encryption, settings load) and assumed
it was both resolvable and accepting connections on the first try. Neither
holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2)
or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting
connections"). Both are transient, so retry with capped backoff behind one
gate ahead of the first DB touch. Credential and missing-database errors
are classified by SQLSTATE and still fail immediately -- waiting cannot fix
those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT /
database.connect_timeout, default 60s) since it governs reaching the DB and
so cannot live in the DB-backed settings.

Tests drive the retry loop off a fake clock, so backoff and timeout
behaviour are deterministic rather than wall-clock dependent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 23:21:58 -04:00
bvandeusen 4b97bd01ea docs: Fabled-Git, not Forgejo, in ci-requirements
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 49s
CI / integration (push) Successful in 2m30s
CI / publish (push) Successful in 1m27s
The instance has run Gitea since the migration; "Forgejo Actions services:"
is Fabled-Git Actions. Prose only — no workflow or path change.
Scribe issue #2272.
2026-07-31 23:42:38 -04:00
bvandeusenandClaude Opus 4.8 8c2d994e4c docs(host-agent): document container-log collection [M79 step 6]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 42s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 5s
Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config
keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the
agent internals, and a wire-format design point on docker_logs riding in the same
push (with server-side toggle/exclude enforcement + the per-container ring).

The end-to-end chain (store → rotate → query + ingest enforcement) is already
covered by the integration tests added across steps 2–5, so no capstone test is
added. AGENT_VERSION bump (1.7.0) landed in step 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 19:07:53 -04:00
bvandeusenandClaude Opus 4.8 414c8efa98 fix(test): avoid autobegin-vs-begin collision in the log toggle/exclude test [M79 step 5]
CI / lint (push) Waiting to run
CI / unit (push) Waiting to run
CI / integration (push) Waiting to run
CI / publish (push) Blocked by required conditions
test_persist_logs_respects_toggle_and_exclude read via s.execute(SELECT) between
two `async with s.begin()` blocks; that SELECT autobegins a transaction, so the
next begin() raised "A transaction is already begun". Move both reads after the
final begin block (the toggle-off persist stores nothing, so end-state still
reflects the exclude phase), and pass a SimpleNamespace stand-in host to sidestep
ORM attribute-expiry across the commit boundaries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 19:04:25 -04:00
bvandeusenandClaude Opus 4.8 8f1c8c5cf7 feat(docker): container log viewer + Settings controls [M79 step 5]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 41s
CI / integration (push) Failing after 2m24s
CI / publish (push) Has been skipped
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.

Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
  every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
  text search; newest-first so the latest lines survive the poll's scroll reset;
  empty/loading states; a "View logs" link from the container detail page.
  Jinja auto-escapes log content (no markup injection).

Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
  per container — DB-backed, no restart. The toggle + exclude are enforced
  authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
  the push model has no channel to tell an agent to stop.

Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 18:58:20 -04:00
bvandeusenandClaude Opus 4.8 07a841d91e feat(docker): per-container size+age log ring rotation [M79 step 4]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 41s
CI / integration (push) Successful in 2m26s
CI / publish (push) Successful in 59s
Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.

- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
  max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 18:51:12 -04:00
bvandeusenandClaude Opus 4.8 a8de3570fe feat(docker): persist pushed container logs into docker_logs [M79 step 3]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 41s
CI / integration (push) Successful in 2m25s
CI / publish (push) Successful in 8s
Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.

- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
  records (drops the _steward truncation marker, malformed + lineless records,
  normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 18:46:31 -04:00
bvandeusenandClaude Opus 4.8 45565b2c01 feat(docker): docker_logs table + docker_009 migration [M79 step 2]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 40s
CI / integration (push) Successful in 2m20s
CI / publish (push) Successful in 7s
Dedicated host-scoped table for pushed container log lines (one row per line),
chained after docker_008. host_id FK CASCADE; twin indexes — (host_id,
container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring
docker_events. Integration schema-shape test asserts columns + both indexes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 18:43:14 -04:00
bvandeusenandClaude Opus 4.8 c95194747d feat(host-agent): collect container logs (incremental push, byte-capped) [M79 step 1]
CI / lint (push) Successful in 7s
CI / unit (push) Successful in 43s
CI / integration (push) Successful in 2m26s
CI / publish (push) Successful in 1m29s
Agent-side of Docker container-log collection (milestone 79). Each interval the
agent fetches new log lines per running container over the local Docker socket,
demuxes the multiplexed stream, and folds sample["docker_logs"] into the same
push as metrics — no inbound channel needed.

- since-cursor per container kept in rate-state; a container's first interval
  seeds from a short tail, then fetches incrementally via ?since=<cursor>
- _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib,
  TTY/unframed fallback); one row per (stream, ts, line)
- per-batch byte cap with a truncation marker; logs are dropped (never buffered)
  across a backoff so an outage keeps metrics but not stale logs
- docker_logs_enabled (default on) + docker_log_exclude per-host config keys
- AGENT_VERSION 1.6.0 -> 1.7.0

Server ignores the new sample key until the ingest step lands, so this commit is
CI-safe on its own.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 18:38:06 -04:00
bvandeusenandClaude Opus 4.8 a9b3b11327 feat(docker): admin-gated per-host prune buttons on the image/disk page
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 2m27s
CI / unit (push) Successful in 48s
CI / publish (push) Successful in 1m10s
M78 step 2. The /disk page surfaced reclaimable space + stopped counts but
said "prune deferred to a later milestone" — this wires the cleanup.

Adds POST /plugins/docker/disk/<host_id>/prune (admin only) that resolves
the host's linked AnsibleTarget → steward:target:<id> scope and fires the
bundled maintenance/docker_prune.yml through the ansible.run_playbook
capability (audited runner; the collection agent stays read-only). Mirrors
the host_agent update route. `target` ∈ {containers, images, system}:
- containers → docker container prune -f
- images     → docker image prune -af (prune_all_images)
- system     → docker system prune -f (conservative)

The disk() route now resolves each host group's linked target + capability
availability; disk.html renders three confirm-gated buttons per host, shown
only to admins when Ansible is available and the host has a linked target,
otherwise a clear "link a target" / "runner unavailable" hint (rules 24/26/27).

Tests: the prune_target→extra_vars mapping (the -af-only-for-images rule) is
extracted to a pure helper and unit-tested; disk_prune added to the module
smoke; disk.html covered by the existing template-parse test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 16:22:30 -04:00
bvandeusenandClaude Opus 4.8 e8ac99174a feat(ansible): parameterize docker_prune with prune_target (containers/images/system)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 44s
CI / integration (push) Successful in 2m36s
CI / publish (push) Successful in 1m11s
M78 step 1. The bundled maintenance/docker_prune.yml was a single
`docker system prune -f`; the upcoming per-host prune buttons need a
granular split. Add a `prune_target` extra-var:
- containers → docker container prune -f
- images     → docker image prune -f (+ -a when prune_all_images)
- system     → docker system prune -f (+ -a / --volumes) — the DEFAULT,
               so existing manual/scheduled callers (which set no var)
               keep today's behavior.

Reuses the existing bundled playbook rather than adding parallel files;
keeps the description/category/confirm meta so it still self-describes as
a confirm-gated maintenance run. Adds unit coverage: the playbook stays
valid YAML, its meta still discovers, and all three targets are guarded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 16:09:39 -04:00
bvandeusenandClaude Opus 4.8 01f5805139 fix(ci): serialize CI per ref + tag images at build time (fd race #1093)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 43s
CI / integration (push) Successful in 2m29s
CI / publish (push) Successful in 1m7s
The publish lane raced on the runner's shared docker daemon when two dev
pushes landed seconds apart: a concurrent run evicted the freshly-built
:<sha> image mid-push, so the post-push `docker tag :<sha> :dev` failed
with "No such image" (a flake — a lone re-run went green).

Two guards, both matching CI-runner's canonical build workflows:
- Workflow-level `concurrency: ci-${{ github.ref }}` with
  cancel-in-progress:false serializes runs per ref, so no two publishes
  ever share the daemon. false (not true) because rule 46 requires every
  push to emit its own immutable :<sha> image — a superseded run must
  still finish.
- Build both the :<sha> and moving (:dev/:latest) tags in one
  `docker build -t ... -t ...`, then push each. Removes the fragile
  post-push `docker tag` of an image a concurrent run could have evicted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 12:52:29 -04:00
bvandeusenandClaude Opus 4.8 524fd1f509 feat(ansible): default source/playbook dropdowns to the first item
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m18s
CI / publish (push) Successful in 1m0s
The schedule and host run forms opened their Source/Playbook <select>s on a
"— choose … —" placeholder, forcing an extra click. Default them to the first
item and cascade the dependent fields so the initial state is real:

- Remove the placeholder options (schedules source+playbook, host source+
  playbook, and the shared _playbook_options fragment).
- schedules(): pass sel_source (first source, or the edited schedule's) and
  that source's pre-rendered playbooks so the playbook dropdown starts populated
  (create mode too, no longer the all-sources union).
- Cascade on load/change: the source <select> fires playbook-options then
  (hx-on::after-settle) re-triggers the playbook <select>, which loads that
  playbook's variable fields. Host form drives the whole chain from a load
  trigger; schedule create mode loads vars via the playbook <select>'s own load
  trigger, while edit mode keeps its server-rendered prefill untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 23:35:01 -04:00
bvandeusenandClaude Opus 4.8 efc11c1837 feat(ansible): structured playbook-variable fields on the schedule form
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 45s
CI / integration (push) Successful in 2m20s
CI / publish (push) Successful in 1m14s
The Ansible Schedules form only offered a free-form extra-vars textarea,
unlike the browse/host run forms which auto-populate a field per declared
playbook variable (vars:/vars_prompt:) plus the playbook description. Wire
the schedule form to the same shared `_playbook_vars.html` partial:

- Playbook <select> now loads /ansible/playbook-vars?schedule=1 on change
  into a #sch-playbook-vars container (mirrors the host detail form).
- Edit mode pre-renders the saved playbook's variables server-side with
  their stored values; the extra-vars textarea keeps only leftover
  (non-declared) vars.
- Partial gains optional `values` (prefill) and `schedule` flags. In the
  schedule context secret vars render disabled ("secrets can't be
  scheduled") since they're dropped downstream, and the destructive-confirm
  gate is shown but not required.

Backend already parsed var__* fields for schedules, so this is the
front-end/partial parity fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 23:03:54 -04:00
bvandeusenandClaude Opus 4.8 f40063a74d feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m19s
CI / publish (push) Successful in 1m3s
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.

- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
  open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
  minute through the normal alert pipeline, so the operator can alert on them
  via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
  zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
  off Linux. Registered as a core ScheduledTask in app.py.

- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
  that skips a tick while a task's prior run is still in flight, so a hung poll
  can't stack overlapping runs (which amplify per-poll resource/fd use). A
  skipped task isn't penalised — it retries the next tick after it completes.

- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
  metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
  and asserts /proc/self/fd doesn't grow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 10:36:53 -04:00
bvandeusenandClaude Opus 4.8 0c5a1573da fix(unifi): close the cached client before discarding it (fd leak)
CI / unit (push) Successful in 47s
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 2m19s
CI / publish (push) Successful in 8s
The UniFi poller caches a UnifiClient holding a long-lived httpx.AsyncClient
(a real socket pool). On a login failure or a mid-poll error it set
`_client = None` to force re-auth, but never called close() — orphaning the
connection pool until GC. Under a flapping/unreachable controller that leaks
a file descriptor every failure tick: the same class of bug as the SNMP
engine leak (OSError: [Errno 24] Too many open files).

Add _drop_client() that aclose()s the client (best-effort) before nulling
the cache, and route both discard paths through it. Regression tests cover
both failure paths plus the noop/close-error contracts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 09:38:41 -04:00
bvandeusenandClaude Opus 4.8 2df5fc94a3 fix(snmp): close SnmpEngine after each poll to stop fd leak
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 45s
CI / integration (push) Successful in 2m20s
CI / publish (push) Successful in 1m2s
A fresh SnmpEngine() was created on every poll_device() call and never
closed. pysnmp opens a UDP transport socket per engine and doesn't release
it on GC, so each scheduler tick (default 60s, per device) leaked a file
descriptor. Over hours of polling the process hit its fd ceiling and the
listening socket could no longer accept connections — OSError: [Errno 24]
Too many open files on socket.accept(), locking up the app.

Wrap the engine in try/finally and release its transport socket via a new
_close_engine() helper that probes both pysnmp API shapes (6.2.x lextudio
camelCase, canonical 7.x snake_case); all close paths are best-effort so a
failed close never breaks the poll loop. Regression tests cover both shapes
and the never-raises contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 09:10:30 -04:00
bvandeusenandClaude Opus 4.8 b1aabb7dfa feat(snmp): explicit host binding (host_id / steward_host) with implicit fallback
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 45s
CI / integration (push) Successful in 2m18s
CI / publish (push) Successful in 1m2s
SNMP devices map onto a Steward host's detail page via _devices_for_host,
which previously matched the device's `host` field (the SNMP poll target)
against the Host's address/name — a coincidence of strings that breaks when
the poll target differs from how the Host is recorded.

Add two optional per-device config fields that decouple the binding from the
poll target:
  • host_id      — exact Steward Host UUID match (the explicit link)
  • steward_host — friendly bind by Host name or address (case-insensitive)

An explicit binding is exclusive: a device bound to host A never implicitly
matches host B by a coincidental address. No explicit binding → existing
implicit `host`-string match (fully backward compatible).

host_panel passes host.id and badges explicitly-bound devices. plugin.yaml
documents the new fields. Tests cover explicit id/name, exclusivity, wrong
uuid, and legacy fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-21 13:47:20 -04:00
bvandeusenandClaude Opus 4.8 24df8458ea feat(docker): swarm-aware container views — collapse replicas, show host, surface agent-less tasks
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 2m19s
CI / publish (push) Successful in 1m3s
CI / unit (push) Successful in 45s
The container list and dashboard widget listed each swarm task as its own cryptic
row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent.
Make both views service-centric and cluster-complete.

- swarm_view.build_swarm_services(): model-free builder that merges real container
  rows (collected local-per-node, so host_id = the node a task runs on) with the
  managers' placement (DockerSwarmService.placement_json). Where placement counts
  exceed the local rows on a node — i.e. nodes with no agent — it synthesizes
  "ghost" replicas so the replica list matches the cluster. Non-swarm containers
  pass through grouped by host.
- /rows + rows.html: a Swarm-services panel collapses each service to one block,
  every replica a host chip (status + cpu); ghosts render dashed "N · no agent".
  Non-swarm/compose containers keep the per-host table below. New Services stat.
- /widget + widget.html: same service collapse with host chips; standalone
  containers stay grouped by host.
- Unit tests for the builder (real+ghost merge, full coverage, partial agent-less,
  synthesized service, down state).

No migration — node_id, placement_json and swarm-node hostnames are already stored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-21 12:50:15 -04:00
bvandeusenandClaude Opus 4.8 431f804037 fix(docker): move dedup helper to a model-free module (CI green)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 2m18s
CI / unit (push) Successful in 45s
CI / publish (push) Successful in 1m2s
The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.

Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-20 23:07:33 -04:00
bvandeusenandClaude Opus 4.8 e289b6c49f fix(docker): dedup widget counts across swarm managers; show failures not stopped
CI / publish (push) Has been skipped
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 46s
CI / integration (push) Failing after 2m19s
The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.

- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
  (globally unique, so only true duplicates merge); first occurrence wins under
  the running-first ordering. Applied in both the containers and resources
  widgets before counting/limiting. Rows without a container_id (older agents)
  are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
  distinct containers with a die/oom event in the last 24h (DISTINCT so the
  managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
  all kept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-20 23:02:19 -04:00
bvandeusenandClaude Opus 4.8 0c055ed6fa fix(docker): reap vanished containers; clarify the resources widget
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 2m20s
CI / unit (push) Successful in 44s
CI / publish (push) Successful in 1m0s
The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.

- ingest.py: after the upsert loop, delete this host's containers whose name is
  notin the newest snapshot (the listing is authoritative — event derivation
  already treats absence as removal). Mirrors images/swarm/disk. The existing
  `if not snapshots: return` keeps swarm/disk-only samples from touching
  containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
  Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
  % value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-20 22:06:43 -04:00
bvandeusenandClaude Opus 4.8 51682f130a feat(snmp): surface SNMP device readings on the matching host page
CI / lint (push) Successful in 3s
CI / publish (push) Successful in 53s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m17s
SNMP devices are config-defined by IP/hostname, not Steward Host records, so
they had no presence on a host's page. Map them by address and embed a fragment
(mirrors the Docker per-host fragment).

- _devices_for_host(devices_cfg, address, name): case-insensitive match of a
  device's configured host to the Steward host's address or name (tolerates
  non-dict / host-less entries).
- Route /plugins/snmp/host/<id>: renders the matched device(s) + latest readings,
  or nothing when none map (so hosts without SNMP carry no empty card).
- snmp/host_panel.html: per-device card (name · address · reachability) with a
  readings grid (K/M scaling) and a History link to the full device page.
- hosts/detail.html embeds it after the Docker fragment, gated on snmp enabled.
- Unit test for the matching helper (by address, by name, no-match, blank).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-20 21:22:39 -04:00