dc22a35e9d34abb6f4b398a731bed4ab408d658a
92
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5.5
|
b009a8dd30 |
test(alerts): fix the silent-finding time and keep cpu_pct first in the host_agent catalog (#5062)
CI 8325: NOW - 10 min is 11:50, not 09:50; and rule_fields' refusal lists the catalog in order, so seconds_since_seen goes last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
c368e93e2c |
feat(alerts): silence is alertable — agent offline and no-data (#5062)
Agent offline is an ordinary rule now. The host_agent scheduler records host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or registration time for an agent that never reported), replacing the task that only logged stale agents. The catalog offers the metric, and picking it on a new rule fills "> 180". No data: each rule's state tracks its last sample and its usual gap, smoothed so that an outage counts as at most three gaps. A core task, alert_silence_check, runs every 60s and marks a rule no_data_since once its series has had no sample for three usual gaps (at least 5 minutes; an hour while the cadence is unknown). A never-sampled rule counts from when it started watching. The mark sits beside the state, so a firing alert that goes silent stays firing. Each silence records one event and sends one notification (rule option notify_no_data, default on; muted in maintenance windows). The next sample clears it with a "data resumed" event and notification. Creating, changing, disabling or re-enabling a rule restarts its watch. Migration 0029 adds the columns and seeds each existing rule's last sample and gap from its last day of data, so an upgrade does not flag every rule. Shown on the Alerts page (a NO DATA badge with its start time), in MCP list_alerts, list_alert_rules and get_host (state "no_data", no_data_since), in get_overview findings (alert_no_data, or a firing alert marked silent) and in the weekly report. The rule form has the notify option, and its metric reference now matches what traefik records and lists host_agent. Emails say "no data" and carry the event's note. The metric-catalog guard also reads positional MetricBatch.add arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
0742b7a2b7 |
feat(alerts): the metric catalog is written by what emits it (#5061)
#5061 "Metric catalog belongs to what emits: plugins register their metrics; names match what is recorded". - The static METRIC_CATALOG is gone. metric_catalog() is core's own sources (the four monitor types and "steward", the app's open_fds/open_fds_pct, which were recorded but never offered) plus each loaded plugin's get_alert_metrics() export. A disabled plugin's sources leave the picker. - traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct. The old "error_rate" was never recorded, so a rule on it could not fire. - tests/core/test_metric_catalog.py walks every MetricBatch.add / record_metric call (traefik's tuple loop included) and fails on a recorded metric the catalog lacks; it shows on a synthetic writer that it can fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
58bfc7c361 |
feat(alerts): one evaluation path; host-agent ingest fires rules (#5060)
#5060 "One evaluation path: record_metrics batch API; host-agent ingest evaluates rules". - record_metrics(session, rows) writes metric rows and evaluates the rules that watch them, reading rules, states and maintenance windows once per batch. Each rule sees its key's samples oldest first, so an agent catching up after an outage counts consecutive checks as they happened. - Host-agent ingest now goes through it. Before, it wrote plugin_metrics directly and no host_agent rule could ever fire. The catalog gains the per-mount disk_used_pct ("<host>:<mount>"). - MetricBatch gathers a transaction's samples; the docker ingest, traefik, unifi, snmp, monitor and self-monitor writers record one batch each instead of a rule query per sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
faa605cead |
feat(secrets): webhook URL and git tokens sealed; run output masks its secrets (#5037)
#5037 "Secrets at rest and in run output: webhook URL and git token encrypted, askpass via env, secret values scrubbed from logs". - webhook.url joins SECRET_KEYS; existing values are sealed at startup by migrate_plaintext_secrets. Settings -> Notifications no longer echoes the URL: it names the saved host, keeps it when left blank, and has a Remove box. - Git source http_token is sealed inside ansible.sources on save and at startup, opened in get_sources, and counted by the undecryptable-secrets scan as ansible.sources[<name>].http_token. - The askpass script holds no token: git runs it with STEWARD_GIT_TOKEN in its environment and printf prints it as data. - The executor masks every secret the run is handed (become/vault passwords, bootstrap passwords, sensitive run vars, and the per-host agent tokens the host_agent plugin now passes as redact) in each line before it is broadcast, logged or stored. - Bundled "Write agent config" tasks are no_log; a test fails if any bundled task templating steward_token lacks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
6a1bc437c9 |
feat(auth): share links open only their own widgets; unmarked routes are closed (#5033)
#5033 "Share links open only their own dashboard; every route without a role is on an explicit allow-list". Any live share token in ?s= made the caller a viewer on every viewer route — host pages, container logs, any dashboard — and the host-agent widget routes had no role check at all. - steward/dashboard/share.py: a share token opens, read-only, exactly the widget fragment URLs its own dashboard loads — same path, same configured parameters; only the time window (hours/range) may change. - require_role marks its view; public_route marks the deliberately open ones (login, setup, health, the share page, agent ingest, install.sh, agent.py) with the reason. A before_request refuses any view with neither, so a route that forgets its decorator is closed, not open. - /plugins/host_agent/widget and /widget/history now require viewer. - The history widget's range buttons keep the rest of the query (and so a share link's token) instead of rebuilding it; the uptime widget's "Full SLA" link shows to logged-in users, not to share viewers. - tests/test_route_access.py: a source scan that fails on any unmarked route in core or bundled plugins, the default-deny refusal, and the share matcher's cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
96126b1c83 |
feat(auth): forged browser requests are refused (#5031)
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token on every non-GET, HTMX header". A page on another site could make a logged-in browser POST to Steward with the session cookie attached. - steward/auth/csrf.py: a per-session token, checked in one before_request for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field csrf_token, constant-time compare). The bearer-authenticated host-agent ingest is exempt; /mcp never reaches Quart. - Every POST form renders {{ csrf_field() }}; base.html carries the token in a meta tag and adds it to every HTMX request; the dashboard's fetch() calls send it. - The session cookie is SameSite=Lax; login issues a fresh token. - PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set when Settings → General was saved). - tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption, and a guard that every POST form in the templates carries the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
190a5d1e80 |
feat(docker): a removed container's logs and events stay reachable by name
#5010 "Logs of a removed container cannot be reached by name" - get_container and search_container_logs fall back to the names Steward still holds logs or events for when no live container matches: the exact name from logs and events, or a part of it that matches one removed container's events. - get_container answers with removed: true, last_seen_at, its history and events; search_container_logs marks container_removed and notes when it was last seen, and says when that is older than the range. - A miss now says the container may have been removed, that Swarm replaces a dead task under a new name, and where to look instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
e9341c5639 |
feat(docker): overview findings say when the problem began
#5013 "Overview findings for containers and Swarm services carry since=null" - Container problems are dated from their own evidence: a crash or restart loop from its first death in the hour, an OOM kill or error exit from the death that ended it, a failing healthcheck from its last turn to unhealthy. list_containers and get_container carry problem_since / problem_age_s. - Swarm services carry trouble_since: a crash loop's first failure in the hour, a short service's oldest troubled task in each failing slot, a recovered service's first failure in 24h. list_swarm_services shows it. - get_overview findings from the docker plugin now have since and age_s, so they rank by how long they have stood. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
bac2859455 |
feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
680a6bf3b6 |
feat(mcp): automation tools — playbooks, runs, schedules, alert actions, docker prune
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions". Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory (variable names only), list_ansible_runs, get_ansible_run (log tail + grep), list_schedules, wait_for_run (polls, 120s cap). Operator: run_playbook (check mode unless check=false; a steward:confirm playbook applies only with confirmed=true; secret variables reach the run but are never stored, audited or returned), cancel_ansible_run, and create/update/delete_schedule (secrets refused, check by default). Admin: set_alert_action, and the docker plugin's prune_docker, which previews from the disk report unless check=false (Ansible's check mode skips the prune commands, so a check run would show nothing). Every write is audited as "<user> via MCP" with the token id; the audit tool's user filter includes those rows. prune_extra_vars moves to the docker service so the page and the tool share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
21338dbcd9 |
feat(mcp): docker and host-agent tools; TLS and crash-loop cutoffs in Settings
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". Contributed through each plugin's register_mcp_tools hook, so they exist only while the plugin is enabled. docker (viewer, read-only): - list_containers: filter by host, status, compose project, or problem (restarting, crash_looping, unhealthy, oom_killed, exited_nonzero); problems sort first and each row names what is wrong. - get_container: state, CPU/memory history across the raw/hourly seam, lifecycle events. A unique part of a swarm task name is enough. - list_swarm_services: replicas per node (agent-less placements as ghosts) and the swarm's nodes. - search_container_logs: text (% and _ literal), stream, range, context lines, paging by a full-precision `before` cursor; over-long lines keep both ends and say how much of the middle was cut. - get_docker_disk_usage: bytes per host, reclaimable first. - mcp_overview: failing containers and under-replicated swarm services join get_overview. host_agent: list_agents (reporting / stale / pending, version, OS), and an overview finding for agents registered over 30 minutes ago that never checked in. Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in code) and the crash-loop deaths-per-hour count (3), which get_overview and the docker tools now read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
312ea95bcf |
refactor: move host, docker, alert and monitor queries into service modules
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable, LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so the queries they share with the pages now live in plain functions that take an open session: - steward/hosts/service.py: uptime, status, monitors, agent metrics, agent overview (stale threshold passed in, not read from current_app), CPU sparks. - plugins/docker/service.py: bucket, container_history, host_map, list_containers, container_events, query_logs. - steward/alerts/service.py: active_alerts (also used by core/reports), list_rules, list_maintenance_windows. - steward/monitors/service.py: monitor_rows (was routes._rows_data). Routes are repointed and render the same data. One behavior fix rides along: the container-log search now escapes % and _ so they match literally instead of acting as LIKE wildcards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
8f1c8c5cf7 |
feat(docker): container log viewer + Settings controls [M79 step 5]
The user-facing half (rule 27). Per-container log viewer + admin controls, so container logs are something the operator can actually touch. Viewer (plugins/docker): - /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive text search; newest-first so the latest lines survive the poll's scroll reset; empty/loading states; a "View logs" link from the container detail page. Jinja auto-escapes log content (no markup injection). Settings (Thresholds & Retention tab): - global on/off toggle, comma-separated exclude list, retention days + max MB per container — DB-backed, no restart. The toggle + exclude are enforced authoritatively at ingest (_persist_logs drops disabled/excluded lines), since the push model has no channel to tell an agent to stop. Tests: route-defined smoke; template-parse covers the new templates; integration test for the ingest-time toggle + exclude enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
07a841d91e |
feat(docker): per-container size+age log ring rotation [M79 step 4]
Bound docker_logs growth in the periodic cleanup task (same architecture as the metrics/events retention). run_docker_retention gains logs_retention_days + logs_max_bytes_per_container: it prunes lines past the age window, then keeps only the newest ~cap bytes per (host, container) via a window-function ring (exclusive-prefix sum, so the newest line always survives even if it alone exceeds the cap). Containers rotate independently. - settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/ max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB - cleanup.py reads the two windows fresh each run (rule 25, no restart) - integration rotate test: age prune + per-container byte cap + isolation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
a8de3570fe |
feat(docker): persist pushed container logs into docker_logs [M79 step 3]
Ingest side of container logs. The host_agent ingest route extracts sample["docker_logs"] as time-series batches (append every line, each carrying its own Docker ts with recorded_at as fallback) and hands them to the docker capability alongside the existing container/swarm/disk data — still under the begin_nested SAVEPOINT so a logs failure can't sink host metrics. - ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters records (drops the _steward truncation marker, malformed + lineless records, normalises unknown stream → stdout); persist_host_docker gains a logs= param - routes.py: accumulate docker_log_batches, add to the guard + capability call - unit test for _log_rows shaping/filtering; integration push→store→query test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
45565b2c01 |
feat(docker): docker_logs table + docker_009 migration [M79 step 2]
Dedicated host-scoped table for pushed container log lines (one row per line), chained after docker_008. host_id FK CASCADE; twin indexes — (host_id, container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring docker_events. Integration schema-shape test asserts columns + both indexes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
c95194747d |
feat(host-agent): collect container logs (incremental push, byte-capped) [M79 step 1]
Agent-side of Docker container-log collection (milestone 79). Each interval the agent fetches new log lines per running container over the local Docker socket, demuxes the multiplexed stream, and folds sample["docker_logs"] into the same push as metrics — no inbound channel needed. - since-cursor per container kept in rate-state; a container's first interval seeds from a short tail, then fetches incrementally via ?since=<cursor> - _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib, TTY/unframed fallback); one row per (stream, ts, line) - per-batch byte cap with a truncation marker; logs are dropped (never buffered) across a backoff so an outage keeps metrics but not stale logs - docker_logs_enabled (default on) + docker_log_exclude per-host config keys - AGENT_VERSION 1.6.0 -> 1.7.0 Server ignores the new sample key until the ingest step lands, so this commit is CI-safe on its own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
a9b3b11327 |
feat(docker): admin-gated per-host prune buttons on the image/disk page
M78 step 2. The /disk page surfaced reclaimable space + stopped counts but
said "prune deferred to a later milestone" — this wires the cleanup.
Adds POST /plugins/docker/disk/<host_id>/prune (admin only) that resolves
the host's linked AnsibleTarget → steward:target:<id> scope and fires the
bundled maintenance/docker_prune.yml through the ansible.run_playbook
capability (audited runner; the collection agent stays read-only). Mirrors
the host_agent update route. `target` ∈ {containers, images, system}:
- containers → docker container prune -f
- images → docker image prune -af (prune_all_images)
- system → docker system prune -f (conservative)
The disk() route now resolves each host group's linked target + capability
availability; disk.html renders three confirm-gated buttons per host, shown
only to admins when Ansible is available and the host has a linked target,
otherwise a clear "link a target" / "runner unavailable" hint (rules 24/26/27).
Tests: the prune_target→extra_vars mapping (the -af-only-for-images rule) is
extracted to a pure helper and unit-tested; disk_prune added to the module
smoke; disk.html covered by the existing template-parse test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
|
||
|
bvandeusenandClaude Opus 4.8
|
0c5a1573da |
fix(unifi): close the cached client before discarding it (fd leak)
The UniFi poller caches a UnifiClient holding a long-lived httpx.AsyncClient (a real socket pool). On a login failure or a mid-poll error it set `_client = None` to force re-auth, but never called close() — orphaning the connection pool until GC. Under a flapping/unreachable controller that leaks a file descriptor every failure tick: the same class of bug as the SNMP engine leak (OSError: [Errno 24] Too many open files). Add _drop_client() that aclose()s the client (best-effort) before nulling the cache, and route both discard paths through it. Regression tests cover both failure paths plus the noop/close-error contracts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
2df5fc94a3 |
fix(snmp): close SnmpEngine after each poll to stop fd leak
A fresh SnmpEngine() was created on every poll_device() call and never closed. pysnmp opens a UDP transport socket per engine and doesn't release it on GC, so each scheduler tick (default 60s, per device) leaked a file descriptor. Over hours of polling the process hit its fd ceiling and the listening socket could no longer accept connections — OSError: [Errno 24] Too many open files on socket.accept(), locking up the app. Wrap the engine in try/finally and release its transport socket via a new _close_engine() helper that probes both pysnmp API shapes (6.2.x lextudio camelCase, canonical 7.x snake_case); all close paths are best-effort so a failed close never breaks the poll loop. Regression tests cover both shapes and the never-raises contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
b1aabb7dfa |
feat(snmp): explicit host binding (host_id / steward_host) with implicit fallback
SNMP devices map onto a Steward host's detail page via _devices_for_host, which previously matched the device's `host` field (the SNMP poll target) against the Host's address/name — a coincidence of strings that breaks when the poll target differs from how the Host is recorded. Add two optional per-device config fields that decouple the binding from the poll target: • host_id — exact Steward Host UUID match (the explicit link) • steward_host — friendly bind by Host name or address (case-insensitive) An explicit binding is exclusive: a device bound to host A never implicitly matches host B by a coincidental address. No explicit binding → existing implicit `host`-string match (fully backward compatible). host_panel passes host.id and badges explicitly-bound devices. plugin.yaml documents the new fields. Tests cover explicit id/name, exclusivity, wrong uuid, and legacy fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
24df8458ea |
feat(docker): swarm-aware container views — collapse replicas, show host, surface agent-less tasks
The container list and dashboard widget listed each swarm task as its own cryptic row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent. Make both views service-centric and cluster-complete. - swarm_view.build_swarm_services(): model-free builder that merges real container rows (collected local-per-node, so host_id = the node a task runs on) with the managers' placement (DockerSwarmService.placement_json). Where placement counts exceed the local rows on a node — i.e. nodes with no agent — it synthesizes "ghost" replicas so the replica list matches the cluster. Non-swarm containers pass through grouped by host. - /rows + rows.html: a Swarm-services panel collapses each service to one block, every replica a host chip (status + cpu); ghosts render dashed "N · no agent". Non-swarm/compose containers keep the per-host table below. New Services stat. - /widget + widget.html: same service collapse with host chips; standalone containers stay grouped by host. - Unit tests for the builder (real+ghost merge, full coverage, partial agent-less, synthesized service, down state). No migration — node_id, placement_json and swarm-node hostnames are already stored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
431f804037 |
fix(docker): move dedup helper to a model-free module (CI green)
The widget-dedup test imported plugins.docker.routes, which re-imports plugins.docker.models after the app already loaded the docker plugin — "Table 'docker_containers' is already defined". Same plugin-loader gotcha the host_agent query helpers avoid. Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM models, so it's safe to import in either environment); routes.py imports it from there and the test targets the model-free module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
e289b6c49f |
fix(docker): dedup widget counts across swarm managers; show failures not stopped
The running count was inflated: swarm-aware agents on every manager list cluster-wide tasks, so the same container (identical container_id) was reported once per manager and the dashboard widgets summed them. The earlier swarm dedup only covered the Swarm topology page — the widgets read raw rows. - _dedup_by_container_id(): collapse rows sharing a non-empty container_id (globally unique, so only true duplicates merge); first occurrence wins under the running-first ordering. Applied in both the containers and resources widgets before counting/limiting. Rows without a container_id (older agents) are kept as-is. - Replaced the static "stopped" tally — not actionable — with "failed (24h)": distinct containers with a die/oom event in the last 24h (DISTINCT so the managers' duplicate events don't double it), rendered red when non-zero. - widget.html empty-state now keys off total_count. - Regression test: same container_id from two managers counts once; id-less rows all kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
0c055ed6fa |
fix(docker): reap vanished containers; clarify the resources widget
The container persister upserted current state keyed (host_id, name) but — unlike the image/swarm/disk persisters — never deleted containers that vanished from the host's latest listing. Every removed one-shot container (CI job runners, buildkit builders, codex jobs) left a permanent "stopped" row, so the dashboard counts ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was a missing reaper. - ingest.py: after the upsert loop, delete this host's containers whose name is notin the newest snapshot (the listing is authoritative — event derivation already treats absence as removal). Mirrors images/swarm/disk. The existing `if not snapshots: return` keeps swarm/disk-only samples from touching containers; an empty container snapshot legitimately means none exist. - widget_resources.html: this widget shows the busiest running containers by CPU. Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar · % value, warn/crit colors) so it's self-explanatory. - Regression test: a container present then absent across snapshots is reaped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
51682f130a |
feat(snmp): surface SNMP device readings on the matching host page
SNMP devices are config-defined by IP/hostname, not Steward Host records, so they had no presence on a host's page. Map them by address and embed a fragment (mirrors the Docker per-host fragment). - _devices_for_host(devices_cfg, address, name): case-insensitive match of a device's configured host to the Steward host's address or name (tolerates non-dict / host-less entries). - Route /plugins/snmp/host/<id>: renders the matched device(s) + latest readings, or nothing when none map (so hosts without SNMP carry no empty card). - snmp/host_panel.html: per-device card (name · address · reachability) with a readings grid (K/M scaling) and a History link to the full device page. - hosts/detail.html embeds it after the Docker fragment, gated on snmp enabled. - Unit test for the matching helper (by address, by name, no-match, blank). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
8af297670e |
feat(metrics): roll plugin_metrics up to hourly to bound storage
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
b0d3e83bdd |
feat(hosts): true live vitals strip; agent panel becomes management-only
Realizes the chosen host-summary layout: a thin live vitals bar at the very top, separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD). - New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) + Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing until the agent reports. - The metric computation (latest snapshot + 6h sparkline query + load/core + PSI) moves from host_panel into host_vitals. host_panel slims to management only (reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html drops the gauge row + pressure block, keeping status + lifecycle actions. - hosts/detail.html: vitals strip on top (full width, live), then a 2-col [Monitors | Agent] grid, Ansible full width below, docker fragment last. UI only. Templates parse; plugin-template parse test covers the new fragment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
28c9a4dd2f |
fix(docker): collapse Swarm view per-cluster so multi-manager doesn't duplicate
Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.
Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.
View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
10dfd8ffd2 |
fix(host_agent): update full-metrics charts in place to kill refresh flicker
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
9e4f1983f8 |
feat(host_agent): lazy-load full-metrics charts + live-poll current state
The full-metrics page rendered once server-side with the history query inline, so the charts blocked first paint and nothing refreshed without a reload (it reads the latest snapshot the server holds — the agent pushes ~every 30s). Split it into a shell + two HTMX fragments: - Shell (host_detail.html): header + shared time-range toggle + two containers; paints instantly. - Current state (/<id>/metrics → _host_metrics.html): identity + gauges + per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence. - History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js, from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so they lazy-load (never block paint), refresh slowly, and follow the range selector. Leak-safe: previous Chart instances are destroyed before re-render. - Range toggle switched from full-reload links to the shared _time_range.html (setTimeRange + rangeChange), so only the fragments refetch. - routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a _split_host_metrics helper. - tests/test_templates_parse.py now also parses plugin templates (these fragments aren't rendered in the unit lane). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
a78af23793 |
refactor(host_agent): extract metric-query helpers to a model-free module
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
aff0c36d37 |
perf(host_agent): aggregate metric history in SQL; DISTINCT ON for latest
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention- bounded (core cleanup prunes > data.retention_days, default 90d) and charts top out at 30d, so the cost wasn't growth — it was the read path shipping raw rows to Python. - _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120 buckets) instead of fetching every raw sample (a 30d range was hundreds of thousands of rows) and downsampling in Python. Uses the new (source_module, resource_name, recorded_at) index. - _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY recorded_at DESC — newest row per group in one index-ordered pass, replacing the GROUP-BY-max subquery self-joined back to the whole history. - Integration test validates both against Postgres. Deliberately not a materialized raw→hourly rollup: these query-side changes deliver the speed; a rollup would additionally cut storage and remains a future option if scale demands it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
7d144780df |
fix(host_agent): TB/PB byte scaling + rebalance the full-metrics layout
Two issues on the host full-metrics view: - fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of "24.8 TB". Add TB and PB tiers. - Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards stretched to the Temperatures height — a 40-core CPU made one very tall column and left Interfaces/Disks with large empty space. Split them: Interfaces + Disks sit side-by-side at natural height (align-items:start); Temperatures becomes a full-width card whose readings flow into a compact multi-column grid (wide-and-short instead of one tall column). UI-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
2545e8c6ce |
fix(docker): widen memory byte columns to BIGINT (int32 overflow on ingest)
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes / mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM (e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32 range" and the entire docker ingest batch failed — no metrics stored for any host with a large container. The I/O counters and the milestone-77 rollup/disk tables already used BigInteger; this trio (docker_001-era) was missed. - models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes, DockerContainer.mem_limit_bytes → BigInteger. - migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place int4→int8 promotion). - integration regression test: persist an ~8 GB container, assert the current-state and time-series rows round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
c8b6719b37 |
feat(ui): plugin get_nav() hook → sidebar Infrastructure links (slice 2)
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an optional get_nav() plugin export and surface it in the sidebar. - plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant _collect_plugin_nav() (missing hook = fine; raising/malformed = logged & skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in both the startup load path and the hot-reload path. - app.py: inject plugin_nav into the template context, filtered to enabled plugins so a hot-disabled plugin's link can't linger before restart. - base.html: render plugin links under the Infrastructure group, with the same request.path active-state treatment as core links. - docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view. - Tests: collector behavior (collect, missing hook, reload-replace, malformed item, raising hook, sorted output). With docker enabled, "Docker" now appears under Infrastructure → its fleet view. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
4fc8c96c41 |
fix(snmp): bundle pysnmp in image and port poller to the asyncio HLAPI
The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP polling disabled" on every poll, so polling never worked. Two coupled defects: 1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp) was never bundled even though the plugin is first-party and shipped. 2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep present, polling would have thrown and silently returned nothing. The 5.x line that still has the sync API isn't safe on the image's Python 3.13. Fix: - Dockerfile: install `.[ansible,snmp]`. - poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI, with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd` + async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x `pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency bump can't silently re-break it. - scheduler.py: await poll_device directly; drop the run_in_executor wrapper and the now-unused asyncio import. - Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping, that the poller is a coroutine, and the graceful no-pysnmp path. Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target — live polling against real devices is verified after deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
88091936c5 |
fix(ui): unify header/breadcrumb treatment, drop redundant back buttons
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
9615f9abcd |
feat(docker): group containers by compose/swarm + enrich widget (milestone 77 #942)
Container list now sub-groups each host's containers by compose project (or swarm service) with a small subheading, preserving the running-first order; hosts with no such labels render flat as before. The dashboard status widget links each container to its detail page and surfaces enriched state inline — health dot (healthy/unhealthy), restart count, and last non-zero exit code for stopped containers. Completes the #942 UI surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
114262dbf9 |
feat(docker): swarm topology view + image/disk usage page (milestone 77 #942)
Two new read-only sub-pages, linked from the Docker index header only when the data exists (non-swarm / Docker-less installs aren't offered empty pages): - /plugins/docker/swarm — services with replica health (running/desired, colour-coded green/amber/red), Swarm nodes (role/availability/status, leader badge), and task→node placement with node ids resolved to hostnames. Grouped by reporting manager host. Empty state explains manager-only collection. - /plugins/docker/disk — per-host reclaimable space, image/layer/container/ volume/build-cache sizes, stopped-container count, and a per-image table (size, shared, ref count, reclaimable badge). Notes prune actions are deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
3e4e35de96 |
feat(docker): container detail page + lifecycle timeline (milestone 77 #942)
New /plugins/docker/container/<host_id>/<name> detail page (v1 quality): status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM), net + block I/O (humanised), image/compose/swarm-service/node/ports, a range-toggled CPU/mem history graph (HTMX fragment reusing the time-range selector), and a lifecycle timeline rendered from docker_events (glyph+colour per event kind). Not-found and empty-history/empty-timeline states included. Container names in the main list and the per-host hub panel now link to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
a840d6f823 |
feat(docker): collect + persist /system/df image/disk usage (agent 1.6.0)
Backend for the image/disk panel (milestone 77 #942). Agent gains collect_disk_usage() — one /system/df call (gated on containers existing, so Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by unreferenced images), layers/containers/volumes/build-cache sizes, and the top 50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks the newest sample's copy and hands it to the docker capability as a 5th arg. New current-state tables docker_disk_usage (one row/host) + docker_images (per-host image rows), docker_007 migration; ingest upserts the summary and replaces the image set per host (stale images pruned). Unit tests for the df parsing/reclaimable math + build_sample gating; integration test for persistence + image-set replacement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
faecac3ec6 |
feat(docker): retention + hourly rollup for metrics/events with Settings windows
Bounds Docker time-series growth (the main scaling concern). New docker_metrics_hourly table + docker_006 migration; a plugin retention module (docker.run_retention capability) rolls raw docker_metrics older than the raw window into hourly averages (idempotent upsert), deletes the rolled raw rows, then prunes stale rollups + lifecycle events. Core cleanup.py drives it each hourly run via the capability (no plugin-model import), reading the three retention windows fresh from settings so changes apply without restart (rule 25). Settings → "Thresholds & Retention" gains a Docker retention card (raw / rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the hour-aligned cutoff/bucketing helpers; integration test exercises the real rollup-average + prune across both windows. Milestone 77 task #941. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
578cc33cc0 |
feat(docker): ingest swarm topology + lifecycle events + health/restart alerts
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
448258c5b4 |
feat(docker): agent manager-only swarm collector (AGENT_VERSION 1.5.0)
Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:
* services roll desired-vs-running replicas up from the task list (replica
health isn't on the service object), handle replicated + global mode, strip
the @sha256 image digest, and carry cross-node task→node placement.
* nodes normalise role / availability / status + the manager leader flag.
build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
fee654b53a |
feat(docker): schema for lifecycle events + swarm topology
Adds the milestone-77 storage that doesn't fit on the per-container row:
* docker_events — lifecycle (start/stop/die/oom/health_change), to be
derived by diffing consecutive host snapshots; host-scoped, indexed for
timeline lookups (host_id, container_name, at) and retention pruning (at).
* docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
topology (desired-vs-running replicas, node role/availability/status).
Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
82c3d2cf36 |
feat(docker): per-container enrichment — health, restarts, exit code, I/O, grouping
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
7b80552a7d |
feat(docker): per-host collection via the host agent; drop central scrape
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
35f658b573 |
feat(monitors): unify ping/dns/http into one Monitor entity + custom targets
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|