bvandeusenandClaude Opus 5.5
6c4ef368f9
The root helper gains one read-only action, `drives`: it finds the drives itself (smartctl --scan-open), reads each as JSON, and writes a summary the agent carries on every sample (agent 1.20.0). Steward judges the figures in drives.py (failing / worn / watch / ok / no SMART / unreadable), records them per drive and as host roll-ups for covering rules, keeps the newest report on the registration and lists it on the host page. install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or apk, run by install.sh and every agent playbook; one it can't install is reported, never fatal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
671 lines
44 KiB
Markdown
671 lines
44 KiB
Markdown
# Host Agent Plugin — Design Spec
|
||
|
||
**Date:** 2026-04-14
|
||
**Status:** Approved, ready for implementation planning
|
||
**Scope:** Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.
|
||
|
||
---
|
||
|
||
## Goal
|
||
|
||
Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.
|
||
|
||
## Non-goals
|
||
|
||
- Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
|
||
- Not a log collector. Time-series metrics only.
|
||
- Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it.
|
||
- Not a network monitor. Traffic counters are deferred to a follow-up.
|
||
- Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────┐ POST /plugins/host_agent/ingest ┌────────────────────────┐
|
||
│ Agent │ ──────────────────────────────────────────→ │ Steward │
|
||
│ (Python) │ Authorization: Bearer <per-host-token> │ host_agent plugin │
|
||
│ on target │ JSON body: metrics snapshot │ │
|
||
│ host │ │ routes.py (ingest + │
|
||
│ │ ← 200 OK or 401/400 │ install.sh render + │
|
||
│ │ │ UI) │
|
||
│ systemd │ │ │
|
||
│ unit, │ │ writes to: │
|
||
│ dedicated │ │ - PluginMetric │
|
||
│ user, │ │ (time-series bus) │
|
||
│ 30s │ │ - HostAgentRegistr- │
|
||
│ interval, │ │ ation (plugin- │
|
||
│ in-memory │ │ private table) │
|
||
│ ring buffer │ │ │
|
||
└──────────────┘ │ registers widgets + │
|
||
│ METRIC_CATALOG entry │
|
||
└────────────────────────┘
|
||
```
|
||
|
||
### Core principles
|
||
|
||
- **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
|
||
- **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it.
|
||
- **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
|
||
- **Install** is a one-line `curl | sh` command rendered per-host by Steward with the token already baked in.
|
||
- **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data.
|
||
|
||
---
|
||
|
||
## Agent design
|
||
|
||
### File layout on the target host
|
||
|
||
```
|
||
/usr/local/lib/steward-agent/agent.py # the agent (runs as steward-agent)
|
||
/usr/local/lib/steward-agent/maint.py # the maintenance helper (runs as root, #5766)
|
||
/etc/steward-agent.conf # key=value config, 0640 root:steward-agent
|
||
/etc/systemd/system/steward-agent.service # the agent's unit
|
||
/etc/systemd/system/steward-maint.{path,service} # starts the helper on a request
|
||
/var/lib/steward-agent/requests/ # the one place the agent writes (0700 steward-agent)
|
||
/var/lib/steward-maint/results/ # the helper's outcomes (root, readable)
|
||
# dedicated system user: steward-agent
|
||
```
|
||
|
||
### Config file format
|
||
|
||
Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):
|
||
|
||
```
|
||
url = https://steward.home.lan
|
||
token = a1b2c3d4...
|
||
interval_seconds = 30
|
||
hostname = myhost # optional; defaults to uname -n
|
||
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
|
||
docker_logs_enabled = true # optional; on by default — per-host container-log kill-switch
|
||
docker_log_exclude = watchtower, noisy-svc # optional; container names whose logs this host skips collecting
|
||
```
|
||
|
||
Container-log collection is on by default (no config needed). The two keys above
|
||
are the per-host opt-outs; the operator-facing global toggle + exclude live in
|
||
Settings and are enforced server-side (the push model has no channel to command
|
||
an agent), so they don't require touching a host's conf.
|
||
|
||
### Agent internals (function list, not classes)
|
||
|
||
- `read_config(path)` — parses the conf file into a dict.
|
||
- `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float).
|
||
- `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun).
|
||
- `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`.
|
||
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
|
||
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
|
||
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
|
||
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
|
||
container, fetches new log lines over the Docker socket with an incremental
|
||
`since` cursor kept per container in `state` (a container's first interval
|
||
seeds from a short tail). Demuxes the multiplexed stream, parses each line's
|
||
RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
|
||
a truncation; the deferred lines come next interval). Folded into the sample as
|
||
`docker_logs`; omitted when empty.
|
||
- `build_payload()` — assembles a snapshot from all collectors into one dict.
|
||
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
|
||
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
|
||
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).
|
||
|
||
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
|
||
|
||
### Dependencies
|
||
|
||
Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries.
|
||
|
||
### Signals and lifecycle
|
||
|
||
- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly).
|
||
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time, with a `stopping` event saying whether the host is going down too (`systemctl is-system-running` says `stopping`) → exit.
|
||
- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`.
|
||
|
||
### Mounts that don't answer (1.14.0, #5760)
|
||
|
||
Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks `statvfs` in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as `{"mount", "stalled": true}`, and isn't probed again while its last probe is still blocked. Steward counts them as `mounts_stalled`.
|
||
|
||
### Every Docker read is bounded (1.15.0, #5764)
|
||
|
||
One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a `_steward` note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own `agent` footprint (`rss_bytes`, `cpu_pct`), which Steward records as `agent_rss_bytes` and `agent_cpu_pct`.
|
||
|
||
### What the host has waiting (1.16.0, #5765)
|
||
|
||
Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses `/usr/lib/update-notifier/apt-check` (pending;security), or `apt list --upgradable` when that's missing. On dnf hosts it uses `dnf -C check-update`, `dnf -C updateinfo list --security`, and `needs-restarting -r` for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as `updates`. Every sample also reads Debian's `/var/run/reboot-required` (and `.pkgs`) as `reboot`. Steward records `updates_pending`, `updates_security` and `reboot_required` (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. `state["updates_due"] = True` asks for a check on the next sample; a maintenance job uses it once it has updated the host.
|
||
|
||
|
||
### A root helper for updates and reboots (1.17.0, #5766)
|
||
|
||
The agent can't update or reboot its host: it runs as `steward-agent` under
|
||
`NoNewPrivileges` and `ProtectSystem=strict`, so no sudo is possible, and that
|
||
is how it should stay. A separate helper, `maint.py`, runs as root, started by
|
||
systemd when the agent leaves it a request:
|
||
|
||
- The agent writes `<id>.json` = `{"id", "action", ["node"]}` into
|
||
`/var/lib/steward-agent/requests`, the one path its unit lets it write
|
||
(`ReadWritePaths=-…`; the `-` keeps an old host without the directory
|
||
starting). It writes a dot-file first and renames it, so the helper never
|
||
sees half a request.
|
||
- `steward-maint.path` (`PathExistsGlob=…/requests/*.json`) starts
|
||
`steward-maint.service`, a oneshot running `maint.py` as root.
|
||
- The helper opens each request with `O_NOFOLLOW`, wants a regular file under
|
||
4 KiB, and accepts only `update`, `reboot`, `drain` or `undrain`, an id of
|
||
letters, digits and dashes, and a node id in the swarm's own 25-character
|
||
form. It deletes every request, valid or not, and writes the outcome to
|
||
`/var/lib/steward-maint/results/<id>.json` (root's; kept a week). A request
|
||
whose id already has a result is not run again.
|
||
- **update**: `apt-get update` then `dist-upgrade` keeping existing config
|
||
files, with `NEEDRESTART_MODE=l` so needrestart only lists services rather
|
||
than restarting the Docker daemon under its containers; or
|
||
`dnf -y upgrade --refresh`. The result says how many packages changed and
|
||
whether a reboot is now required.
|
||
- **reboot**: writes `rebooting`, syncs, `systemctl reboot`. Any request after
|
||
it waits for the next boot.
|
||
- **drain / undrain**: `docker node update --availability drain|active`, on
|
||
the given node or this host's own. Only a manager can; a worker's helper says
|
||
so in its result.
|
||
|
||
The agent reports each result's new state once, as a `maintenance` event with
|
||
its next post that gets through (results older than a day are not news). When
|
||
an update finishes it re-checks what's waiting straight away rather than at
|
||
the next hourly check. Its metadata says `maint_helper: true` once the path
|
||
unit is installed and it can write its request directory; the host's Agent
|
||
panel shows that under Manage agent. **Update agent** installs the helper on a
|
||
host that has an older agent.
|
||
|
||
So a stolen agent token can make its host update, reboot or drain, and nothing
|
||
else. Steward handing a host a job and following it through the reboot is
|
||
#5767 (below); the cadence that does it one host at a time is #5768.
|
||
|
||
### A maintenance job, followed through the reboot (1.18.0, #5767)
|
||
|
||
Steward keeps one `host_maintenance_jobs` row per job and moves it on only from
|
||
what the agents post, so a restart of Steward, or a reboot of the host Steward
|
||
runs on, loses nothing. `plugins/host_agent/maintenance.py` has the states and
|
||
transitions (model-free, unit-tested) and the database side.
|
||
|
||
```
|
||
queued → updating → [draining →] rebooting → [undraining →] done
|
||
↘ done (no reboot needed, or the job may never reboot)
|
||
any state → failed (a step's result said so) | timed_out (its deadline passed)
|
||
```
|
||
|
||
- **Hand-off.** Steward's answer to an ingest post carries
|
||
`"maintenance": [{"id", "action", ["node"]}]`, the step that host's agent
|
||
should carry out now. It is repeated on every post until a result for that
|
||
step id arrives; the agent leaves each id for the helper once
|
||
(`take_maintenance`), and the helper refuses an id it already has a result
|
||
for, so a repeat, or a re-hand after the agent restarts, never runs twice.
|
||
A step id is `<job id>-<action>`.
|
||
- **Results** are the `maintenance` events of #5766, matched to the job by step
|
||
id, so a drain reported by a manager reaches the job of the node it drained.
|
||
- **The reboot completes** when the host's agent sends `started` with a
|
||
`boot_at` after the reboot was handed out (60 s of clock slack). An agent
|
||
restart on a host that stayed up has an old `boot_at` and changes nothing.
|
||
- **Reboot only when required**: the update's result says whether a reboot is
|
||
needed; a job made with "never reboot" stops after the update.
|
||
- **Swarm nodes** are drained first and made active after. The node is found
|
||
by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
|
||
worker is drained by a manager whose agent has the helper. With no such
|
||
manager the job is refused rather than rebooting an undrained node.
|
||
- **Deadlines** (`DEADLINES`): queued 1 h, updating 90 min, draining 10 min,
|
||
rebooting 30 min, undraining 10 min. A minute tick (`host_agent_maintenance`)
|
||
times out a job past its deadline.
|
||
- **A maintenance window** scoped to the host's agent metrics
|
||
(`host_agent/<host>`) opens when the job starts, lasts as long as the rest of
|
||
the job could take, and ends when it does.
|
||
- Ingest applies all of this in a savepoint of its own: a job going wrong is
|
||
logged and never costs the host its samples.
|
||
|
||
On the host's Agent panel: a running job leads what the host has waiting, with
|
||
since-when; a finished one says what it did ("Done: 12 packages updated,
|
||
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
|
||
job, or the panel says why one can't start.
|
||
|
||
### A maintenance cadence: a group, one host at a time (#5768)
|
||
|
||
`plugins/host_agent/cadence.py`, driven by the same minute tick as the jobs.
|
||
A **plan** (`host_maintenance_plans`) names an inventory group, a weekday, a
|
||
time in a time zone (the creator's, by default), a window and a reboot policy:
|
||
`if_required` (the default), `always` or `never`. The page is **Hosts →
|
||
Maintenance** (`/plugins/host_agent/maintenance`).
|
||
|
||
- **When.** A plan's slot is the latest weekday-and-time at or before now, in
|
||
its zone. A slot fires once (`last_slot_at`) and only while its window is
|
||
open, so a Steward that was down over the slot starts late, not a day late.
|
||
A new or resumed plan starts from the next slot. **Run now** starts a run
|
||
without touching the schedule (the Ansible ruling on Run now).
|
||
- **A run** (`host_maintenance_runs`) fixes its hosts at the start, by name:
|
||
the group's hosts through their linked Ansible targets. A host with no agent,
|
||
or no maintenance helper, is skipped and listed.
|
||
- **One at a time.** Each host gets a #5767 job (`run_id` on the job). The next
|
||
host starts only when the last is **back in service**: its agent posting in
|
||
the last 3 min and, for a swarm node, the node `ready`/`active` and every
|
||
replicated service at its replicas, judged only from swarm reports made
|
||
after the job finished. A host not back within 30 min fails the run, and
|
||
its job is marked failed with the reason.
|
||
- **Stops.** A failed or timed-out job fails the run. No host is started after
|
||
the window; the run then stops and says how many weren't reached.
|
||
- **Alerting** is an ordinary alert rule on `host_agent/<host>/maintenance_failed`:
|
||
1 when the host's latest job (in 30 days) failed, timed out or didn't come
|
||
back into service, 0 once a later one succeeds.
|
||
- The page leads with a running run and its hosts, then failed runs from the
|
||
last week, then each plan with its schedule, next slot and last five runs.
|
||
Each host's Agent panel shows its last maintenance job.
|
||
|
||
### The swarm's health, alertable (1.19.0, #5788)
|
||
|
||
Every manager's agent reports the whole swarm. From each report the docker plugin (`plugins/docker/swarm_health.py`) records three swarm-wide figures on resource `swarm`, source `docker`:
|
||
|
||
- `swarm_services_short`: services running fewer tasks than they want. A service scaled to 0 is never short.
|
||
- `swarm_nodes_down`: nodes not `ready`.
|
||
- `swarm_managers_unreachable`: managers the swarm reports as not `reachable`. Agent 1.19.0 sends `ManagerStatus.Reachability`; before that, a manager whose status isn't `ready` counts.
|
||
|
||
Drained nodes are left out of the last two. A node is drained on purpose (VDNT-Util01 always, any node during its maintenance), so its going away is expected, and its host's own "Agent offline" rule still covers it.
|
||
|
||
The figures are recorded at most once every 30 s, whichever manager sent them, so three managers make one series and losing one doesn't silence it.
|
||
|
||
A maintenance job on a swarm node opens a second window over `docker` / `swarm` beside its host's (`maintenance._swarm_window`). An alert the drain or reboot causes then waits for the job, and one still firing when the window ends is sent then.
|
||
|
||
### Drive health (1.20.0, #5789)
|
||
|
||
Reading SMART needs root, so the root helper does it, as the one action on its fixed list that only reads: `drives`. Hourly, and at start, the agent leaves a `drives` request. The helper runs `smartctl --scan-open` to find the drives itself (the request names none), reads each with `smartctl --json -a -d <type> <dev>`, and writes a summary to `/var/lib/steward-maint/drives.json` (0644). It writes no job result, because a drive read is not a maintenance job. The agent's newest copy rides every sample, as updates do.
|
||
|
||
smartctl ≥ 7.0 answers in JSON, the same on every distribution and for SATA, SAS and NVMe drives. A missing or older smartctl is reported (`smartctl not installed`, `… too old`) and its text output is never parsed. The helper reports figures only:
|
||
|
||
- `passed` and `temp_c` for every drive;
|
||
- `critical_warning`, `life_used_pct`, `media_errors` and spare % against threshold for NVMe;
|
||
- reallocated / pending / offline-uncorrectable sectors for ATA, and life used where the drive reports it;
|
||
- grown defects for SCSI.
|
||
|
||
A drive with no SMART (a virtual disk, a bridge that hides it) reads `available: false`, and one smartctl couldn't open carries `error`.
|
||
|
||
Steward judges the figures in `drives.py`, so the judgment can change without an agent update:
|
||
|
||
- **failing**: the drive's self-assessment fails, an NVMe critical warning is raised, or spare blocks fall under threshold;
|
||
- **worn**: at least 90% of its rated life is used;
|
||
- **watch**: it has bad sectors or media errors but otherwise passes. Old disks carry a few for years, so a count alone isn't a failure.
|
||
|
||
The figures become metrics on `<host>:drive:<device>`: `drive_failing`, `drive_life_used_pct`, `drive_bad_sectors` and `drive_temp_c`. The host gets `drives_failing` and `drive_life_used_max_pct` for covering rules; they are recorded only when at least one drive was read, so a VM's virtual disks don't claim "0 failing". The registration keeps the newest report (`drive_health`), and the host page lists it under **Drive health**, what needs attention first.
|
||
|
||
**Installing smartctl on any common distribution.** `files/install-deps.sh` installs smartmontools with whichever package manager the host has: apt (without recommends, which would pull in a mail server), dnf, yum, zypper, pacman or apk. install.sh and the install / provision / update playbooks all fetch and run that one copy. A package that can't be installed is reported and never fails the install.
|
||
|
||
### What happened to the host (1.14.0, #5756)
|
||
|
||
A post may carry `events`, said once each, with the next post that gets through:
|
||
|
||
- `started` — the agent starting, with `boot_at` (from `/proc/uptime`).
|
||
- `stopping` — on SIGTERM, with `host_shutdown`: true, false, or null when systemd can't say.
|
||
- `lost_contact` — `from` the first post that got no answer (or a proxy's 5xx) `to` the post carrying it.
|
||
|
||
Steward turns them into the host's downtime (`host_downtime`): **reboot** (a stop, then a boot after it), **agent_stop** (the host stayed up), **unclean** (a boot after the agent was last heard that nobody announced: power loss or a crash) and **lost_contact**. An announced stop explains the silence for `reboot_grace_seconds` (Settings → Plugins → host_agent, 10 minutes): `seconds_since_seen` reads 0 until then, and the host's other rules go quiet with it, as when it is offline. Past the grace, agent offline fires. The events live in memory, so a span the agent saw is lost if it restarts mid-outage; the boot time still reads the restart as unclean.
|
||
|
||
### Which temperature sensors are thermometers (#5782, #5783)
|
||
|
||
A board's spare thermistor inputs (a Nuvoton's SYSTIN, AUXTIN*) are often wired to nothing, and the kernel reports them anyway, at whatever the open input reads. The host's temperature (`temp_c_max`) leaves a sensor out when:
|
||
|
||
- **its value is impossible** (at or below 0 °C, or at or above 125 °C; `counts_as_hottest`), or
|
||
- **it doesn't behave like a thermometer.** Once an hour, `sensors.refresh` judges each sensor against the host's CPU (Intel `Package id N`, AMD `Tctl`/`Tdie`/`TccdN`, else `PECI Agent N`), using the last 7 days of hourly averages. It needs at least 72 shared hours and a CPU that moved at least 10 °C. A sensor is ruled out when it **falls as the CPU warms** (Pearson r ≤ −0.5, the shape of an open input reading the chip's own reference) or **holds within 0.5 °C** while the CPU moved. A wired board or drive sensor that barely follows the CPU (r near 0) stays in.
|
||
|
||
The verdicts are kept in memory and rebuilt at startup, so ingest and the host page read them without a query. The host page lists the sensors left out under "Not counted", each with its reason. A host with no CPU reading, or too little history, falls back to the value rule alone.
|
||
|
||
`temp_c_max` can drive an alert rule (#5787), e.g. "Host running hot": all hosts, over 85 °C for 10 consecutive checks. So can each sensor's `temp_c`, and the per-interface and per-disk throughput. `tests/core/test_metric_catalog.py` holds every metric ingest writes to the alert catalog, except the few `NOT_ALERTABLE` names in `plugins/host_agent/__init__.py`.
|
||
|
||
### Logging
|
||
|
||
To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:
|
||
- `INFO` on start and on successful flush-after-buffered-samples.
|
||
- `WARN` on POST failure.
|
||
- `ERROR` on config errors, token rejection.
|
||
|
||
### Identity
|
||
|
||
The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.
|
||
|
||
### Failure behavior
|
||
|
||
- POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success.
|
||
- On next success → flush all buffered samples in a single batched POST.
|
||
- Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
|
||
- Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
|
||
- Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.
|
||
|
||
---
|
||
|
||
## Plugin design (server-side)
|
||
|
||
### File layout
|
||
|
||
Mirrors the existing `plugins/snmp/` structure.
|
||
|
||
```
|
||
plugins/host_agent/
|
||
├── __init__.py
|
||
├── plugin.yaml # metadata + default config
|
||
├── routes.py # blueprint: ingest, install.sh, UI pages, settings actions
|
||
├── models.py # HostAgentRegistration SQLAlchemy model
|
||
├── agent.py # the Python agent, served to targets at GET /agent.py
|
||
├── migrations/
|
||
│ ├── env.py
|
||
│ ├── __init__.py
|
||
│ └── versions/
|
||
│ └── host_agent_001_initial.py # creates host_agent_registrations table
|
||
├── templates/
|
||
│ ├── detail.html # per-host detail page
|
||
│ ├── widget_history.html # dashboard history chart widget
|
||
│ ├── settings_list.html # plugin settings: list of registered hosts
|
||
│ └── install.sh.j2 # install script template (Jinja)
|
||
└── scheduler.py # periodic task: mark stale agents as "offline"
|
||
```
|
||
|
||
### `HostAgentRegistration` model (plugin-private table)
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `id` | UUID str | primary key |
|
||
| `host_id` | str, FK → `hosts.id`, unique | one registration per Host |
|
||
| `token_hash` | str | `sha256(token)`; raw token shown only once at creation |
|
||
| `token_created_at` | datetime | for audit and rotation history |
|
||
| `agent_version` | str, nullable | reported on each ingest; updated on change |
|
||
| `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` |
|
||
| `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` |
|
||
| `arch` | str, nullable | e.g. `x86_64` |
|
||
| `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" |
|
||
| `created_at` / `updated_at` | datetime | standard |
|
||
|
||
### Routes (blueprint prefix `/plugins/host_agent`)
|
||
|
||
| Route | Auth | Purpose |
|
||
|---|---|---|
|
||
| `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. |
|
||
| `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. |
|
||
| `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. |
|
||
| `GET /<host_id>/` | session | Per-host detail page. |
|
||
| `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. |
|
||
| `POST /settings/<host_id>/rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. |
|
||
| `POST /settings/<host_id>/delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. |
|
||
| `GET /widget` | session | HTMX partial: fleet table widget body. |
|
||
| `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). |
|
||
|
||
### Token handling
|
||
|
||
- Tokens are generated with `secrets.token_urlsafe(32)` at host creation.
|
||
- Only `sha256(token)` is persisted.
|
||
- The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
|
||
- Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.
|
||
|
||
### Stale-agent scheduler
|
||
|
||
One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.
|
||
|
||
**Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.
|
||
|
||
**As built:** the task records `host_agent/<host>/seconds_since_seen` each minute, and "Agent offline" is an ordinary alert rule on it (`> 180`). It counts from the later of `last_seen_at` and the agent's newest post to *arrive* (kept in memory, `scheduler.arrivals`). `last_seen_at` is written in the ingest's own transaction, so a post Steward received but couldn't store (its storage stalled, #5514) would otherwise read as the agent's silence. Nothing counts from before Steward last started watching without a break (#5441).
|
||
|
||
### `METRIC_CATALOG` registration
|
||
|
||
Add to `steward/alerts/routes.py`:
|
||
|
||
```python
|
||
"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
|
||
"swap_used_bytes", "disk_used_pct_worst", "load_1m",
|
||
"load_5m", "load_15m", "uptime_secs"],
|
||
```
|
||
|
||
This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work.
|
||
|
||
---
|
||
|
||
## Wire format
|
||
|
||
### Request
|
||
|
||
```http
|
||
POST /plugins/host_agent/ingest HTTP/1.1
|
||
Host: steward.home.lan
|
||
Authorization: Bearer a1b2c3d4e5f6...
|
||
Content-Type: application/json
|
||
```
|
||
|
||
```json
|
||
{
|
||
"agent_version": "1.0.0",
|
||
"hostname": "myhost",
|
||
"metadata": {
|
||
"kernel": "6.8.0-45-generic",
|
||
"distro": "Ubuntu 24.04",
|
||
"arch": "x86_64"
|
||
},
|
||
"samples": [
|
||
{
|
||
"ts": "2026-04-14T02:55:00Z",
|
||
"cpu_pct": 12.4,
|
||
"mem": {
|
||
"total_bytes": 33554432000,
|
||
"used_bytes": 18253611008,
|
||
"available_bytes": 15300820992,
|
||
"swap_used_bytes": 0
|
||
},
|
||
"load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
|
||
"uptime_secs": 482934,
|
||
"storage": [
|
||
{ "mount": "/", "total_bytes": 499289948160, "used_bytes": 312456789012 },
|
||
{ "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
|
||
]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
### Design points
|
||
|
||
- **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code.
|
||
- **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes.
|
||
- **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
|
||
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
|
||
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
|
||
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
|
||
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.
|
||
|
||
### Server expansion into `PluginMetric` rows
|
||
|
||
Per sample, the server writes:
|
||
|
||
| `metric_name` | `resource_name` | derivation |
|
||
|---|---|---|
|
||
| `cpu_pct` | `<host.name>` | `sample.cpu_pct` as-is |
|
||
| `mem_used_pct` | `<host.name>` | `100 * (total - available) / total` |
|
||
| `mem_available_bytes` | `<host.name>` | `sample.mem.available_bytes` |
|
||
| `swap_used_bytes` | `<host.name>` | `sample.mem.swap_used_bytes` |
|
||
| `load_1m` | `<host.name>` | `sample.load["1m"]` |
|
||
| `load_5m` | `<host.name>` | `sample.load["5m"]` |
|
||
| `load_15m` | `<host.name>` | `sample.load["15m"]` |
|
||
| `uptime_secs` | `<host.name>` | `sample.uptime_secs` |
|
||
| `disk_used_pct` | `<host.name>:<mount>` | `100 * used / total` per mount |
|
||
| `disk_used_bytes` | `<host.name>:<mount>` | raw per mount |
|
||
| `disk_total_bytes` | `<host.name>:<mount>` | raw per mount |
|
||
| `disk_used_pct_worst` | `<host.name>` | `max(disk_used_pct across mounts)` — fleet table column |
|
||
|
||
All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path.
|
||
|
||
Plus one update to `HostAgentRegistration`:
|
||
|
||
```python
|
||
reg.last_seen_at = max(sample.ts for sample in samples)
|
||
reg.agent_version = payload.agent_version
|
||
reg.kernel = payload.metadata.kernel
|
||
reg.distro = payload.metadata.distro
|
||
reg.arch = payload.metadata.arch
|
||
```
|
||
|
||
(Only written if changed, to keep `updated_at` meaningful.)
|
||
|
||
### Response
|
||
|
||
```json
|
||
{ "ok": true, "samples_accepted": 3 }
|
||
```
|
||
|
||
Failures:
|
||
|
||
```json
|
||
{ "ok": false, "error": "invalid_token" } // 401
|
||
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" } // 400
|
||
```
|
||
|
||
Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).
|
||
|
||
---
|
||
|
||
## Install script
|
||
|
||
### The one-liner (what the UI shows)
|
||
|
||
```
|
||
curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh
|
||
```
|
||
|
||
The UI also offers a "review script before running" link that opens a modal showing the two-step form:
|
||
|
||
```
|
||
curl -sSL '…' -o install.sh
|
||
less install.sh
|
||
sudo sh install.sh
|
||
```
|
||
|
||
### Rendered script structure (Jinja template `install.sh.j2`)
|
||
|
||
`plugins/host_agent/templates/install.sh.j2` is the script; it is not copied
|
||
here, because a copy drifts. It checks it runs as root with systemd and
|
||
python3, handles `--uninstall`, creates the `steward-agent` user, fetches
|
||
`agent.py`, writes the config, then fetches the maintenance helper and the
|
||
three units from `/plugins/host_agent/files/` (the same files the playbooks
|
||
install), creates the maintenance directories, and enables
|
||
`steward-maint.path` and the agent.
|
||
|
||
### Design points
|
||
|
||
- **One copy of every file a host installs.** The units and the maintenance helper live in `plugins/host_agent/files/` and are served at `/plugins/host_agent/files/<name>` (a fixed list; anything else is 404). install.sh and the install, provision and update playbooks all fetch them, so the unit can't differ between a host installed by hand and one installed by Ansible (#5766).
|
||
- **Agent binary served by Steward itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
|
||
- **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
|
||
- **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.
|
||
- **Fail-fast on preflight.** Missing systemd or python3 → clear error, exit. No half-installed agent.
|
||
- **Query-string token for install fetch is acceptable.** It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the `Authorization` header.
|
||
- **No auto-upgrade.** Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.
|
||
|
||
---
|
||
|
||
## UI surfaces
|
||
|
||
### Dashboard widgets (`steward/core/widgets.py`)
|
||
|
||
- The fleet glance (CPU, mem, disk /, load 1m, stale mark) is the core **Hosts — Overview** widget (`hosts_overview`); the old `host_resources` table was folded into it (#5213).
|
||
- **`host_resource_history`** — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.
|
||
|
||
### Per-host detail page
|
||
|
||
`GET /plugins/host_agent/<host_id>/`:
|
||
|
||
- Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
|
||
- Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
|
||
- Full history charts, reusing `widget_history.html` with a wider layout.
|
||
- Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
|
||
- Agent status: last heartbeat timestamp, "live/stale" flag.
|
||
- "Copy install command" button (for re-install / rotation) that fetches the current one-liner.
|
||
|
||
### Plugin settings page
|
||
|
||
List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host.
|
||
|
||
---
|
||
|
||
## Error handling
|
||
|
||
| Failure | Who handles it | Response |
|
||
|---|---|---|
|
||
| Agent can't reach Steward | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. |
|
||
| Steward rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. |
|
||
| Steward rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. |
|
||
| Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). |
|
||
| Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. |
|
||
| `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. |
|
||
| Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. |
|
||
| Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. |
|
||
| Ingest body malformed | Server | 400 + descriptive `detail`. |
|
||
| Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. |
|
||
| Agent clock skew > 5 min | Server | Accept, log WARN. |
|
||
| Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. |
|
||
|
||
### Edge cases explicitly accepted, not handled
|
||
|
||
- Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this.
|
||
- Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care.
|
||
- Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow.
|
||
|
||
### Idempotency note
|
||
|
||
v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later.
|
||
|
||
---
|
||
|
||
## Testing strategy
|
||
|
||
- **Unit tests**
|
||
- Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content.
|
||
- Config file parser: valid, invalid, comments, whitespace edge cases.
|
||
- Ring buffer: fill, drain, overflow drops oldest.
|
||
- Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
|
||
- Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`.
|
||
- **Integration test**
|
||
- Spin up the plugin's ingest route in-process (Quart test client).
|
||
- Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content.
|
||
- Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated.
|
||
- **Install script verification**
|
||
- `sh -n install.sh` (syntax).
|
||
- `shellcheck install.sh`.
|
||
- Snapshot test of the rendered output against a committed fixture.
|
||
- No live systemd-in-container test — high cost, low value for a one-time install script.
|
||
- **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable.
|
||
|
||
---
|
||
|
||
## Scope boundaries — deferred to follow-ups
|
||
|
||
- **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
|
||
- **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
|
||
- **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
|
||
- **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
|
||
- **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
|
||
- **macOS / BSD / Windows agents** — separate code paths.
|
||
- **Self-upgrade mechanism** — re-run installer for now.
|
||
- **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change.
|
||
- **Network and per-process metrics** — additive; agent v2.
|
||
- **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation.
|
||
|
||
---
|
||
|
||
## Constraints worth being aware of
|
||
|
||
These are not blockers; they are feature boundaries inherent to the design.
|
||
|
||
1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
|
||
2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design.
|
||
3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.
|
||
|
||
---
|
||
|
||
## Implementation order (rough sketch — real plan comes from writing-plans)
|
||
|
||
1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`.
|
||
2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all.
|
||
3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth.
|
||
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
|
||
5. Install script template + `/install.sh` and `/agent.py` routes.
|
||
6. Settings page: list, add-host flow, rotate-token, delete.
|
||
7. Dashboard widgets: table + history chart. Register in `core/widgets.py`.
|
||
8. Per-host detail page.
|
||
9. `METRIC_CATALOG` entry in `steward/alerts/routes.py`.
|
||
10. Scheduler: stale-agent marker.
|
||
11. Tests at every stage; final integration test to tie it together.
|
||
|
||
The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.
|