# Host Agent Plugin — Design Spec **Date:** 2026-04-14 **Status:** Approved, ready for implementation planning **Scope:** Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent. --- ## Goal Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target. ## Non-goals - Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install). - Not a log collector. Time-series metrics only. - Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it. - Not a network monitor. Traffic counters are deferred to a follow-up. - Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups). --- ## Architecture ``` ┌──────────────┐ POST /plugins/host_agent/ingest ┌────────────────────────┐ │ Agent │ ──────────────────────────────────────────→ │ Steward │ │ (Python) │ Authorization: Bearer │ host_agent plugin │ │ on target │ JSON body: metrics snapshot │ │ │ host │ │ routes.py (ingest + │ │ │ ← 200 OK or 401/400 │ install.sh render + │ │ │ │ UI) │ │ systemd │ │ │ │ unit, │ │ writes to: │ │ dedicated │ │ - PluginMetric │ │ user, │ │ (time-series bus) │ │ 30s │ │ - HostAgentRegistr- │ │ interval, │ │ ation (plugin- │ │ in-memory │ │ private table) │ │ ring buffer │ │ │ └──────────────┘ │ registers widgets + │ │ METRIC_CATALOG entry │ └────────────────────────┘ ``` ### Core principles - **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib. - **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it. - **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page. - **Install** is a one-line `curl | sh` command rendered per-host by Steward with the token already baked in. - **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data. --- ## Agent design ### File layout on the target host ``` /usr/local/lib/steward-agent/agent.py # the script, target ~300 lines /etc/steward-agent.conf # key=value config, 0640 root:steward-agent /etc/systemd/system/steward-agent.service # unit file # dedicated system user: steward-agent ``` ### Config file format Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency): ``` url = https://steward.home.lan token = a1b2c3d4... interval_seconds = 30 hostname = myhost # optional; defaults to uname -n mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.) ``` ### Agent internals (function list, not classes) - `read_config(path)` — parses the conf file into a dict. - `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float). - `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun). - `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`. - `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`. - `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int). - `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached. - `build_payload()` — assembles a snapshot from all collectors into one dict. - `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure. - `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full. - `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. **Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping. ### Dependencies Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries. ### Signals and lifecycle - **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly). - **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time → exit. - **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`. ### Logging To stderr only (systemd captures to journal). No file logging, no log rotation. Levels: - `INFO` on start and on successful flush-after-buffered-samples. - `WARN` on POST failure. - `ERROR` on config errors, token rejection. ### Identity The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide. ### Failure behavior - POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success. - On next success → flush all buffered samples in a single batched POST. - Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages). - Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew. - Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed. --- ## Plugin design (server-side) ### File layout Mirrors the existing `plugins/snmp/` structure. ``` plugins/host_agent/ ├── __init__.py ├── plugin.yaml # metadata + default config ├── routes.py # blueprint: ingest, install.sh, UI pages, settings actions ├── models.py # HostAgentRegistration SQLAlchemy model ├── agent.py # the Python agent, served to targets at GET /agent.py ├── migrations/ │ ├── env.py │ ├── __init__.py │ └── versions/ │ └── host_agent_001_initial.py # creates host_agent_registrations table ├── templates/ │ ├── detail.html # per-host detail page │ ├── widget_table.html # dashboard fleet table widget │ ├── widget_history.html # dashboard history chart widget │ ├── settings_list.html # plugin settings: list of registered hosts │ └── install.sh.j2 # install script template (Jinja) └── scheduler.py # periodic task: mark stale agents as "offline" ``` ### `HostAgentRegistration` model (plugin-private table) | Column | Type | Notes | |---|---|---| | `id` | UUID str | primary key | | `host_id` | str, FK → `hosts.id`, unique | one registration per Host | | `token_hash` | str | `sha256(token)`; raw token shown only once at creation | | `token_created_at` | datetime | for audit and rotation history | | `agent_version` | str, nullable | reported on each ingest; updated on change | | `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` | | `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` | | `arch` | str, nullable | e.g. `x86_64` | | `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" | | `created_at` / `updated_at` | datetime | standard | ### Routes (blueprint prefix `/plugins/host_agent`) | Route | Auth | Purpose | |---|---|---| | `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. | | `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. | | `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. | | `GET //` | session | Per-host detail page. | | `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. | | `POST /settings//rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. | | `POST /settings//delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. | | `GET /widget` | session | HTMX partial: fleet table widget body. | | `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). | ### Token handling - Tokens are generated with `secrets.token_urlsafe(32)` at host creation. - Only `sha256(token)` is persisted. - The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation. - Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat. ### Stale-agent scheduler One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column. **Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update. ### `METRIC_CATALOG` registration Add to `steward/alerts/routes.py`: ```python "host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes", "disk_used_pct_worst", "load_1m", "load_5m", "load_15m", "uptime_secs"], ``` This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work. --- ## Wire format ### Request ```http POST /plugins/host_agent/ingest HTTP/1.1 Host: steward.home.lan Authorization: Bearer a1b2c3d4e5f6... Content-Type: application/json ``` ```json { "agent_version": "1.0.0", "hostname": "myhost", "metadata": { "kernel": "6.8.0-45-generic", "distro": "Ubuntu 24.04", "arch": "x86_64" }, "samples": [ { "ts": "2026-04-14T02:55:00Z", "cpu_pct": 12.4, "mem": { "total_bytes": 33554432000, "used_bytes": 18253611008, "available_bytes": 15300820992, "swap_used_bytes": 0 }, "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 }, "uptime_secs": 482934, "storage": [ { "mount": "/", "total_bytes": 499289948160, "used_bytes": 312456789012 }, { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 } ] } ] } ``` ### Design points - **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code. - **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes. - **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew. - **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts. - **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent. - **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it. ### Server expansion into `PluginMetric` rows Per sample, the server writes: | `metric_name` | `resource_name` | derivation | |---|---|---| | `cpu_pct` | `` | `sample.cpu_pct` as-is | | `mem_used_pct` | `` | `100 * (total - available) / total` | | `mem_available_bytes` | `` | `sample.mem.available_bytes` | | `swap_used_bytes` | `` | `sample.mem.swap_used_bytes` | | `load_1m` | `` | `sample.load["1m"]` | | `load_5m` | `` | `sample.load["5m"]` | | `load_15m` | `` | `sample.load["15m"]` | | `uptime_secs` | `` | `sample.uptime_secs` | | `disk_used_pct` | `:` | `100 * used / total` per mount | | `disk_used_bytes` | `:` | raw per mount | | `disk_total_bytes` | `:` | raw per mount | | `disk_used_pct_worst` | `` | `max(disk_used_pct across mounts)` — fleet table column | All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path. Plus one update to `HostAgentRegistration`: ```python reg.last_seen_at = max(sample.ts for sample in samples) reg.agent_version = payload.agent_version reg.kernel = payload.metadata.kernel reg.distro = payload.metadata.distro reg.arch = payload.metadata.arch ``` (Only written if changed, to keep `updated_at` meaningful.) ### Response ```json { "ok": true, "samples_accepted": 3 } ``` Failures: ```json { "ok": false, "error": "invalid_token" } // 401 { "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" } // 400 ``` Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart). --- ## Install script ### The one-liner (what the UI shows) ``` curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh ``` The UI also offers a "review script before running" link that opens a modal showing the two-step form: ``` curl -sSL '…' -o install.sh less install.sh sudo sh install.sh ``` ### Rendered script structure (Jinja template `install.sh.j2`) ```sh #!/bin/sh # Steward host agent installer # Generated for: {{ host_name }} ({{ host_address }}) # Steward URL: {{ url }} set -e STEWARD_URL="{{ url }}" AGENT_TOKEN="{{ token }}" AGENT_VERSION="{{ agent_version }}" AGENT_USER="steward-agent" AGENT_DIR="/usr/local/lib/steward-agent" CONF_FILE="/etc/steward-agent.conf" UNIT_FILE="/etc/systemd/system/steward-agent.service" # ── preflight ──────────────────────────────────────────────────────────────── [ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; } command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; } command -v python3 >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; } # Handle --uninstall if [ "${1:-}" = "--uninstall" ]; then systemctl disable --now steward-agent.service 2>/dev/null || true rm -f "$UNIT_FILE" "$CONF_FILE" rm -rf "$AGENT_DIR" systemctl daemon-reload # keep the system user — removing it risks orphaned files elsewhere echo "uninstalled" exit 0 fi # ── create system user ─────────────────────────────────────────────────────── if ! id "$AGENT_USER" >/dev/null 2>&1; then useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER" fi # ── drop the agent file ────────────────────────────────────────────────────── mkdir -p "$AGENT_DIR" curl -sSL "${STEWARD_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py" chmod 0755 "$AGENT_DIR/agent.py" chown root:root "$AGENT_DIR/agent.py" # ── write config ───────────────────────────────────────────────────────────── cat > "$CONF_FILE" < "$UNIT_FILE" </`: - Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m). - Per-mount storage breakdown (the metric that doesn't fit in a dashboard row). - Full history charts, reusing `widget_history.html` with a wider layout. - Metadata block: kernel, distro, arch, uptime (human-readable), agent version. - Agent status: last heartbeat timestamp, "live/stale" flag. - "Copy install command" button (for re-install / rotation) that fetches the current one-liner. ### Plugin settings page List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host. --- ## Error handling | Failure | Who handles it | Response | |---|---|---| | Agent can't reach Steward | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. | | Steward rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. | | Steward rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. | | Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). | | Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. | | `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. | | Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. | | Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. | | Ingest body malformed | Server | 400 + descriptive `detail`. | | Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. | | Agent clock skew > 5 min | Server | Accept, log WARN. | | Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. | ### Edge cases explicitly accepted, not handled - Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this. - Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care. - Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow. ### Idempotency note v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later. --- ## Testing strategy - **Unit tests** - Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content. - Config file parser: valid, invalid, comments, whitespace edge cases. - Ring buffer: fill, drain, overflow drops oldest. - Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU). - Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`. - **Integration test** - Spin up the plugin's ingest route in-process (Quart test client). - Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content. - Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated. - **Install script verification** - `sh -n install.sh` (syntax). - `shellcheck install.sh`. - Snapshot test of the rendered output against a committed fixture. - No live systemd-in-container test — high cost, low value for a one-time install script. - **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable. --- ## Scope boundaries — deferred to follow-ups - **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook. - **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself. - **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation. - **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension. - **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change. - **macOS / BSD / Windows agents** — separate code paths. - **Self-upgrade mechanism** — re-run installer for now. - **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change. - **Network and per-process metrics** — additive; agent v2. - **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation. --- ## Constraints worth being aware of These are not blockers; they are feature boundaries inherent to the design. 1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable. 2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design. 3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses. --- ## Implementation order (rough sketch — real plan comes from writing-plans) 1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`. 2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all. 3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth. 4. Agent ring buffer + backoff + retry logic on top of the working collectors. 5. Install script template + `/install.sh` and `/agent.py` routes. 6. Settings page: list, add-host flow, rotate-token, delete. 7. Dashboard widgets: table + history chart. Register in `core/widgets.py`. 8. Per-host detail page. 9. `METRIC_CATALOG` entry in `steward/alerts/routes.py`. 10. Scheduler: stale-agent marker. 11. Tests at every stage; final integration test to tie it together. The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.