diff --git a/docs/plugins/host-agent-design.md b/docs/plugins/host-agent-design.md new file mode 100644 index 0000000..49c1fed --- /dev/null +++ b/docs/plugins/host-agent-design.md @@ -0,0 +1,535 @@ +# Host Agent Plugin — Design Spec + +**Date:** 2026-04-14 +**Status:** Approved, ready for implementation planning +**Scope:** Roundtable plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent. + +--- + +## Goal + +Give Roundtable a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target. + +## Non-goals + +- Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install). +- Not a log collector. Time-series metrics only. +- Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it. +- Not a network monitor. Traffic counters are deferred to a follow-up. +- Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups). + +--- + +## Architecture + +``` +┌──────────────┐ POST /plugins/host_agent/ingest ┌────────────────────────┐ +│ Agent │ ──────────────────────────────────────────→ │ Roundtable │ +│ (Python) │ Authorization: Bearer │ host_agent plugin │ +│ on target │ JSON body: metrics snapshot │ │ +│ host │ │ routes.py (ingest + │ +│ │ ← 200 OK or 401/400 │ install.sh render + │ +│ │ │ UI) │ +│ systemd │ │ │ +│ unit, │ │ writes to: │ +│ dedicated │ │ - PluginMetric │ +│ user, │ │ (time-series bus) │ +│ 30s │ │ - HostAgentRegistr- │ +│ interval, │ │ ation (plugin- │ +│ in-memory │ │ private table) │ +│ ring buffer │ │ │ +└──────────────┘ │ registers widgets + │ + │ METRIC_CATALOG entry │ + └────────────────────────┘ +``` + +### Core principles + +- **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib. +- **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it. +- **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page. +- **Install** is a one-line `curl | sh` command rendered per-host by Roundtable with the token already baked in. +- **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Roundtable outages don't lose brief-window data. + +--- + +## Agent design + +### File layout on the target host + +``` +/usr/local/lib/roundtable-agent/agent.py # the script, target ~300 lines +/etc/roundtable-agent.conf # key=value config, 0640 root:roundtable-agent +/etc/systemd/system/roundtable-agent.service # unit file + # dedicated system user: roundtable-agent +``` + +### Config file format + +Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency): + +``` +url = https://roundtable.home.lan +token = a1b2c3d4... +interval_seconds = 30 +hostname = myhost # optional; defaults to uname -n +mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.) +``` + +### Agent internals (function list, not classes) + +- `read_config(path)` — parses the conf file into a dict. +- `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float). +- `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun). +- `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`. +- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`. +- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int). +- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached. +- `build_payload()` — assembles a snapshot from all collectors into one dict. +- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure. +- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full. +- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. + +**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping. + +### Dependencies + +Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries. + +### Signals and lifecycle + +- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly). +- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time → exit. +- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`. + +### Logging + +To stderr only (systemd captures to journal). No file logging, no log rotation. Levels: +- `INFO` on start and on successful flush-after-buffered-samples. +- `WARN` on POST failure. +- `ERROR` on config errors, token rejection. + +### Identity + +The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Roundtable looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide. + +### Failure behavior + +- POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success. +- On next success → flush all buffered samples in a single batched POST. +- Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages). +- Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew. +- Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed. + +--- + +## Plugin design (server-side) + +### File layout + +Mirrors the existing `plugins/snmp/` structure. + +``` +plugins/host_agent/ +├── __init__.py +├── plugin.yaml # metadata + default config +├── routes.py # blueprint: ingest, install.sh, UI pages, settings actions +├── models.py # HostAgentRegistration SQLAlchemy model +├── agent.py # the Python agent, served to targets at GET /agent.py +├── migrations/ +│ ├── env.py +│ ├── __init__.py +│ └── versions/ +│ └── host_agent_001_initial.py # creates host_agent_registrations table +├── templates/ +│ ├── detail.html # per-host detail page +│ ├── widget_table.html # dashboard fleet table widget +│ ├── widget_history.html # dashboard history chart widget +│ ├── settings_list.html # plugin settings: list of registered hosts +│ └── install.sh.j2 # install script template (Jinja) +└── scheduler.py # periodic task: mark stale agents as "offline" +``` + +### `HostAgentRegistration` model (plugin-private table) + +| Column | Type | Notes | +|---|---|---| +| `id` | UUID str | primary key | +| `host_id` | str, FK → `hosts.id`, unique | one registration per Host | +| `token_hash` | str | `sha256(token)`; raw token shown only once at creation | +| `token_created_at` | datetime | for audit and rotation history | +| `agent_version` | str, nullable | reported on each ingest; updated on change | +| `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` | +| `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` | +| `arch` | str, nullable | e.g. `x86_64` | +| `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" | +| `created_at` / `updated_at` | datetime | standard | + +### Routes (blueprint prefix `/plugins/host_agent`) + +| Route | Auth | Purpose | +|---|---|---| +| `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. | +| `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. | +| `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. | +| `GET //` | session | Per-host detail page. | +| `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. | +| `POST /settings//rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. | +| `POST /settings//delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. | +| `GET /widget` | session | HTMX partial: fleet table widget body. | +| `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). | + +### Token handling + +- Tokens are generated with `secrets.token_urlsafe(32)` at host creation. +- Only `sha256(token)` is persisted. +- The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation. +- Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat. + +### Stale-agent scheduler + +One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column. + +**Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update. + +### `METRIC_CATALOG` registration + +Add to `roundtable/alerts/routes.py`: + +```python +"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes", + "swap_used_bytes", "disk_used_pct_worst", "load_1m", + "load_5m", "load_15m", "uptime_secs"], +``` + +This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Roundtable core grows a plugin-registered catalog API — deferred as future core work. + +--- + +## Wire format + +### Request + +```http +POST /plugins/host_agent/ingest HTTP/1.1 +Host: roundtable.home.lan +Authorization: Bearer a1b2c3d4e5f6... +Content-Type: application/json +``` + +```json +{ + "agent_version": "1.0.0", + "hostname": "myhost", + "metadata": { + "kernel": "6.8.0-45-generic", + "distro": "Ubuntu 24.04", + "arch": "x86_64" + }, + "samples": [ + { + "ts": "2026-04-14T02:55:00Z", + "cpu_pct": 12.4, + "mem": { + "total_bytes": 33554432000, + "used_bytes": 18253611008, + "available_bytes": 15300820992, + "swap_used_bytes": 0 + }, + "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 }, + "uptime_secs": 482934, + "storage": [ + { "mount": "/", "total_bytes": 499289948160, "used_bytes": 312456789012 }, + { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 } + ] + } + ] +} +``` + +### Design points + +- **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code. +- **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes. +- **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew. +- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts. +- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent. +- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it. + +### Server expansion into `PluginMetric` rows + +Per sample, the server writes: + +| `metric_name` | `resource_name` | derivation | +|---|---|---| +| `cpu_pct` | `` | `sample.cpu_pct` as-is | +| `mem_used_pct` | `` | `100 * (total - available) / total` | +| `mem_available_bytes` | `` | `sample.mem.available_bytes` | +| `swap_used_bytes` | `` | `sample.mem.swap_used_bytes` | +| `load_1m` | `` | `sample.load["1m"]` | +| `load_5m` | `` | `sample.load["5m"]` | +| `load_15m` | `` | `sample.load["15m"]` | +| `uptime_secs` | `` | `sample.uptime_secs` | +| `disk_used_pct` | `:` | `100 * used / total` per mount | +| `disk_used_bytes` | `:` | raw per mount | +| `disk_total_bytes` | `:` | raw per mount | +| `disk_used_pct_worst` | `` | `max(disk_used_pct across mounts)` — fleet table column | + +All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path. + +Plus one update to `HostAgentRegistration`: + +```python +reg.last_seen_at = max(sample.ts for sample in samples) +reg.agent_version = payload.agent_version +reg.kernel = payload.metadata.kernel +reg.distro = payload.metadata.distro +reg.arch = payload.metadata.arch +``` + +(Only written if changed, to keep `updated_at` meaningful.) + +### Response + +```json +{ "ok": true, "samples_accepted": 3 } +``` + +Failures: + +```json +{ "ok": false, "error": "invalid_token" } // 401 +{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" } // 400 +``` + +Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart). + +--- + +## Install script + +### The one-liner (what the UI shows) + +``` +curl -sSL 'https://roundtable.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh +``` + +The UI also offers a "review script before running" link that opens a modal showing the two-step form: + +``` +curl -sSL '…' -o install.sh +less install.sh +sudo sh install.sh +``` + +### Rendered script structure (Jinja template `install.sh.j2`) + +```sh +#!/bin/sh +# Roundtable host agent installer +# Generated for: {{ host_name }} ({{ host_address }}) +# Roundtable URL: {{ url }} +set -e + +ROUNDTABLE_URL="{{ url }}" +AGENT_TOKEN="{{ token }}" +AGENT_VERSION="{{ agent_version }}" + +AGENT_USER="roundtable-agent" +AGENT_DIR="/usr/local/lib/roundtable-agent" +CONF_FILE="/etc/roundtable-agent.conf" +UNIT_FILE="/etc/systemd/system/roundtable-agent.service" + +# ── preflight ──────────────────────────────────────────────────────────────── +[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; } +command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; } +command -v python3 >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; } + +# Handle --uninstall +if [ "${1:-}" = "--uninstall" ]; then + systemctl disable --now roundtable-agent.service 2>/dev/null || true + rm -f "$UNIT_FILE" "$CONF_FILE" + rm -rf "$AGENT_DIR" + systemctl daemon-reload + # keep the system user — removing it risks orphaned files elsewhere + echo "uninstalled" + exit 0 +fi + +# ── create system user ─────────────────────────────────────────────────────── +if ! id "$AGENT_USER" >/dev/null 2>&1; then + useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER" +fi + +# ── drop the agent file ────────────────────────────────────────────────────── +mkdir -p "$AGENT_DIR" +curl -sSL "${ROUNDTABLE_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py" +chmod 0755 "$AGENT_DIR/agent.py" +chown root:root "$AGENT_DIR/agent.py" + +# ── write config ───────────────────────────────────────────────────────────── +cat > "$CONF_FILE" < "$UNIT_FILE" </`: + +- Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m). +- Per-mount storage breakdown (the metric that doesn't fit in a dashboard row). +- Full history charts, reusing `widget_history.html` with a wider layout. +- Metadata block: kernel, distro, arch, uptime (human-readable), agent version. +- Agent status: last heartbeat timestamp, "live/stale" flag. +- "Copy install command" button (for re-install / rotation) that fetches the current one-liner. + +### Plugin settings page + +List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host. + +--- + +## Error handling + +| Failure | Who handles it | Response | +|---|---|---| +| Agent can't reach Roundtable | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. | +| Roundtable rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. | +| Roundtable rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. | +| Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). | +| Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. | +| `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. | +| Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. | +| Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. | +| Ingest body malformed | Server | 400 + descriptive `detail`. | +| Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. | +| Agent clock skew > 5 min | Server | Accept, log WARN. | +| Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. | + +### Edge cases explicitly accepted, not handled + +- Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this. +- Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care. +- Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow. + +### Idempotency note + +v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later. + +--- + +## Testing strategy + +- **Unit tests** + - Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content. + - Config file parser: valid, invalid, comments, whitespace edge cases. + - Ring buffer: fill, drain, overflow drops oldest. + - Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU). + - Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`. +- **Integration test** + - Spin up the plugin's ingest route in-process (Quart test client). + - Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content. + - Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated. +- **Install script verification** + - `sh -n install.sh` (syntax). + - `shellcheck install.sh`. + - Snapshot test of the rendered output against a committed fixture. + - No live systemd-in-container test — high cost, low value for a one-time install script. +- **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable. + +--- + +## Scope boundaries — deferred to follow-ups + +- **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook. +- **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself. +- **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation. +- **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension. +- **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change. +- **macOS / BSD / Windows agents** — separate code paths. +- **Self-upgrade mechanism** — re-run installer for now. +- **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change. +- **Network and per-process metrics** — additive; agent v2. +- **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation. + +--- + +## Constraints worth being aware of + +These are not blockers; they are feature boundaries inherent to the design. + +1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable. +2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design. +3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses. + +--- + +## Implementation order (rough sketch — real plan comes from writing-plans) + +1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`. +2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all. +3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth. +4. Agent ring buffer + backoff + retry logic on top of the working collectors. +5. Install script template + `/install.sh` and `/agent.py` routes. +6. Settings page: list, add-host flow, rotate-token, delete. +7. Dashboard widgets: table + history chart. Register in `core/widgets.py`. +8. Per-host detail page. +9. `METRIC_CATALOG` entry in `roundtable/alerts/routes.py`. +10. Scheduler: stale-agent marker. +11. Tests at every stage; final integration test to tie it together. + +The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.