FabledSteward/docs/plugins/host-agent-design.md

Host Agent Plugin — Design Spec

Date: 2026-04-14 Status: Approved, ready for implementation planning Scope: Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.

---

Goal

Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.

Non-goals

• Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
• Not a log collector. Time-series metrics only.
• Not a container monitor — that is the existing docker plugin's job. The host agent observes the host itself, not containers running on it.
• Not a network monitor. Traffic counters are deferred to a follow-up.
• Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).

---

Architecture

┌──────────────┐         POST /plugins/host_agent/ingest        ┌────────────────────────┐
│  Agent       │  ──────────────────────────────────────────→  │ Steward             │
│  (Python)    │   Authorization: Bearer <per-host-token>       │  host_agent plugin     │
│  on target   │   JSON body: metrics snapshot                  │                        │
│  host        │                                                │  routes.py  (ingest +  │
│              │  ← 200 OK or 401/400                           │   install.sh render +  │
│              │                                                │   UI)                  │
│  systemd     │                                                │                        │
│  unit,       │                                                │  writes to:            │
│  dedicated   │                                                │   - PluginMetric       │
│  user,       │                                                │     (time-series bus)  │
│  30s         │                                                │   - HostAgentRegistr-  │
│  interval,   │                                                │     ation (plugin-     │
│  in-memory   │                                                │     private table)    │
│  ring buffer │                                                │                        │
└──────────────┘                                                │  registers widgets +   │
                                                                │  METRIC_CATALOG entry  │
                                                                └────────────────────────┘

Core principles

• Agent is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
• Plugin is self-contained under plugins/host_agent/. Writes to the core PluginMetric time-series bus (the designed cross-plugin integration point) and its own private host_agent_registrations table. Writes to the core Host model for identity, but adds no new columns to it.
• Auth is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
• Install is a one-line curl | sh command rendered per-host by Steward with the token already baked in.
• Failure is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data.

---

Agent design

File layout on the target host

/usr/local/lib/steward-agent/agent.py       # the script, target ~300 lines
/etc/steward-agent.conf                      # key=value config, 0640 root:steward-agent
/etc/systemd/system/steward-agent.service    # unit file
                                                # dedicated system user: steward-agent

Config file format

Flat key = value, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):

url = https://steward.home.lan
token = a1b2c3d4...
interval_seconds = 30
hostname = myhost              # optional; defaults to uname -n
mounts = /, /mnt/data          # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)

Agent internals (function list, not classes)

• read_config(path) — parses the conf file into a dict.
• collect_cpu() — reads /proc/stat twice around a short sleep, returns overall CPU % (0–100 float).
• collect_memory() — parses /proc/meminfo, returns {total, used, available, swap_used} in bytes. used = total - available (the available field, not free — Linux monitoring footgun).
• collect_storage(mounts) — shutil.disk_usage() per mount, returns [{mount, total, used}, ...].
• collect_load() — reads /proc/loadavg, returns [1m, 5m, 15m].
• collect_uptime() — reads /proc/uptime, returns seconds since boot (int).
• collect_metadata() — os.uname() for kernel + arch, /etc/os-release for distro. Called once at startup and cached.
• build_payload() — assembles a snapshot from all collectors into one dict.
• post_payload(url, token, payloads) — POSTs a list of samples, returns success/failure.
• RingBuffer(maxlen=20) — tiny FIFO wrapper, drops oldest when full.
• main_loop() — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer.

Target: ~300 lines total including docstrings. More than that is a smell that the agent is over-scoping.

Dependencies

Python 3.8+ stdlib only. urllib.request for the POST, plus json, os, shutil, time, socket, signal. No requests, no httpx, no TOML libraries.

Signals and lifecycle

• SIGHUP → re-read config file without restart (lets admins change interval/mounts cleanly).
• SIGTERM (from systemd) → finish current poll cycle → flush buffer one last time → exit.
• Systemd restart policy → Restart=on-failure, RestartSec=10.

Logging

To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:

• INFO on start and on successful flush-after-buffered-samples.
• WARN on POST failure.
• ERROR on config errors, token rejection.

Identity

The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the Host row via HostAgentRegistration.token_hash, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.

Failure behavior

• POST fails → push sample to ring buffer (max 20), exponential backoff 30s → 60s → 120s → 300s cap, resets on first success.
• On next success → flush all buffered samples in a single batched POST.
• Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
• Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
• Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.

---

Plugin design (server-side)

File layout

Mirrors the existing plugins/snmp/ structure.

plugins/host_agent/
├── __init__.py
├── plugin.yaml                          # metadata + default config
├── routes.py                            # blueprint: ingest, install.sh, UI pages, settings actions
├── models.py                            # HostAgentRegistration SQLAlchemy model
├── agent.py                             # the Python agent, served to targets at GET /agent.py
├── migrations/
│   ├── env.py
│   ├── __init__.py
│   └── versions/
│       └── host_agent_001_initial.py    # creates host_agent_registrations table
├── templates/
│   ├── detail.html                      # per-host detail page
│   ├── widget_table.html                # dashboard fleet table widget
│   ├── widget_history.html              # dashboard history chart widget
│   ├── settings_list.html               # plugin settings: list of registered hosts
│   └── install.sh.j2                    # install script template (Jinja)
└── scheduler.py                         # periodic task: mark stale agents as "offline"

HostAgentRegistration model (plugin-private table)

Column	Type	Notes
id	UUID str	primary key
host_id	str, FK → hosts.id, unique	one registration per Host
token_hash	str	sha256(token); raw token shown only once at creation
token_created_at	datetime	for audit and rotation history
agent_version	str, nullable	reported on each ingest; updated on change
kernel	str, nullable	from os.uname() — e.g. 6.8.0-45-generic
distro	str, nullable	parsed from /etc/os-release — e.g. Ubuntu 24.04
arch	str, nullable	e.g. x86_64
last_seen_at	datetime, nullable	updated on every successful ingest; source of "is this agent alive"
created_at / updated_at	datetime	standard

Routes (blueprint prefix /plugins/host_agent)

Route	Auth	Purpose
POST /ingest	bearer token	Agent push. Validates sha256(Authorization header) against token_hash → writes PluginMetric rows + updates HostAgentRegistration. Returns 200, 400, or 401.
GET /install.sh	query-param token	Renders the Jinja install.sh.j2 template with URL, token, and agent version substituted. Returns text/plain.
GET /agent.py	none	Serves the bundled agent source from plugins/host_agent/agent.py as text/x-python. Fetched by the install script.
GET /<host_id>/	session	Per-host detail page.
POST /settings/add-host	admin	Creates (or reuses) a Host row + creates HostAgentRegistration + generates token. Returns the install one-liner for UI display.
POST /settings/<host_id>/rotate-token	admin	Generates a new token, invalidates the old one, returns new install one-liner.
POST /settings/<host_id>/delete	admin	Removes HostAgentRegistration. Does not delete the backing Host row unless no other monitors reference it.
GET /widget	session	HTMX partial: fleet table widget body.
GET /widget/history	session	HTMX partial: history chart widget body (takes host_id query param).

Token handling

• Tokens are generated with secrets.token_urlsafe(32) at host creation.
• Only sha256(token) is persisted.
• The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
• Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.

Stale-agent scheduler

One ScheduledTask running every 60 seconds that flags HostAgentRegistration rows with last_seen_at > (3 × interval_seconds) ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.

Note: the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.

METRIC_CATALOG registration

Add to steward/alerts/routes.py:

"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
               "swap_used_bytes", "disk_used_pct_worst", "load_1m",
               "load_5m", "load_15m", "uptime_secs"],

This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work.

---

Wire format

Request

POST /plugins/host_agent/ingest HTTP/1.1
Host: steward.home.lan
Authorization: Bearer a1b2c3d4e5f6...
Content-Type: application/json

{
  "agent_version": "1.0.0",
  "hostname": "myhost",
  "metadata": {
    "kernel": "6.8.0-45-generic",
    "distro": "Ubuntu 24.04",
    "arch": "x86_64"
  },
  "samples": [
    {
      "ts": "2026-04-14T02:55:00Z",
      "cpu_pct": 12.4,
      "mem": {
        "total_bytes": 33554432000,
        "used_bytes":  18253611008,
        "available_bytes": 15300820992,
        "swap_used_bytes": 0
      },
      "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
      "uptime_secs": 482934,
      "storage": [
        { "mount": "/",         "total_bytes": 499289948160, "used_bytes": 312456789012 },
        { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
      ]
    }
  ]
}

Design points

• samples is always a list, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs [current, ...buffered] in a single request. One-path server code.
• ts is agent-reported, ISO-8601 UTC. Server trusts it for PluginMetric.recorded_at. Host clocks drift but NTP is reliable enough for our purposes.
• If |ts - now| > 5 minutes, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
• metadata is sent on every POST, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
• Raw bytes, not percentages, for memory and storage. Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
• CPU is the one exception — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.

Server expansion into PluginMetric rows

Per sample, the server writes:

metric_name	resource_name	derivation
cpu_pct	<host.name>	sample.cpu_pct as-is
mem_used_pct	<host.name>	100 * (total - available) / total
mem_available_bytes	<host.name>	sample.mem.available_bytes
swap_used_bytes	<host.name>	sample.mem.swap_used_bytes
load_1m	<host.name>	sample.load["1m"]
load_5m	<host.name>	sample.load["5m"]
load_15m	<host.name>	sample.load["15m"]
uptime_secs	<host.name>	sample.uptime_secs
disk_used_pct	<host.name>:<mount>	100 * used / total per mount
disk_used_bytes	<host.name>:<mount>	raw per mount
disk_total_bytes	<host.name>:<mount>	raw per mount
disk_used_pct_worst	<host.name>	max(disk_used_pct across mounts) — fleet table column

All values are float. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within PluginMetric and the existing cleanup path.

Plus one update to HostAgentRegistration:

reg.last_seen_at = max(sample.ts for sample in samples)
reg.agent_version = payload.agent_version
reg.kernel = payload.metadata.kernel
reg.distro = payload.metadata.distro
reg.arch   = payload.metadata.arch

(Only written if changed, to keep updated_at meaningful.)

Response

{ "ok": true, "samples_accepted": 3 }

Failures:

{ "ok": false, "error": "invalid_token" }                              // 401
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" }  // 400

Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).

---

Install script

The one-liner (what the UI shows)

curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh

The UI also offers a "review script before running" link that opens a modal showing the two-step form:

curl -sSL '…' -o install.sh
less install.sh
sudo sh install.sh

Rendered script structure (Jinja template install.sh.j2)

#!/bin/sh
# Steward host agent installer
# Generated for: {{ host_name }}  ({{ host_address }})
# Steward URL: {{ url }}
set -e

STEWARD_URL="{{ url }}"
AGENT_TOKEN="{{ token }}"
AGENT_VERSION="{{ agent_version }}"

AGENT_USER="steward-agent"
AGENT_DIR="/usr/local/lib/steward-agent"
CONF_FILE="/etc/steward-agent.conf"
UNIT_FILE="/etc/systemd/system/steward-agent.service"

# ── preflight ────────────────────────────────────────────────────────────────
[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; }
command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; }
command -v python3   >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; }

# Handle --uninstall
if [ "${1:-}" = "--uninstall" ]; then
    systemctl disable --now steward-agent.service 2>/dev/null || true
    rm -f "$UNIT_FILE" "$CONF_FILE"
    rm -rf "$AGENT_DIR"
    systemctl daemon-reload
    # keep the system user — removing it risks orphaned files elsewhere
    echo "uninstalled"
    exit 0
fi

# ── create system user ───────────────────────────────────────────────────────
if ! id "$AGENT_USER" >/dev/null 2>&1; then
    useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER"
fi

# ── drop the agent file ──────────────────────────────────────────────────────
mkdir -p "$AGENT_DIR"
curl -sSL "${STEWARD_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py"
chmod 0755 "$AGENT_DIR/agent.py"
chown root:root "$AGENT_DIR/agent.py"

# ── write config ─────────────────────────────────────────────────────────────
cat > "$CONF_FILE" <<EOF
url = $STEWARD_URL
token = $AGENT_TOKEN
interval_seconds = 30
EOF
chown "root:$AGENT_USER" "$CONF_FILE"
chmod 0640 "$CONF_FILE"

# ── write systemd unit ───────────────────────────────────────────────────────
cat > "$UNIT_FILE" <<EOF
[Unit]
Description=Steward host agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=$AGENT_USER
ExecStart=/usr/bin/python3 $AGENT_DIR/agent.py
Restart=on-failure
RestartSec=10
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/proc /sys

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now steward-agent.service

echo
echo "Steward host agent $AGENT_VERSION installed and running."
echo "Check status:  systemctl status steward-agent"
echo "Logs:          journalctl -u steward-agent -f"

Design points

• Agent binary served by Steward itself. GET /plugins/host_agent/agent.py serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
• Systemd hardening is cheap and correct. NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, ReadOnlyPaths=/proc /sys. The agent only needs read access to /proc, /sys, and mount points; denying everything else narrows blast radius.
• Uninstall is a first-class flag, not a separate script. Same one-liner with --uninstall appended.
• Fail-fast on preflight. Missing systemd or python3 → clear error, exit. No half-installed agent.
• Query-string token for install fetch is acceptable. It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the Authorization header.
• No auto-upgrade. Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.

---

UI surfaces

Dashboard widgets (steward/core/widgets.py)

• host_resources — table widget. One row per monitored host: name, CPU %, mem %, disk % (worst mount), load 1m, "last seen Xs ago" with red/yellow/green coloring. Fleet glance.
• host_resource_history — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.

Per-host detail page

GET /plugins/host_agent/<host_id>/:

• Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
• Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
• Full history charts, reusing widget_history.html with a wider layout.
• Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
• Agent status: last heartbeat timestamp, "live/stale" flag.
• "Copy install command" button (for re-install / rotation) that fetches the current one-liner.

Plugin settings page

List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses Host row → creates HostAgentRegistration → generates token → displays the curl install one-liner), and a "rotate token" action per host.

---

Error handling

Failure	Who handles it	Response
Agent can't reach Steward	Agent	Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success.
Steward rejects with 401	Agent	Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed.
Steward rejects with 400	Agent	Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew.
Ring buffer full	Agent	Drop oldest, keep newest. DEBUG log (expected during outages).
Config file missing or malformed	Agent	Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal.
/proc/stat read fails between samples	Agent	Skip CPU for this cycle, still POST the rest. Partial samples allowed.
Mount disappears between reads	Agent	Skip that mount for this sample. If permanent, its rows simply stop appearing.
Ingest token not found	Server	401 + {"error": "invalid_token"}.
Ingest body malformed	Server	400 + descriptive detail.
Ingest writes fail mid-batch	Server	500 + rollback. Agent retries whole batch on next cycle.
Agent clock skew > 5 min	Server	Accept, log WARN.
Host deleted from UI	Server	HostAgentRegistration removed; agent's next POST → 401 → admin re-adds or uninstalls.

Edge cases explicitly accepted, not handled

• Two agents sharing a token → last-write-wins on last_seen_at, metrics get stored under the same resource_name. Admin should never do this.
• Agent clock running backward (NTP correction) → recorded_at values non-monotonic. Charts render fine; the DB doesn't care.
• Hostname change on target → HostAgentRegistration.host_id is still the token's identity; resource_name tracks Host.name. Admin edits Host.name if they want the display to follow.

Idempotency note

v1 doesn't enforce unique constraints on PluginMetric rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on (source_module, resource_name, metric_name, recorded_at) later.

---

Testing strategy

• Unit tests
  • Agent collectors against fixture /proc/stat, /proc/meminfo, /proc/loadavg, /proc/uptime content.
  • Config file parser: valid, invalid, comments, whitespace edge cases.
  • Ring buffer: fill, drain, overflow drops oldest.
  • Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
  • Plugin migration smoke test: alembic upgrade head in a temp DB, then downgrade base.
• Integration test
  • Spin up the plugin's ingest route in-process (Quart test client).
  • Feed a real payload generated by the agent's build_payload() using mocked /proc content.
  • Assert PluginMetric rows land correctly and HostAgentRegistration is updated.
• Install script verification
  • sh -n install.sh (syntax).
  • shellcheck install.sh.
  • Snapshot test of the rendered output against a committed fixture.
  • No live systemd-in-container test — high cost, low value for a one-time install script.
• Agent is importable as a Python module. main_loop() is guarded by if __name__ == "__main__":, so everything else is unit-testable.

---

Scope boundaries — deferred to follow-ups

• Ansible-action alert type (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
• Plugin synergy detection (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
• Fleet-provision mode — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
• Event signals — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
• Non-systemd init systems — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
• macOS / BSD / Windows agents — separate code paths.
• Self-upgrade mechanism — re-run installer for now.
• Disk buffering when RAM ring fills — drop-in upgrade, no protocol change.
• Network and per-process metrics — additive; agent v2.
• metadata_json column on core Host — rejected in favor of strict plugin isolation.

---

Constraints worth being aware of

These are not blockers; they are feature boundaries inherent to the design.

1. Stdlib-only Python is a ceiling. Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, psutil-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
2. Interval push is not event push. OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a sibling endpoint, not an extension of this design.
3. PluginMetric only stores floats. String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.

---

Implementation order (rough sketch — real plan comes from writing-plans)

1. Plugin scaffolding: plugin.yaml, __init__.py, empty routes blueprint, migration for host_agent_registrations.
2. Agent v1 as a standalone script: all collectors + config parser + build_payload() working locally. Testable without any server at all.
3. Server /ingest route: accepts payload, writes PluginMetric, updates HostAgentRegistration. Token auth.
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
5. Install script template + /install.sh and /agent.py routes.
6. Settings page: list, add-host flow, rotate-token, delete.
7. Dashboard widgets: table + history chart. Register in core/widgets.py.
8. Per-host detail page.
9. METRIC_CATALOG entry in steward/alerts/routes.py.
10. Scheduler: stale-agent marker.
11. Tests at every stage; final integration test to tie it together.

The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.