FabledSteward/docs/plugins/host-agent-design.md

# Host Agent Plugin — Design Spec

**Date:** 2026-04-14
**Status:** Approved, ready for implementation planning
**Scope:** Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.

---

## Goal

Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.

## Non-goals

- Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
- Not a log collector. Time-series metrics only.
- Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it.
- Not a network monitor. Traffic counters are deferred to a follow-up.
- Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).

---

## Architecture

```
┌──────────────┐         POST /plugins/host_agent/ingest        ┌────────────────────────┐
│  Agent       │  ──────────────────────────────────────────→  │ Steward             │
│  (Python)    │   Authorization: Bearer <per-host-token>       │  host_agent plugin     │
│  on target   │   JSON body: metrics snapshot                  │                        │
│  host        │                                                │  routes.py  (ingest +  │
│              │  ← 200 OK or 401/400                           │   install.sh render +  │
│              │                                                │   UI)                  │
│  systemd     │                                                │                        │
│  unit,       │                                                │  writes to:            │
│  dedicated   │                                                │   - PluginMetric       │
│  user,       │                                                │     (time-series bus)  │
│  30s         │                                                │   - HostAgentRegistr-  │
│  interval,   │                                                │     ation (plugin-     │
│  in-memory   │                                                │     private table)    │
│  ring buffer │                                                │                        │
└──────────────┘                                                │  registers widgets +   │
                                                                │  METRIC_CATALOG entry  │
                                                                └────────────────────────┘
```

### Core principles

- **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
- **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it.
- **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
- **Install** is a one-line `curl | sh` command rendered per-host by Steward with the token already baked in.
- **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data.

---

## Agent design

### File layout on the target host

```
/usr/local/lib/steward-agent/agent.py       # the script, target ~300 lines
/etc/steward-agent.conf                      # key=value config, 0640 root:steward-agent
/etc/systemd/system/steward-agent.service    # unit file
                                                # dedicated system user: steward-agent
```

### Config file format

Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):

```
url = https://steward.home.lan
token = a1b2c3d4...
interval_seconds = 30
hostname = myhost              # optional; defaults to uname -n
mounts = /, /mnt/data          # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
docker_logs_enabled = true     # optional; on by default — per-host container-log kill-switch
docker_log_exclude = watchtower, noisy-svc   # optional; container names whose logs this host skips collecting
```

Container-log collection is on by default (no config needed). The two keys above
are the per-host opt-outs; the operator-facing global toggle + exclude live in
Settings and are enforced server-side (the push model has no channel to command
an agent), so they don't require touching a host's conf.

### Agent internals (function list, not classes)

- `read_config(path)` — parses the conf file into a dict.
- `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float).
- `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun).
- `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`.
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
  container, fetches new log lines over the Docker socket with an incremental
  `since` cursor kept per container in `state` (a container's first interval
  seeds from a short tail). Demuxes the multiplexed stream, parses each line's
  RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
  a truncation; the deferred lines come next interval). Folded into the sample as
  `docker_logs`; omitted when empty.
- `build_payload()` — assembles a snapshot from all collectors into one dict.
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).

**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.

### Dependencies

Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries.

### Signals and lifecycle

- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly).
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time, with a `stopping` event saying whether the host is going down too (`systemctl is-system-running` says `stopping`) → exit.
- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`.

### Mounts that don't answer (1.14.0, #5760)

Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks `statvfs` in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as `{"mount", "stalled": true}`, and isn't probed again while its last probe is still blocked. Steward counts them as `mounts_stalled`.

### Every Docker read is bounded (1.15.0, #5764)

One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a `_steward` note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own `agent` footprint (`rss_bytes`, `cpu_pct`), which Steward records as `agent_rss_bytes` and `agent_cpu_pct`.

### What the host has waiting (1.16.0, #5765)

Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses `/usr/lib/update-notifier/apt-check` (pending;security), or `apt list --upgradable` when that's missing. On dnf hosts it uses `dnf -C check-update`, `dnf -C updateinfo list --security`, and `needs-restarting -r` for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as `updates`. Every sample also reads Debian's `/var/run/reboot-required` (and `.pkgs`) as `reboot`. Steward records `updates_pending`, `updates_security` and `reboot_required` (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. `state["updates_due"] = True` asks for a check on the next sample; a maintenance job uses it once it has updated the host.

### What happened to the host (1.14.0, #5756)

A post may carry `events`, said once each, with the next post that gets through:

- `started` — the agent starting, with `boot_at` (from `/proc/uptime`).
- `stopping` — on SIGTERM, with `host_shutdown`: true, false, or null when systemd can't say.
- `lost_contact` — `from` the first post that got no answer (or a proxy's 5xx) `to` the post carrying it.

Steward turns them into the host's downtime (`host_downtime`): **reboot** (a stop, then a boot after it), **agent_stop** (the host stayed up), **unclean** (a boot after the agent was last heard that nobody announced: power loss or a crash) and **lost_contact**. An announced stop explains the silence for `reboot_grace_seconds` (Settings → Plugins → host_agent, 10 minutes): `seconds_since_seen` reads 0 until then, and the host's other rules go quiet with it, as when it is offline. Past the grace, agent offline fires. The events live in memory, so a span the agent saw is lost if it restarts mid-outage; the boot time still reads the restart as unclean.

### Logging

To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:
- `INFO` on start and on successful flush-after-buffered-samples.
- `WARN` on POST failure.
- `ERROR` on config errors, token rejection.

### Identity

The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.

### Failure behavior

- POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success.
- On next success → flush all buffered samples in a single batched POST.
- Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
- Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
- Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.

---

## Plugin design (server-side)

### File layout

Mirrors the existing `plugins/snmp/` structure.

```
plugins/host_agent/
├── __init__.py
├── plugin.yaml                          # metadata + default config
├── routes.py                            # blueprint: ingest, install.sh, UI pages, settings actions
├── models.py                            # HostAgentRegistration SQLAlchemy model
├── agent.py                             # the Python agent, served to targets at GET /agent.py
├── migrations/
│   ├── env.py
│   ├── __init__.py
│   └── versions/
│       └── host_agent_001_initial.py    # creates host_agent_registrations table
├── templates/
│   ├── detail.html                      # per-host detail page
│   ├── widget_history.html              # dashboard history chart widget
│   ├── settings_list.html               # plugin settings: list of registered hosts
│   └── install.sh.j2                    # install script template (Jinja)
└── scheduler.py                         # periodic task: mark stale agents as "offline"
```

### `HostAgentRegistration` model (plugin-private table)

| Column | Type | Notes |
|---|---|---|
| `id` | UUID str | primary key |
| `host_id` | str, FK → `hosts.id`, unique | one registration per Host |
| `token_hash` | str | `sha256(token)`; raw token shown only once at creation |
| `token_created_at` | datetime | for audit and rotation history |
| `agent_version` | str, nullable | reported on each ingest; updated on change |
| `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` |
| `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` |
| `arch` | str, nullable | e.g. `x86_64` |
| `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" |
| `created_at` / `updated_at` | datetime | standard |

### Routes (blueprint prefix `/plugins/host_agent`)

| Route | Auth | Purpose |
|---|---|---|
| `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. |
| `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. |
| `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. |
| `GET /<host_id>/` | session | Per-host detail page. |
| `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. |
| `POST /settings/<host_id>/rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. |
| `POST /settings/<host_id>/delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. |
| `GET /widget` | session | HTMX partial: fleet table widget body. |
| `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). |

### Token handling

- Tokens are generated with `secrets.token_urlsafe(32)` at host creation.
- Only `sha256(token)` is persisted.
- The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
- Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.

### Stale-agent scheduler

One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.

**Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.

**As built:** the task records `host_agent/<host>/seconds_since_seen` each minute, and "Agent offline" is an ordinary alert rule on it (`> 180`). It counts from the later of `last_seen_at` and the agent's newest post to *arrive* (kept in memory, `scheduler.arrivals`). `last_seen_at` is written in the ingest's own transaction, so a post Steward received but couldn't store (its storage stalled, #5514) would otherwise read as the agent's silence. Nothing counts from before Steward last started watching without a break (#5441).

### `METRIC_CATALOG` registration

Add to `steward/alerts/routes.py`:

```python
"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
               "swap_used_bytes", "disk_used_pct_worst", "load_1m",
               "load_5m", "load_15m", "uptime_secs"],
```

This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work.

---

## Wire format

### Request

```http
POST /plugins/host_agent/ingest HTTP/1.1
Host: steward.home.lan
Authorization: Bearer a1b2c3d4e5f6...
Content-Type: application/json
```

```json
{
  "agent_version": "1.0.0",
  "hostname": "myhost",
  "metadata": {
    "kernel": "6.8.0-45-generic",
    "distro": "Ubuntu 24.04",
    "arch": "x86_64"
  },
  "samples": [
    {
      "ts": "2026-04-14T02:55:00Z",
      "cpu_pct": 12.4,
      "mem": {
        "total_bytes": 33554432000,
        "used_bytes":  18253611008,
        "available_bytes": 15300820992,
        "swap_used_bytes": 0
      },
      "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
      "uptime_secs": 482934,
      "storage": [
        { "mount": "/",         "total_bytes": 499289948160, "used_bytes": 312456789012 },
        { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
      ]
    }
  ]
}
```

### Design points

- **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code.
- **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes.
- **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.

### Server expansion into `PluginMetric` rows

Per sample, the server writes:

| `metric_name` | `resource_name` | derivation |
|---|---|---|
| `cpu_pct` | `<host.name>` | `sample.cpu_pct` as-is |
| `mem_used_pct` | `<host.name>` | `100 * (total - available) / total` |
| `mem_available_bytes` | `<host.name>` | `sample.mem.available_bytes` |
| `swap_used_bytes` | `<host.name>` | `sample.mem.swap_used_bytes` |
| `load_1m` | `<host.name>` | `sample.load["1m"]` |
| `load_5m` | `<host.name>` | `sample.load["5m"]` |
| `load_15m` | `<host.name>` | `sample.load["15m"]` |
| `uptime_secs` | `<host.name>` | `sample.uptime_secs` |
| `disk_used_pct` | `<host.name>:<mount>` | `100 * used / total` per mount |
| `disk_used_bytes` | `<host.name>:<mount>` | raw per mount |
| `disk_total_bytes` | `<host.name>:<mount>` | raw per mount |
| `disk_used_pct_worst` | `<host.name>` | `max(disk_used_pct across mounts)` — fleet table column |

All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path.

Plus one update to `HostAgentRegistration`:

```python
reg.last_seen_at = max(sample.ts for sample in samples)
reg.agent_version = payload.agent_version
reg.kernel = payload.metadata.kernel
reg.distro = payload.metadata.distro
reg.arch   = payload.metadata.arch
```

(Only written if changed, to keep `updated_at` meaningful.)

### Response

```json
{ "ok": true, "samples_accepted": 3 }
```

Failures:

```json
{ "ok": false, "error": "invalid_token" }                              // 401
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" }  // 400
```

Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).

---

## Install script

### The one-liner (what the UI shows)

```
curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh
```

The UI also offers a "review script before running" link that opens a modal showing the two-step form:

```
curl -sSL '…' -o install.sh
less install.sh
sudo sh install.sh
```

### Rendered script structure (Jinja template `install.sh.j2`)

```sh
#!/bin/sh
# Steward host agent installer
# Generated for: {{ host_name }}  ({{ host_address }})
# Steward URL: {{ url }}
set -e

STEWARD_URL="{{ url }}"
AGENT_TOKEN="{{ token }}"
AGENT_VERSION="{{ agent_version }}"

AGENT_USER="steward-agent"
AGENT_DIR="/usr/local/lib/steward-agent"
CONF_FILE="/etc/steward-agent.conf"
UNIT_FILE="/etc/systemd/system/steward-agent.service"

# ── preflight ────────────────────────────────────────────────────────────────
[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; }
command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; }
command -v python3   >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; }

# Handle --uninstall
if [ "${1:-}" = "--uninstall" ]; then
    systemctl disable --now steward-agent.service 2>/dev/null || true
    rm -f "$UNIT_FILE" "$CONF_FILE"
    rm -rf "$AGENT_DIR"
    systemctl daemon-reload
    # keep the system user — removing it risks orphaned files elsewhere
    echo "uninstalled"
    exit 0
fi

# ── create system user ───────────────────────────────────────────────────────
if ! id "$AGENT_USER" >/dev/null 2>&1; then
    useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER"
fi

# ── drop the agent file ──────────────────────────────────────────────────────
mkdir -p "$AGENT_DIR"
curl -sSL "${STEWARD_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py"
chmod 0755 "$AGENT_DIR/agent.py"
chown root:root "$AGENT_DIR/agent.py"

# ── write config ─────────────────────────────────────────────────────────────
cat > "$CONF_FILE" <<EOF
url = $STEWARD_URL
token = $AGENT_TOKEN
interval_seconds = 30
EOF
chown "root:$AGENT_USER" "$CONF_FILE"
chmod 0640 "$CONF_FILE"

# ── write systemd unit ───────────────────────────────────────────────────────
cat > "$UNIT_FILE" <<EOF
[Unit]
Description=Steward host agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=$AGENT_USER
ExecStart=/usr/bin/python3 $AGENT_DIR/agent.py
Restart=on-failure
RestartSec=10
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/proc /sys

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now steward-agent.service

echo
echo "Steward host agent $AGENT_VERSION installed and running."
echo "Check status:  systemctl status steward-agent"
echo "Logs:          journalctl -u steward-agent -f"
```

### Design points

- **Agent binary served by Steward itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
- **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
- **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.
- **Fail-fast on preflight.** Missing systemd or python3 → clear error, exit. No half-installed agent.
- **Query-string token for install fetch is acceptable.** It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the `Authorization` header.
- **No auto-upgrade.** Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.

---

## UI surfaces

### Dashboard widgets (`steward/core/widgets.py`)

- The fleet glance (CPU, mem, disk /, load 1m, stale mark) is the core **Hosts — Overview** widget (`hosts_overview`); the old `host_resources` table was folded into it (#5213).
- **`host_resource_history`** — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.

### Per-host detail page

`GET /plugins/host_agent/<host_id>/`:

- Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
- Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
- Full history charts, reusing `widget_history.html` with a wider layout.
- Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
- Agent status: last heartbeat timestamp, "live/stale" flag.
- "Copy install command" button (for re-install / rotation) that fetches the current one-liner.

### Plugin settings page

List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host.

---

## Error handling

| Failure | Who handles it | Response |
|---|---|---|
| Agent can't reach Steward | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. |
| Steward rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. |
| Steward rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. |
| Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). |
| Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. |
| `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. |
| Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. |
| Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. |
| Ingest body malformed | Server | 400 + descriptive `detail`. |
| Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. |
| Agent clock skew > 5 min | Server | Accept, log WARN. |
| Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. |

### Edge cases explicitly accepted, not handled

- Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this.
- Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care.
- Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow.

### Idempotency note

v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later.

---

## Testing strategy

- **Unit tests**
  - Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content.
  - Config file parser: valid, invalid, comments, whitespace edge cases.
  - Ring buffer: fill, drain, overflow drops oldest.
  - Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
  - Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`.
- **Integration test**
  - Spin up the plugin's ingest route in-process (Quart test client).
  - Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content.
  - Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated.
- **Install script verification**
  - `sh -n install.sh` (syntax).
  - `shellcheck install.sh`.
  - Snapshot test of the rendered output against a committed fixture.
  - No live systemd-in-container test — high cost, low value for a one-time install script.
- **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable.

---

## Scope boundaries — deferred to follow-ups

- **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
- **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
- **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
- **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
- **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
- **macOS / BSD / Windows agents** — separate code paths.
- **Self-upgrade mechanism** — re-run installer for now.
- **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change.
- **Network and per-process metrics** — additive; agent v2.
- **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation.

---

## Constraints worth being aware of

These are not blockers; they are feature boundaries inherent to the design.

1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design.
3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.

---

## Implementation order (rough sketch — real plan comes from writing-plans)

1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`.
2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all.
3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth.
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
5. Install script template + `/install.sh` and `/agent.py` routes.
6. Settings page: list, add-host flow, rotate-token, delete.
7. Dashboard widgets: table + history chart. Register in `core/widgets.py`.
8. Per-host detail page.
9. `METRIC_CATALOG` entry in `steward/alerts/routes.py`.
10. Scheduler: stale-agent marker.
11. Tests at every stage; final integration test to tie it together.

The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.