FabledSteward/docs/plugins/host-agent-design.md

# Host Agent Plugin — Design Spec

**Date:** 2026-04-14
**Status:** Approved, ready for implementation planning
**Scope:** Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.

---

## Goal

Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.

## Non-goals

- Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
- Not a log collector. Time-series metrics only.
- Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it.
- Not a network monitor. Traffic counters are deferred to a follow-up.
- Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).

---

## Architecture

```
┌──────────────┐         POST /plugins/host_agent/ingest        ┌────────────────────────┐
│  Agent       │  ──────────────────────────────────────────→  │ Steward             │
│  (Python)    │   Authorization: Bearer <per-host-token>       │  host_agent plugin     │
│  on target   │   JSON body: metrics snapshot                  │                        │
│  host        │                                                │  routes.py  (ingest +  │
│              │  ← 200 OK or 401/400                           │   install.sh render +  │
│              │                                                │   UI)                  │
│  systemd     │                                                │                        │
│  unit,       │                                                │  writes to:            │
│  dedicated   │                                                │   - PluginMetric       │
│  user,       │                                                │     (time-series bus)  │
│  30s         │                                                │   - HostAgentRegistr-  │
│  interval,   │                                                │     ation (plugin-     │
│  in-memory   │                                                │     private table)    │
│  ring buffer │                                                │                        │
└──────────────┘                                                │  registers widgets +   │
                                                                │  METRIC_CATALOG entry  │
                                                                └────────────────────────┘
```

### Core principles

- **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
- **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it.
- **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
- **Install** is a one-line `curl | sh` command rendered per-host by Steward with the token already baked in.
- **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data.

---

## Agent design

### File layout on the target host

```
/usr/local/lib/steward-agent/agent.py       # the agent (runs as steward-agent)
/usr/local/lib/steward-agent/maint.py       # the maintenance helper (runs as root, #5766)
/etc/steward-agent.conf                      # key=value config, 0640 root:steward-agent
/etc/systemd/system/steward-agent.service    # the agent's unit
/etc/systemd/system/steward-maint.{path,service}  # starts the helper on a request
/var/lib/steward-agent/requests/             # the one place the agent writes (0700 steward-agent)
/var/lib/steward-maint/results/              # the helper's outcomes (root, readable)
                                                # dedicated system user: steward-agent
```

### Config file format

Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):

```
url = https://steward.home.lan
token = a1b2c3d4...
interval_seconds = 30
hostname = myhost              # optional; defaults to uname -n
mounts = /, /mnt/data          # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
docker_logs_enabled = true     # optional; on by default — per-host container-log kill-switch
docker_log_exclude = watchtower, noisy-svc   # optional; container names whose logs this host skips collecting
```

Container-log collection is on by default (no config needed). The two keys above
are the per-host opt-outs; the operator-facing global toggle + exclude live in
Settings and are enforced server-side (the push model has no channel to command
an agent), so they don't require touching a host's conf.

### Agent internals (function list, not classes)

- `read_config(path)` — parses the conf file into a dict.
- `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float).
- `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun).
- `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`.
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
  container, fetches new log lines over the Docker socket with an incremental
  `since` cursor kept per container in `state` (a container's first interval
  seeds from a short tail). Demuxes the multiplexed stream, parses each line's
  RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
  a truncation; the deferred lines come next interval). Folded into the sample as
  `docker_logs`; omitted when empty.
- `build_payload()` — assembles a snapshot from all collectors into one dict.
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).

**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.

### Dependencies

Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries.

### Signals and lifecycle

- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly).
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time, with a `stopping` event saying whether the host is going down too (`systemctl is-system-running` says `stopping`) → exit.
- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`.

### Mounts that don't answer (1.14.0, #5760)

Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks `statvfs` in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as `{"mount", "stalled": true}`, and isn't probed again while its last probe is still blocked. Steward counts them as `mounts_stalled`.

### Every Docker read is bounded (1.15.0, #5764)

One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a `_steward` note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own `agent` footprint (`rss_bytes`, `cpu_pct`), which Steward records as `agent_rss_bytes` and `agent_cpu_pct`.

### What the host has waiting (1.16.0, #5765)

Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses `/usr/lib/update-notifier/apt-check` (pending;security), or `apt list --upgradable` when that's missing. On dnf hosts it uses `dnf -C check-update`, `dnf -C updateinfo list --security`, and `needs-restarting -r` for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as `updates`. Every sample also reads Debian's `/var/run/reboot-required` (and `.pkgs`) as `reboot`. Steward records `updates_pending`, `updates_security` and `reboot_required` (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. `state["updates_due"] = True` asks for a check on the next sample; a maintenance job uses it once it has updated the host.


### A root helper for updates and reboots (1.17.0, #5766)

The agent can't update or reboot its host: it runs as `steward-agent` under
`NoNewPrivileges` and `ProtectSystem=strict`, so no sudo is possible, and that
is how it should stay. A separate helper, `maint.py`, runs as root, started by
systemd when the agent leaves it a request:

- The agent writes `<id>.json` = `{"id", "action", ["node"]}` into
  `/var/lib/steward-agent/requests`, the one path its unit lets it write
  (`ReadWritePaths=-…`; the `-` keeps an old host without the directory
  starting). It writes a dot-file first and renames it, so the helper never
  sees half a request.
- `steward-maint.path` (`PathExistsGlob=…/requests/*.json`) starts
  `steward-maint.service`, a oneshot running `maint.py` as root.
- The helper opens each request with `O_NOFOLLOW`, wants a regular file under
  4 KiB, and accepts only `update`, `reboot`, `drain` or `undrain`, an id of
  letters, digits and dashes, and a node id in the swarm's own 25-character
  form. It deletes every request, valid or not, and writes the outcome to
  `/var/lib/steward-maint/results/<id>.json` (root's; kept a week). A request
  whose id already has a result is not run again.
- **update**: `apt-get update` then `dist-upgrade` keeping existing config
  files, with `NEEDRESTART_MODE=l` so needrestart only lists services rather
  than restarting the Docker daemon under its containers; or
  `dnf -y upgrade --refresh`. The result says how many packages changed and
  whether a reboot is now required.
- **reboot**: writes `rebooting`, syncs, `systemctl reboot`. Any request after
  it waits for the next boot.
- **drain / undrain**: `docker node update --availability drain|active`, on
  the given node or this host's own. Only a manager can; a worker's helper says
  so in its result.

The agent reports each result's new state once, as a `maintenance` event with
its next post that gets through (results older than a day are not news). When
an update finishes it re-checks what's waiting straight away rather than at
the next hourly check. Its metadata says `maint_helper: true` once the path
unit is installed and it can write its request directory; the host's Agent
panel shows that under Manage agent. **Update agent** installs the helper on a
host that has an older agent.

So a stolen agent token can make its host update, reboot or drain, and nothing
else. Steward handing a host a job and following it through the reboot is
#5767 (below); the cadence that does it one host at a time is #5768.

### A maintenance job, followed through the reboot (1.18.0, #5767)

Steward keeps one `host_maintenance_jobs` row per job and moves it on only from
what the agents post, so a restart of Steward, or a reboot of the host Steward
runs on, loses nothing. `plugins/host_agent/maintenance.py` has the states and
transitions (model-free, unit-tested) and the database side.

```
queued → updating → [draining →] rebooting → [undraining →] done
                  ↘ done (no reboot needed, or the job may never reboot)
any state → failed (a step's result said so) | timed_out (its deadline passed)
```

- **Hand-off.** Steward's answer to an ingest post carries
  `"maintenance": [{"id", "action", ["node"]}]`, the step that host's agent
  should carry out now. It is repeated on every post until a result for that
  step id arrives; the agent leaves each id for the helper once
  (`take_maintenance`), and the helper refuses an id it already has a result
  for, so a repeat, or a re-hand after the agent restarts, never runs twice.
  A step id is `<job id>-<action>`.
- **Results** are the `maintenance` events of #5766, matched to the job by step
  id, so a drain reported by a manager reaches the job of the node it drained.
- **The reboot completes** when the host's agent sends `started` with a
  `boot_at` after the reboot was handed out (60 s of clock slack). An agent
  restart on a host that stayed up has an old `boot_at` and changes nothing.
- **Reboot only when required**: the update's result says whether a reboot is
  needed; a job made with "never reboot" stops after the update.
- **Swarm nodes** are drained first and made active after. The node is found
  by hostname in the Docker plugin's swarm nodes; a manager drains itself, a
  worker is drained by a manager whose agent has the helper. With no such
  manager the job is refused rather than rebooting an undrained node.
- **Deadlines** (`DEADLINES`): queued 1 h, updating 90 min, draining 10 min,
  rebooting 30 min, undraining 10 min. A minute tick (`host_agent_maintenance`)
  times out a job past its deadline.
- **A maintenance window** scoped to the host's agent metrics
  (`host_agent/<host>`) opens when the job starts, lasts as long as the rest of
  the job could take, and ends when it does.
- Ingest applies all of this in a savepoint of its own: a job going wrong is
  logged and never costs the host its samples.

On the host's Agent panel: a running job leads what the host has waiting, with
since-when; a finished one says what it did ("Done: 12 packages updated,
rebooted"). Under Manage agent, **Update now** (with "never reboot") starts a
job, or the panel says why one can't start.

### What happened to the host (1.14.0, #5756)

A post may carry `events`, said once each, with the next post that gets through:

- `started` — the agent starting, with `boot_at` (from `/proc/uptime`).
- `stopping` — on SIGTERM, with `host_shutdown`: true, false, or null when systemd can't say.
- `lost_contact` — `from` the first post that got no answer (or a proxy's 5xx) `to` the post carrying it.

Steward turns them into the host's downtime (`host_downtime`): **reboot** (a stop, then a boot after it), **agent_stop** (the host stayed up), **unclean** (a boot after the agent was last heard that nobody announced: power loss or a crash) and **lost_contact**. An announced stop explains the silence for `reboot_grace_seconds` (Settings → Plugins → host_agent, 10 minutes): `seconds_since_seen` reads 0 until then, and the host's other rules go quiet with it, as when it is offline. Past the grace, agent offline fires. The events live in memory, so a span the agent saw is lost if it restarts mid-outage; the boot time still reads the restart as unclean.

### Logging

To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:
- `INFO` on start and on successful flush-after-buffered-samples.
- `WARN` on POST failure.
- `ERROR` on config errors, token rejection.

### Identity

The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.

### Failure behavior

- POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success.
- On next success → flush all buffered samples in a single batched POST.
- Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
- Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
- Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.

---

## Plugin design (server-side)

### File layout

Mirrors the existing `plugins/snmp/` structure.

```
plugins/host_agent/
├── __init__.py
├── plugin.yaml                          # metadata + default config
├── routes.py                            # blueprint: ingest, install.sh, UI pages, settings actions
├── models.py                            # HostAgentRegistration SQLAlchemy model
├── agent.py                             # the Python agent, served to targets at GET /agent.py
├── migrations/
│   ├── env.py
│   ├── __init__.py
│   └── versions/
│       └── host_agent_001_initial.py    # creates host_agent_registrations table
├── templates/
│   ├── detail.html                      # per-host detail page
│   ├── widget_history.html              # dashboard history chart widget
│   ├── settings_list.html               # plugin settings: list of registered hosts
│   └── install.sh.j2                    # install script template (Jinja)
└── scheduler.py                         # periodic task: mark stale agents as "offline"
```

### `HostAgentRegistration` model (plugin-private table)

| Column | Type | Notes |
|---|---|---|
| `id` | UUID str | primary key |
| `host_id` | str, FK → `hosts.id`, unique | one registration per Host |
| `token_hash` | str | `sha256(token)`; raw token shown only once at creation |
| `token_created_at` | datetime | for audit and rotation history |
| `agent_version` | str, nullable | reported on each ingest; updated on change |
| `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` |
| `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` |
| `arch` | str, nullable | e.g. `x86_64` |
| `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" |
| `created_at` / `updated_at` | datetime | standard |

### Routes (blueprint prefix `/plugins/host_agent`)

| Route | Auth | Purpose |
|---|---|---|
| `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. |
| `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. |
| `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. |
| `GET /<host_id>/` | session | Per-host detail page. |
| `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. |
| `POST /settings/<host_id>/rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. |
| `POST /settings/<host_id>/delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. |
| `GET /widget` | session | HTMX partial: fleet table widget body. |
| `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). |

### Token handling

- Tokens are generated with `secrets.token_urlsafe(32)` at host creation.
- Only `sha256(token)` is persisted.
- The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
- Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.

### Stale-agent scheduler

One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.

**Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.

**As built:** the task records `host_agent/<host>/seconds_since_seen` each minute, and "Agent offline" is an ordinary alert rule on it (`> 180`). It counts from the later of `last_seen_at` and the agent's newest post to *arrive* (kept in memory, `scheduler.arrivals`). `last_seen_at` is written in the ingest's own transaction, so a post Steward received but couldn't store (its storage stalled, #5514) would otherwise read as the agent's silence. Nothing counts from before Steward last started watching without a break (#5441).

### `METRIC_CATALOG` registration

Add to `steward/alerts/routes.py`:

```python
"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
               "swap_used_bytes", "disk_used_pct_worst", "load_1m",
               "load_5m", "load_15m", "uptime_secs"],
```

This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work.

---

## Wire format

### Request

```http
POST /plugins/host_agent/ingest HTTP/1.1
Host: steward.home.lan
Authorization: Bearer a1b2c3d4e5f6...
Content-Type: application/json
```

```json
{
  "agent_version": "1.0.0",
  "hostname": "myhost",
  "metadata": {
    "kernel": "6.8.0-45-generic",
    "distro": "Ubuntu 24.04",
    "arch": "x86_64"
  },
  "samples": [
    {
      "ts": "2026-04-14T02:55:00Z",
      "cpu_pct": 12.4,
      "mem": {
        "total_bytes": 33554432000,
        "used_bytes":  18253611008,
        "available_bytes": 15300820992,
        "swap_used_bytes": 0
      },
      "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
      "uptime_secs": 482934,
      "storage": [
        { "mount": "/",         "total_bytes": 499289948160, "used_bytes": 312456789012 },
        { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
      ]
    }
  ]
}
```

### Design points

- **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code.
- **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes.
- **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.

### Server expansion into `PluginMetric` rows

Per sample, the server writes:

| `metric_name` | `resource_name` | derivation |
|---|---|---|
| `cpu_pct` | `<host.name>` | `sample.cpu_pct` as-is |
| `mem_used_pct` | `<host.name>` | `100 * (total - available) / total` |
| `mem_available_bytes` | `<host.name>` | `sample.mem.available_bytes` |
| `swap_used_bytes` | `<host.name>` | `sample.mem.swap_used_bytes` |
| `load_1m` | `<host.name>` | `sample.load["1m"]` |
| `load_5m` | `<host.name>` | `sample.load["5m"]` |
| `load_15m` | `<host.name>` | `sample.load["15m"]` |
| `uptime_secs` | `<host.name>` | `sample.uptime_secs` |
| `disk_used_pct` | `<host.name>:<mount>` | `100 * used / total` per mount |
| `disk_used_bytes` | `<host.name>:<mount>` | raw per mount |
| `disk_total_bytes` | `<host.name>:<mount>` | raw per mount |
| `disk_used_pct_worst` | `<host.name>` | `max(disk_used_pct across mounts)` — fleet table column |

All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path.

Plus one update to `HostAgentRegistration`:

```python
reg.last_seen_at = max(sample.ts for sample in samples)
reg.agent_version = payload.agent_version
reg.kernel = payload.metadata.kernel
reg.distro = payload.metadata.distro
reg.arch   = payload.metadata.arch
```

(Only written if changed, to keep `updated_at` meaningful.)

### Response

```json
{ "ok": true, "samples_accepted": 3 }
```

Failures:

```json
{ "ok": false, "error": "invalid_token" }                              // 401
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" }  // 400
```

Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).

---

## Install script

### The one-liner (what the UI shows)

```
curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh
```

The UI also offers a "review script before running" link that opens a modal showing the two-step form:

```
curl -sSL '…' -o install.sh
less install.sh
sudo sh install.sh
```

### Rendered script structure (Jinja template `install.sh.j2`)

`plugins/host_agent/templates/install.sh.j2` is the script; it is not copied
here, because a copy drifts. It checks it runs as root with systemd and
python3, handles `--uninstall`, creates the `steward-agent` user, fetches
`agent.py`, writes the config, then fetches the maintenance helper and the
three units from `/plugins/host_agent/files/` (the same files the playbooks
install), creates the maintenance directories, and enables
`steward-maint.path` and the agent.

### Design points

- **One copy of every file a host installs.** The units and the maintenance helper live in `plugins/host_agent/files/` and are served at `/plugins/host_agent/files/<name>` (a fixed list; anything else is 404). install.sh and the install, provision and update playbooks all fetch them, so the unit can't differ between a host installed by hand and one installed by Ansible (#5766).
- **Agent binary served by Steward itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
- **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
- **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.
- **Fail-fast on preflight.** Missing systemd or python3 → clear error, exit. No half-installed agent.
- **Query-string token for install fetch is acceptable.** It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the `Authorization` header.
- **No auto-upgrade.** Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.

---

## UI surfaces

### Dashboard widgets (`steward/core/widgets.py`)

- The fleet glance (CPU, mem, disk /, load 1m, stale mark) is the core **Hosts — Overview** widget (`hosts_overview`); the old `host_resources` table was folded into it (#5213).
- **`host_resource_history`** — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.

### Per-host detail page

`GET /plugins/host_agent/<host_id>/`:

- Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
- Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
- Full history charts, reusing `widget_history.html` with a wider layout.
- Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
- Agent status: last heartbeat timestamp, "live/stale" flag.
- "Copy install command" button (for re-install / rotation) that fetches the current one-liner.

### Plugin settings page

List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host.

---

## Error handling

| Failure | Who handles it | Response |
|---|---|---|
| Agent can't reach Steward | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. |
| Steward rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. |
| Steward rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. |
| Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). |
| Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. |
| `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. |
| Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. |
| Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. |
| Ingest body malformed | Server | 400 + descriptive `detail`. |
| Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. |
| Agent clock skew > 5 min | Server | Accept, log WARN. |
| Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. |

### Edge cases explicitly accepted, not handled

- Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this.
- Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care.
- Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow.

### Idempotency note

v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later.

---

## Testing strategy

- **Unit tests**
  - Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content.
  - Config file parser: valid, invalid, comments, whitespace edge cases.
  - Ring buffer: fill, drain, overflow drops oldest.
  - Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
  - Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`.
- **Integration test**
  - Spin up the plugin's ingest route in-process (Quart test client).
  - Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content.
  - Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated.
- **Install script verification**
  - `sh -n install.sh` (syntax).
  - `shellcheck install.sh`.
  - Snapshot test of the rendered output against a committed fixture.
  - No live systemd-in-container test — high cost, low value for a one-time install script.
- **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable.

---

## Scope boundaries — deferred to follow-ups

- **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
- **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
- **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
- **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
- **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
- **macOS / BSD / Windows agents** — separate code paths.
- **Self-upgrade mechanism** — re-run installer for now.
- **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change.
- **Network and per-process metrics** — additive; agent v2.
- **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation.

---

## Constraints worth being aware of

These are not blockers; they are feature boundaries inherent to the design.

1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design.
3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.

---

## Implementation order (rough sketch — real plan comes from writing-plans)

1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`.
2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all.
3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth.
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
5. Install script template + `/install.sh` and `/agent.py` routes.
6. Settings page: list, add-host flow, rotate-token, delete.
7. Dashboard widgets: table + history chart. Register in `core/widgets.py`.
8. Per-host detail page.
9. `METRIC_CATALOG` entry in `steward/alerts/routes.py`.
10. Scheduler: stale-agent marker.
11. Tests at every stage; final integration test to tie it together.

The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.