FabledSteward/docs/plugins/host-agent-design.md

Host Agent Plugin — Design Spec

Date: 2026-04-14 Status: Approved, ready for implementation planning Scope: Steward plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.

---

Goal

Give Steward a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.

Non-goals

• Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
• Not a log collector. Time-series metrics only.
• Not a container monitor — that is the existing docker plugin's job. The host agent observes the host itself, not containers running on it.
• Not a network monitor. Traffic counters are deferred to a follow-up.
• Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).

---

Architecture

┌──────────────┐         POST /plugins/host_agent/ingest        ┌────────────────────────┐
│  Agent       │  ──────────────────────────────────────────→  │ Steward             │
│  (Python)    │   Authorization: Bearer <per-host-token>       │  host_agent plugin     │
│  on target   │   JSON body: metrics snapshot                  │                        │
│  host        │                                                │  routes.py  (ingest +  │
│              │  ← 200 OK or 401/400                           │   install.sh render +  │
│              │                                                │   UI)                  │
│  systemd     │                                                │                        │
│  unit,       │                                                │  writes to:            │
│  dedicated   │                                                │   - PluginMetric       │
│  user,       │                                                │     (time-series bus)  │
│  30s         │                                                │   - HostAgentRegistr-  │
│  interval,   │                                                │     ation (plugin-     │
│  in-memory   │                                                │     private table)    │
│  ring buffer │                                                │                        │
└──────────────┘                                                │  registers widgets +   │
                                                                │  METRIC_CATALOG entry  │
                                                                └────────────────────────┘

Core principles

• Agent is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
• Plugin is self-contained under plugins/host_agent/. Writes to the core PluginMetric time-series bus (the designed cross-plugin integration point) and its own private host_agent_registrations table. Writes to the core Host model for identity, but adds no new columns to it.
• Auth is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
• Install is a one-line curl | sh command rendered per-host by Steward with the token already baked in.
• Failure is handled at the agent (in-memory ring buffer + exponential backoff) so Steward outages don't lose brief-window data.

---

Agent design

File layout on the target host

/usr/local/lib/steward-agent/agent.py       # the agent (runs as steward-agent)
/usr/local/lib/steward-agent/maint.py       # the maintenance helper (runs as root, #5766)
/etc/steward-agent.conf                      # key=value config, 0640 root:steward-agent
/etc/systemd/system/steward-agent.service    # the agent's unit
/etc/systemd/system/steward-maint.{path,service}  # starts the helper on a request
/var/lib/steward-agent/requests/             # the one place the agent writes (0700 steward-agent)
/var/lib/steward-maint/results/              # the helper's outcomes (root, readable)
                                                # dedicated system user: steward-agent

Config file format

Flat key = value, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):

url = https://steward.home.lan
token = a1b2c3d4...
interval_seconds = 30
hostname = myhost              # optional; defaults to uname -n
mounts = /, /mnt/data          # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
docker_logs_enabled = true     # optional; on by default — per-host container-log kill-switch
docker_log_exclude = watchtower, noisy-svc   # optional; container names whose logs this host skips collecting

Container-log collection is on by default (no config needed). The two keys above are the per-host opt-outs; the operator-facing global toggle + exclude live in Settings and are enforced server-side (the push model has no channel to command an agent), so they don't require touching a host's conf.

Agent internals (function list, not classes)

• read_config(path) — parses the conf file into a dict.
• collect_cpu() — reads /proc/stat twice around a short sleep, returns overall CPU % (0–100 float).
• collect_memory() — parses /proc/meminfo, returns {total, used, available, swap_used} in bytes. used = total - available (the available field, not free — Linux monitoring footgun).
• collect_storage(mounts) — shutil.disk_usage() per mount, returns [{mount, total, used}, ...].
• collect_load() — reads /proc/loadavg, returns [1m, 5m, 15m].
• collect_uptime() — reads /proc/uptime, returns seconds since boot (int).
• collect_metadata() — os.uname() for kernel + arch, /etc/os-release for distro. Called once at startup and cached.
• collect_docker_logs(socket, containers, state, exclude) — (m79) per running container, fetches new log lines over the Docker socket with an incremental since cursor kept per container in state (a container's first interval seeds from a short tail). Demuxes the multiplexed stream, parses each line's RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records a truncation; the deferred lines come next interval). Folded into the sample as docker_logs; omitted when empty.
• build_payload() — assembles a snapshot from all collectors into one dict.
• post_payload(url, token, payloads) — POSTs a list of samples, returns success/failure.
• RingBuffer(maxlen=20) — tiny FIFO wrapper, drops oldest when full.
• main_loop() — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).

Target: ~300 lines total including docstrings. More than that is a smell that the agent is over-scoping.

Dependencies

Python 3.8+ stdlib only. urllib.request for the POST, plus json, os, shutil, time, socket, signal. No requests, no httpx, no TOML libraries.

Signals and lifecycle

• SIGHUP → re-read config file without restart (lets admins change interval/mounts cleanly).
• SIGTERM (from systemd) → finish current poll cycle → flush buffer one last time, with a stopping event saying whether the host is going down too (systemctl is-system-running says stopping) → exit.
• Systemd restart policy → Restart=on-failure, RestartSec=10.

Mounts that don't answer (1.14.0, #5760)

Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks statvfs in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as {"mount", "stalled": true}, and isn't probed again while its last probe is still blocked. Steward counts them as mounts_stalled.

Every Docker read is bounded (1.15.0, #5764)

One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a _steward note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own agent footprint (rss_bytes, cpu_pct), which Steward records as agent_rss_bytes and agent_cpu_pct.

What the host has waiting (1.16.0, #5765)

Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses /usr/lib/update-notifier/apt-check (pending;security), or apt list --upgradable when that's missing. On dnf hosts it uses dnf -C check-update, dnf -C updateinfo list --security, and needs-restarting -r for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as updates. Every sample also reads Debian's /var/run/reboot-required (and .pkgs) as reboot. Steward records updates_pending, updates_security and reboot_required (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. state["updates_due"] = True asks for a check on the next sample; a maintenance job uses it once it has updated the host.

A root helper for updates and reboots (1.17.0, #5766)

The agent can't update or reboot its host: it runs as steward-agent under NoNewPrivileges and ProtectSystem=strict, so no sudo is possible, and that is how it should stay. A separate helper, maint.py, runs as root, started by systemd when the agent leaves it a request:

• The agent writes <id>.json = {"id", "action", ["node"]} into /var/lib/steward-agent/requests, the one path its unit lets it write (ReadWritePaths=-…; the - keeps an old host without the directory starting). It writes a dot-file first and renames it, so the helper never sees half a request.
• steward-maint.path (PathExistsGlob=…/requests/*.json) starts steward-maint.service, a oneshot running maint.py as root.
• The helper opens each request with O_NOFOLLOW, wants a regular file under 4 KiB, and accepts only update, reboot, drain or undrain, an id of letters, digits and dashes, and a node id in the swarm's own 25-character form. It deletes every request, valid or not, and writes the outcome to /var/lib/steward-maint/results/<id>.json (root's; kept a week). A request whose id already has a result is not run again.
• update: apt-get update then dist-upgrade keeping existing config files, with NEEDRESTART_MODE=l so needrestart only lists services rather than restarting the Docker daemon under its containers; or dnf -y upgrade --refresh. The result says how many packages changed and whether a reboot is now required.
• reboot: writes rebooting, syncs, systemctl reboot. Any request after it waits for the next boot.
• drain / undrain: docker node update --availability drain|active, on the given node or this host's own. Only a manager can; a worker's helper says so in its result.

The agent reports each result's new state once, as a maintenance event with its next post that gets through (results older than a day are not news). When an update finishes it re-checks what's waiting straight away rather than at the next hourly check. Its metadata says maint_helper: true once the path unit is installed and it can write its request directory; the host's Agent panel shows that under Manage agent. Update agent installs the helper on a host that has an older agent.

So a stolen agent token can make its host update, reboot or drain, and nothing else. Steward handing a host a job and following it through the reboot is #5767 (below); the cadence that does it one host at a time is #5768.

A maintenance job, followed through the reboot (1.18.0, #5767)

Steward keeps one host_maintenance_jobs row per job and moves it on only from what the agents post, so a restart of Steward, or a reboot of the host Steward runs on, loses nothing. plugins/host_agent/maintenance.py has the states and transitions (model-free, unit-tested) and the database side.

queued → updating → [draining →] rebooting → [undraining →] done
                  ↘ done (no reboot needed, or the job may never reboot)
any state → failed (a step's result said so) | timed_out (its deadline passed)

• Hand-off. Steward's answer to an ingest post carries "maintenance": [{"id", "action", ["node"]}], the step that host's agent should carry out now. It is repeated on every post until a result for that step id arrives; the agent leaves each id for the helper once (take_maintenance), and the helper refuses an id it already has a result for, so a repeat, or a re-hand after the agent restarts, never runs twice. A step id is <job id>-<action>.
• Results are the maintenance events of #5766, matched to the job by step id, so a drain reported by a manager reaches the job of the node it drained.
• The reboot completes when the host's agent sends started with a boot_at after the reboot was handed out (60 s of clock slack). An agent restart on a host that stayed up has an old boot_at and changes nothing.
• Reboot only when required: the update's result says whether a reboot is needed; a job made with "never reboot" stops after the update.
• Swarm nodes are drained first and made active after. The node is found by hostname in the Docker plugin's swarm nodes; a manager drains itself, a worker is drained by a manager whose agent has the helper. With no such manager the job is refused rather than rebooting an undrained node.
• Deadlines (DEADLINES): queued 1 h, updating 90 min, draining 10 min, rebooting 30 min, undraining 10 min. A minute tick (host_agent_maintenance) times out a job past its deadline.
• A maintenance window scoped to the host's agent metrics (host_agent/<host>) opens when the job starts, lasts as long as the rest of the job could take, and ends when it does.
• Ingest applies all of this in a savepoint of its own: a job going wrong is logged and never costs the host its samples.

On the host's Agent panel: a running job leads what the host has waiting, with since-when; a finished one says what it did ("Done: 12 packages updated, rebooted"). Under Manage agent, Update now (with "never reboot") starts a job, or the panel says why one can't start.

A maintenance cadence: a group, one host at a time (#5768)

plugins/host_agent/cadence.py, driven by the same minute tick as the jobs. A plan (host_maintenance_plans) names an inventory group, a weekday, a time in a time zone (the creator's, by default), a window and a reboot policy: if_required (the default), always or never. The page is Hosts → Maintenance (/plugins/host_agent/maintenance).

• When. A plan's slot is the latest weekday-and-time at or before now, in its zone. A slot fires once (last_slot_at) and only while its window is open, so a Steward that was down over the slot starts late, not a day late. A new or resumed plan starts from the next slot. Run now starts a run without touching the schedule (the Ansible ruling on Run now).
• A run (host_maintenance_runs) fixes its hosts at the start, by name: the group's hosts through their linked Ansible targets. A host with no agent, or no maintenance helper, is skipped and listed.
• One at a time. Each host gets a #5767 job (run_id on the job). The next host starts only when the last is back in service: its agent posting in the last 3 min and, for a swarm node, the node ready/active and every replicated service at its replicas, judged only from swarm reports made after the job finished. A host not back within 30 min fails the run, and its job is marked failed with the reason.
• Stops. A failed or timed-out job fails the run. No host is started after the window; the run then stops and says how many weren't reached.
• Alerting is an ordinary alert rule on host_agent/<host>/maintenance_failed: 1 when the host's latest job (in 30 days) failed, timed out or didn't come back into service, 0 once a later one succeeds.
• The page leads with a running run and its hosts, then failed runs from the last week, then each plan with its schedule, next slot and last five runs. Each host's Agent panel shows its last maintenance job.

What happened to the host (1.14.0, #5756)

A post may carry events, said once each, with the next post that gets through:

• started — the agent starting, with boot_at (from /proc/uptime).
• stopping — on SIGTERM, with host_shutdown: true, false, or null when systemd can't say.
• lost_contact — from the first post that got no answer (or a proxy's 5xx) to the post carrying it.

Steward turns them into the host's downtime (host_downtime): reboot (a stop, then a boot after it), agent_stop (the host stayed up), unclean (a boot after the agent was last heard that nobody announced: power loss or a crash) and lost_contact. An announced stop explains the silence for reboot_grace_seconds (Settings → Plugins → host_agent, 10 minutes): seconds_since_seen reads 0 until then, and the host's other rules go quiet with it, as when it is offline. Past the grace, agent offline fires. The events live in memory, so a span the agent saw is lost if it restarts mid-outage; the boot time still reads the restart as unclean.

Logging

To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:

• INFO on start and on successful flush-after-buffered-samples.
• WARN on POST failure.
• ERROR on config errors, token rejection.

Identity

The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Steward looks up the Host row via HostAgentRegistration.token_hash, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.

Failure behavior

• POST fails → push sample to ring buffer (max 20), exponential backoff 30s → 60s → 120s → 300s cap, resets on first success.
• On next success → flush all buffered samples in a single batched POST.
• Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
• Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
• Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.

---

Plugin design (server-side)

File layout

Mirrors the existing plugins/snmp/ structure.

plugins/host_agent/
├── __init__.py
├── plugin.yaml                          # metadata + default config
├── routes.py                            # blueprint: ingest, install.sh, UI pages, settings actions
├── models.py                            # HostAgentRegistration SQLAlchemy model
├── agent.py                             # the Python agent, served to targets at GET /agent.py
├── migrations/
│   ├── env.py
│   ├── __init__.py
│   └── versions/
│       └── host_agent_001_initial.py    # creates host_agent_registrations table
├── templates/
│   ├── detail.html                      # per-host detail page
│   ├── widget_history.html              # dashboard history chart widget
│   ├── settings_list.html               # plugin settings: list of registered hosts
│   └── install.sh.j2                    # install script template (Jinja)
└── scheduler.py                         # periodic task: mark stale agents as "offline"

HostAgentRegistration model (plugin-private table)

Column	Type	Notes
id	UUID str	primary key
host_id	str, FK → hosts.id, unique	one registration per Host
token_hash	str	sha256(token); raw token shown only once at creation
token_created_at	datetime	for audit and rotation history
agent_version	str, nullable	reported on each ingest; updated on change
kernel	str, nullable	from os.uname() — e.g. 6.8.0-45-generic
distro	str, nullable	parsed from /etc/os-release — e.g. Ubuntu 24.04
arch	str, nullable	e.g. x86_64
last_seen_at	datetime, nullable	updated on every successful ingest; source of "is this agent alive"
created_at / updated_at	datetime	standard

Routes (blueprint prefix /plugins/host_agent)

Route	Auth	Purpose
POST /ingest	bearer token	Agent push. Validates sha256(Authorization header) against token_hash → writes PluginMetric rows + updates HostAgentRegistration. Returns 200, 400, or 401.
GET /install.sh	query-param token	Renders the Jinja install.sh.j2 template with URL, token, and agent version substituted. Returns text/plain.
GET /agent.py	none	Serves the bundled agent source from plugins/host_agent/agent.py as text/x-python. Fetched by the install script.
GET /<host_id>/	session	Per-host detail page.
POST /settings/add-host	admin	Creates (or reuses) a Host row + creates HostAgentRegistration + generates token. Returns the install one-liner for UI display.
POST /settings/<host_id>/rotate-token	admin	Generates a new token, invalidates the old one, returns new install one-liner.
POST /settings/<host_id>/delete	admin	Removes HostAgentRegistration. Does not delete the backing Host row unless no other monitors reference it.
GET /widget	session	HTMX partial: fleet table widget body.
GET /widget/history	session	HTMX partial: history chart widget body (takes host_id query param).

Token handling

• Tokens are generated with secrets.token_urlsafe(32) at host creation.
• Only sha256(token) is persisted.
• The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
• Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.

Stale-agent scheduler

One ScheduledTask running every 60 seconds that flags HostAgentRegistration rows with last_seen_at > (3 × interval_seconds) ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.

Note: the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.

As built: the task records host_agent/<host>/seconds_since_seen each minute, and "Agent offline" is an ordinary alert rule on it (> 180). It counts from the later of last_seen_at and the agent's newest post to arrive (kept in memory, scheduler.arrivals). last_seen_at is written in the ingest's own transaction, so a post Steward received but couldn't store (its storage stalled, #5514) would otherwise read as the agent's silence. Nothing counts from before Steward last started watching without a break (#5441).

METRIC_CATALOG registration

Add to steward/alerts/routes.py:

"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
               "swap_used_bytes", "disk_used_pct_worst", "load_1m",
               "load_5m", "load_15m", "uptime_secs"],

This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Steward core grows a plugin-registered catalog API — deferred as future core work.

---

Wire format

Request

POST /plugins/host_agent/ingest HTTP/1.1
Host: steward.home.lan
Authorization: Bearer a1b2c3d4e5f6...
Content-Type: application/json

{
  "agent_version": "1.0.0",
  "hostname": "myhost",
  "metadata": {
    "kernel": "6.8.0-45-generic",
    "distro": "Ubuntu 24.04",
    "arch": "x86_64"
  },
  "samples": [
    {
      "ts": "2026-04-14T02:55:00Z",
      "cpu_pct": 12.4,
      "mem": {
        "total_bytes": 33554432000,
        "used_bytes":  18253611008,
        "available_bytes": 15300820992,
        "swap_used_bytes": 0
      },
      "load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
      "uptime_secs": 482934,
      "storage": [
        { "mount": "/",         "total_bytes": 499289948160, "used_bytes": 312456789012 },
        { "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
      ]
    }
  ]
}

Design points

• samples is always a list, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs [current, ...buffered] in a single request. One-path server code.
• ts is agent-reported, ISO-8601 UTC. Server trusts it for PluginMetric.recorded_at. Host clocks drift but NTP is reliable enough for our purposes.
• If |ts - now| > 5 minutes, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
• metadata is sent on every POST, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
• Raw bytes, not percentages, for memory and storage. Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
• CPU is the one exception — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
• Container logs (m79) ride in the same push as docker_logs: a list of {container, stream, ts, line} records — incremental since the previous interval. This pushes the same direction as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into docker_logs, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. docker_logs is omitted when there's nothing new.

Server expansion into PluginMetric rows

Per sample, the server writes:

metric_name	resource_name	derivation
cpu_pct	<host.name>	sample.cpu_pct as-is
mem_used_pct	<host.name>	100 * (total - available) / total
mem_available_bytes	<host.name>	sample.mem.available_bytes
swap_used_bytes	<host.name>	sample.mem.swap_used_bytes
load_1m	<host.name>	sample.load["1m"]
load_5m	<host.name>	sample.load["5m"]
load_15m	<host.name>	sample.load["15m"]
uptime_secs	<host.name>	sample.uptime_secs
disk_used_pct	<host.name>:<mount>	100 * used / total per mount
disk_used_bytes	<host.name>:<mount>	raw per mount
disk_total_bytes	<host.name>:<mount>	raw per mount
disk_used_pct_worst	<host.name>	max(disk_used_pct across mounts) — fleet table column

All values are float. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within PluginMetric and the existing cleanup path.

Plus one update to HostAgentRegistration:

reg.last_seen_at = max(sample.ts for sample in samples)
reg.agent_version = payload.agent_version
reg.kernel = payload.metadata.kernel
reg.distro = payload.metadata.distro
reg.arch   = payload.metadata.arch

(Only written if changed, to keep updated_at meaningful.)

Response

{ "ok": true, "samples_accepted": 3 }

Failures:

{ "ok": false, "error": "invalid_token" }                              // 401
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" }  // 400

Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).

---

Install script

The one-liner (what the UI shows)

curl -sSL 'https://steward.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh

The UI also offers a "review script before running" link that opens a modal showing the two-step form:

curl -sSL '…' -o install.sh
less install.sh
sudo sh install.sh

Rendered script structure (Jinja template install.sh.j2)

plugins/host_agent/templates/install.sh.j2 is the script; it is not copied here, because a copy drifts. It checks it runs as root with systemd and python3, handles --uninstall, creates the steward-agent user, fetches agent.py, writes the config, then fetches the maintenance helper and the three units from /plugins/host_agent/files/ (the same files the playbooks install), creates the maintenance directories, and enables steward-maint.path and the agent.

Design points

• One copy of every file a host installs. The units and the maintenance helper live in plugins/host_agent/files/ and are served at /plugins/host_agent/files/<name> (a fixed list; anything else is 404). install.sh and the install, provision and update playbooks all fetch them, so the unit can't differ between a host installed by hand and one installed by Ansible (#5766).
• Agent binary served by Steward itself. GET /plugins/host_agent/agent.py serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
• Systemd hardening is cheap and correct. NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, ReadOnlyPaths=/proc /sys. The agent only needs read access to /proc, /sys, and mount points; denying everything else narrows blast radius.
• Uninstall is a first-class flag, not a separate script. Same one-liner with --uninstall appended.
• Fail-fast on preflight. Missing systemd or python3 → clear error, exit. No half-installed agent.
• Query-string token for install fetch is acceptable. It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the Authorization header.
• No auto-upgrade. Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.

---

UI surfaces

Dashboard widgets (steward/core/widgets.py)

• The fleet glance (CPU, mem, disk /, load 1m, stale mark) is the core Hosts — Overview widget (hosts_overview); the old host_resources table was folded into it (#5213).
• host_resource_history — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.

Per-host detail page

GET /plugins/host_agent/<host_id>/:

• Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
• Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
• Full history charts, reusing widget_history.html with a wider layout.
• Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
• Agent status: last heartbeat timestamp, "live/stale" flag.
• "Copy install command" button (for re-install / rotation) that fetches the current one-liner.

Plugin settings page

List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses Host row → creates HostAgentRegistration → generates token → displays the curl install one-liner), and a "rotate token" action per host.

---

Error handling

Failure	Who handles it	Response
Agent can't reach Steward	Agent	Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success.
Steward rejects with 401	Agent	Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed.
Steward rejects with 400	Agent	Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew.
Ring buffer full	Agent	Drop oldest, keep newest. DEBUG log (expected during outages).
Config file missing or malformed	Agent	Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal.
/proc/stat read fails between samples	Agent	Skip CPU for this cycle, still POST the rest. Partial samples allowed.
Mount disappears between reads	Agent	Skip that mount for this sample. If permanent, its rows simply stop appearing.
Ingest token not found	Server	401 + {"error": "invalid_token"}.
Ingest body malformed	Server	400 + descriptive detail.
Ingest writes fail mid-batch	Server	500 + rollback. Agent retries whole batch on next cycle.
Agent clock skew > 5 min	Server	Accept, log WARN.
Host deleted from UI	Server	HostAgentRegistration removed; agent's next POST → 401 → admin re-adds or uninstalls.

Edge cases explicitly accepted, not handled

• Two agents sharing a token → last-write-wins on last_seen_at, metrics get stored under the same resource_name. Admin should never do this.
• Agent clock running backward (NTP correction) → recorded_at values non-monotonic. Charts render fine; the DB doesn't care.
• Hostname change on target → HostAgentRegistration.host_id is still the token's identity; resource_name tracks Host.name. Admin edits Host.name if they want the display to follow.

Idempotency note

v1 doesn't enforce unique constraints on PluginMetric rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on (source_module, resource_name, metric_name, recorded_at) later.

---

Testing strategy

• Unit tests
  • Agent collectors against fixture /proc/stat, /proc/meminfo, /proc/loadavg, /proc/uptime content.
  • Config file parser: valid, invalid, comments, whitespace edge cases.
  • Ring buffer: fill, drain, overflow drops oldest.
  • Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
  • Plugin migration smoke test: alembic upgrade head in a temp DB, then downgrade base.
• Integration test
  • Spin up the plugin's ingest route in-process (Quart test client).
  • Feed a real payload generated by the agent's build_payload() using mocked /proc content.
  • Assert PluginMetric rows land correctly and HostAgentRegistration is updated.
• Install script verification
  • sh -n install.sh (syntax).
  • shellcheck install.sh.
  • Snapshot test of the rendered output against a committed fixture.
  • No live systemd-in-container test — high cost, low value for a one-time install script.
• Agent is importable as a Python module. main_loop() is guarded by if __name__ == "__main__":, so everything else is unit-testable.

---

Scope boundaries — deferred to follow-ups

• Ansible-action alert type (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
• Plugin synergy detection (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
• Fleet-provision mode — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
• Event signals — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
• Non-systemd init systems — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
• macOS / BSD / Windows agents — separate code paths.
• Self-upgrade mechanism — re-run installer for now.
• Disk buffering when RAM ring fills — drop-in upgrade, no protocol change.
• Network and per-process metrics — additive; agent v2.
• metadata_json column on core Host — rejected in favor of strict plugin isolation.

---

Constraints worth being aware of

These are not blockers; they are feature boundaries inherent to the design.

1. Stdlib-only Python is a ceiling. Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, psutil-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
2. Interval push is not event push. OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a sibling endpoint, not an extension of this design.
3. PluginMetric only stores floats. String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.

---

Implementation order (rough sketch — real plan comes from writing-plans)

1. Plugin scaffolding: plugin.yaml, __init__.py, empty routes blueprint, migration for host_agent_registrations.
2. Agent v1 as a standalone script: all collectors + config parser + build_payload() working locally. Testable without any server at all.
3. Server /ingest route: accepts payload, writes PluginMetric, updates HostAgentRegistration. Token auth.
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
5. Install script template + /install.sh and /agent.py routes.
6. Settings page: list, add-host flow, rotate-token, delete.
7. Dashboard widgets: table + history chart. Register in core/widgets.py.
8. Per-host detail page.
9. METRIC_CATALOG entry in steward/alerts/routes.py.
10. Scheduler: stale-agent marker.
11. Tests at every stage; final integration test to tie it together.

The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.