docs(plugins): host-agent plugin design spec
Design for a push-based Python agent that collects CPU/memory/storage/ load/uptime from remote Linux hosts and POSTs to a new host_agent plugin. Uses per-host bearer tokens, systemd install via curl|sh, in-memory ring buffer with exponential backoff, and writes to the existing PluginMetric bus plus a plugin-private registrations table. No core schema changes — plugin stays strictly isolated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,535 @@
|
||||
# Host Agent Plugin — Design Spec
|
||||
|
||||
**Date:** 2026-04-14
|
||||
**Status:** Approved, ready for implementation planning
|
||||
**Scope:** Roundtable plugin for remote host resource monitoring (CPU, memory, storage, load, uptime) via a lightweight Python push agent.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Give Roundtable a fleet-glance view of remote Linux hosts — current resource usage, history, and alert integration — without requiring agentless SSH/Ansible round-trips, a new toolchain, or SNMP gymnastics on every target.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not a fleet-management system (no remote agent updates, no centrally-pushed config beyond the initial install).
|
||||
- Not a log collector. Time-series metrics only.
|
||||
- Not a container monitor — that is the existing `docker` plugin's job. The host agent observes the host itself, not containers running on it.
|
||||
- Not a network monitor. Traffic counters are deferred to a follow-up.
|
||||
- Not event-driven. Sampling is interval-based; OOM-killer/FS-read-only/crash events are explicitly a separate design (see follow-ups).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐ POST /plugins/host_agent/ingest ┌────────────────────────┐
|
||||
│ Agent │ ──────────────────────────────────────────→ │ Roundtable │
|
||||
│ (Python) │ Authorization: Bearer <per-host-token> │ host_agent plugin │
|
||||
│ on target │ JSON body: metrics snapshot │ │
|
||||
│ host │ │ routes.py (ingest + │
|
||||
│ │ ← 200 OK or 401/400 │ install.sh render + │
|
||||
│ │ │ UI) │
|
||||
│ systemd │ │ │
|
||||
│ unit, │ │ writes to: │
|
||||
│ dedicated │ │ - PluginMetric │
|
||||
│ user, │ │ (time-series bus) │
|
||||
│ 30s │ │ - HostAgentRegistr- │
|
||||
│ interval, │ │ ation (plugin- │
|
||||
│ in-memory │ │ private table) │
|
||||
│ ring buffer │ │ │
|
||||
└──────────────┘ │ registers widgets + │
|
||||
│ METRIC_CATALOG entry │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
### Core principles
|
||||
|
||||
- **Agent** is a single Python file. No external dependencies beyond Python 3.8+ stdlib.
|
||||
- **Plugin** is self-contained under `plugins/host_agent/`. Writes to the core `PluginMetric` time-series bus (the designed cross-plugin integration point) and its own private `host_agent_registrations` table. Writes to the core `Host` model for identity, but adds no new columns to it.
|
||||
- **Auth** is per-host bearer tokens, minted on "Add host" in the plugin's settings page.
|
||||
- **Install** is a one-line `curl | sh` command rendered per-host by Roundtable with the token already baked in.
|
||||
- **Failure** is handled at the agent (in-memory ring buffer + exponential backoff) so Roundtable outages don't lose brief-window data.
|
||||
|
||||
---
|
||||
|
||||
## Agent design
|
||||
|
||||
### File layout on the target host
|
||||
|
||||
```
|
||||
/usr/local/lib/roundtable-agent/agent.py # the script, target ~300 lines
|
||||
/etc/roundtable-agent.conf # key=value config, 0640 root:roundtable-agent
|
||||
/etc/systemd/system/roundtable-agent.service # unit file
|
||||
# dedicated system user: roundtable-agent
|
||||
```
|
||||
|
||||
### Config file format
|
||||
|
||||
Flat `key = value`, parsed by a ~20-line homegrown parser (no TOML/YAML dependency):
|
||||
|
||||
```
|
||||
url = https://roundtable.home.lan
|
||||
token = a1b2c3d4...
|
||||
interval_seconds = 30
|
||||
hostname = myhost # optional; defaults to uname -n
|
||||
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
|
||||
```
|
||||
|
||||
### Agent internals (function list, not classes)
|
||||
|
||||
- `read_config(path)` — parses the conf file into a dict.
|
||||
- `collect_cpu()` — reads `/proc/stat` twice around a short sleep, returns overall CPU % (0–100 float).
|
||||
- `collect_memory()` — parses `/proc/meminfo`, returns `{total, used, available, swap_used}` in bytes. `used = total - available` (the *available* field, not *free* — Linux monitoring footgun).
|
||||
- `collect_storage(mounts)` — `shutil.disk_usage()` per mount, returns `[{mount, total, used}, ...]`.
|
||||
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
|
||||
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
|
||||
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
|
||||
- `build_payload()` — assembles a snapshot from all collectors into one dict.
|
||||
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
|
||||
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
|
||||
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer.
|
||||
|
||||
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
|
||||
|
||||
### Dependencies
|
||||
|
||||
Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shutil`, `time`, `socket`, `signal`. No `requests`, no `httpx`, no TOML libraries.
|
||||
|
||||
### Signals and lifecycle
|
||||
|
||||
- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly).
|
||||
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time → exit.
|
||||
- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`.
|
||||
|
||||
### Logging
|
||||
|
||||
To stderr only (systemd captures to journal). No file logging, no log rotation. Levels:
|
||||
- `INFO` on start and on successful flush-after-buffered-samples.
|
||||
- `WARN` on POST failure.
|
||||
- `ERROR` on config errors, token rejection.
|
||||
|
||||
### Identity
|
||||
|
||||
The agent's self-reported hostname in the payload is advisory. The real identity is the bearer token — Roundtable looks up the `Host` row via `HostAgentRegistration.token_hash`, not via hostname. Changing a host's hostname does not break its identity, and two hosts accidentally sharing a hostname can't collide.
|
||||
|
||||
### Failure behavior
|
||||
|
||||
- POST fails → push sample to ring buffer (max 20), exponential backoff `30s → 60s → 120s → 300s cap`, resets on first success.
|
||||
- On next success → flush all buffered samples in a single batched POST.
|
||||
- Ring buffer full → drop oldest, keep newest. Logged at DEBUG (expected during extended outages).
|
||||
- Malformed payload rejection (400) → drop the sample (don't re-buffer, it'll just be rejected again). Indicates agent/server version skew.
|
||||
- Token rejection (401) → log ERROR with remediation hint, continue retry loop at backoff cadence. Admin fixes via UI + conf file edit; no restart needed.
|
||||
|
||||
---
|
||||
|
||||
## Plugin design (server-side)
|
||||
|
||||
### File layout
|
||||
|
||||
Mirrors the existing `plugins/snmp/` structure.
|
||||
|
||||
```
|
||||
plugins/host_agent/
|
||||
├── __init__.py
|
||||
├── plugin.yaml # metadata + default config
|
||||
├── routes.py # blueprint: ingest, install.sh, UI pages, settings actions
|
||||
├── models.py # HostAgentRegistration SQLAlchemy model
|
||||
├── agent.py # the Python agent, served to targets at GET /agent.py
|
||||
├── migrations/
|
||||
│ ├── env.py
|
||||
│ ├── __init__.py
|
||||
│ └── versions/
|
||||
│ └── host_agent_001_initial.py # creates host_agent_registrations table
|
||||
├── templates/
|
||||
│ ├── detail.html # per-host detail page
|
||||
│ ├── widget_table.html # dashboard fleet table widget
|
||||
│ ├── widget_history.html # dashboard history chart widget
|
||||
│ ├── settings_list.html # plugin settings: list of registered hosts
|
||||
│ └── install.sh.j2 # install script template (Jinja)
|
||||
└── scheduler.py # periodic task: mark stale agents as "offline"
|
||||
```
|
||||
|
||||
### `HostAgentRegistration` model (plugin-private table)
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| `id` | UUID str | primary key |
|
||||
| `host_id` | str, FK → `hosts.id`, unique | one registration per Host |
|
||||
| `token_hash` | str | `sha256(token)`; raw token shown only once at creation |
|
||||
| `token_created_at` | datetime | for audit and rotation history |
|
||||
| `agent_version` | str, nullable | reported on each ingest; updated on change |
|
||||
| `kernel` | str, nullable | from `os.uname()` — e.g. `6.8.0-45-generic` |
|
||||
| `distro` | str, nullable | parsed from `/etc/os-release` — e.g. `Ubuntu 24.04` |
|
||||
| `arch` | str, nullable | e.g. `x86_64` |
|
||||
| `last_seen_at` | datetime, nullable | updated on every successful ingest; source of "is this agent alive" |
|
||||
| `created_at` / `updated_at` | datetime | standard |
|
||||
|
||||
### Routes (blueprint prefix `/plugins/host_agent`)
|
||||
|
||||
| Route | Auth | Purpose |
|
||||
|---|---|---|
|
||||
| `POST /ingest` | bearer token | Agent push. Validates `sha256(Authorization header)` against `token_hash` → writes `PluginMetric` rows + updates `HostAgentRegistration`. Returns 200, 400, or 401. |
|
||||
| `GET /install.sh` | query-param token | Renders the Jinja `install.sh.j2` template with URL, token, and agent version substituted. Returns `text/plain`. |
|
||||
| `GET /agent.py` | none | Serves the bundled agent source from `plugins/host_agent/agent.py` as `text/x-python`. Fetched by the install script. |
|
||||
| `GET /<host_id>/` | session | Per-host detail page. |
|
||||
| `POST /settings/add-host` | admin | Creates (or reuses) a `Host` row + creates `HostAgentRegistration` + generates token. Returns the install one-liner for UI display. |
|
||||
| `POST /settings/<host_id>/rotate-token` | admin | Generates a new token, invalidates the old one, returns new install one-liner. |
|
||||
| `POST /settings/<host_id>/delete` | admin | Removes `HostAgentRegistration`. Does not delete the backing `Host` row unless no other monitors reference it. |
|
||||
| `GET /widget` | session | HTMX partial: fleet table widget body. |
|
||||
| `GET /widget/history` | session | HTMX partial: history chart widget body (takes `host_id` query param). |
|
||||
|
||||
### Token handling
|
||||
|
||||
- Tokens are generated with `secrets.token_urlsafe(32)` at host creation.
|
||||
- Only `sha256(token)` is persisted.
|
||||
- The raw token is shown exactly once in the UI (as part of the install one-liner copy box) and is recoverable only via rotation.
|
||||
- Standard "hash-like-a-password-but-not-really" pattern for high-entropy API tokens. Plain SHA-256 is fine; no bcrypt/argon2 needed because the tokens have enough entropy that offline brute force isn't a threat.
|
||||
|
||||
### Stale-agent scheduler
|
||||
|
||||
One `ScheduledTask` running every 60 seconds that flags `HostAgentRegistration` rows with `last_seen_at > (3 × interval_seconds)` ago as stale. This is a computed flag the UI reads; no separate "online/offline" column.
|
||||
|
||||
**Note:** the existing alerts system can fire on "no metric for N seconds" which is arguably a cleaner signal. The scheduled task may end up redundant once alert rules exist. Build it anyway as a minimal safety net; it's one query plus one update.
|
||||
|
||||
### `METRIC_CATALOG` registration
|
||||
|
||||
Add to `roundtable/alerts/routes.py`:
|
||||
|
||||
```python
|
||||
"host_agent": ["cpu_pct", "mem_used_pct", "mem_available_bytes",
|
||||
"swap_used_bytes", "disk_used_pct_worst", "load_1m",
|
||||
"load_5m", "load_15m", "uptime_secs"],
|
||||
```
|
||||
|
||||
This is the one edit outside the plugin directory, matching the pattern every other plugin already follows. Not ideal from a pure-plugin-independence standpoint but unavoidable until Roundtable core grows a plugin-registered catalog API — deferred as future core work.
|
||||
|
||||
---
|
||||
|
||||
## Wire format
|
||||
|
||||
### Request
|
||||
|
||||
```http
|
||||
POST /plugins/host_agent/ingest HTTP/1.1
|
||||
Host: roundtable.home.lan
|
||||
Authorization: Bearer a1b2c3d4e5f6...
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"agent_version": "1.0.0",
|
||||
"hostname": "myhost",
|
||||
"metadata": {
|
||||
"kernel": "6.8.0-45-generic",
|
||||
"distro": "Ubuntu 24.04",
|
||||
"arch": "x86_64"
|
||||
},
|
||||
"samples": [
|
||||
{
|
||||
"ts": "2026-04-14T02:55:00Z",
|
||||
"cpu_pct": 12.4,
|
||||
"mem": {
|
||||
"total_bytes": 33554432000,
|
||||
"used_bytes": 18253611008,
|
||||
"available_bytes": 15300820992,
|
||||
"swap_used_bytes": 0
|
||||
},
|
||||
"load": { "1m": 0.42, "5m": 0.55, "15m": 0.61 },
|
||||
"uptime_secs": 482934,
|
||||
"storage": [
|
||||
{ "mount": "/", "total_bytes": 499289948160, "used_bytes": 312456789012 },
|
||||
{ "mount": "/mnt/data", "total_bytes": 4000787030016, "used_bytes": 2847193820928 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Design points
|
||||
|
||||
- **`samples` is always a list**, even for a single sample. This is how the ring buffer drains after a failure — agent POSTs `[current, ...buffered]` in a single request. One-path server code.
|
||||
- **`ts` is agent-reported, ISO-8601 UTC.** Server trusts it for `PluginMetric.recorded_at`. Host clocks drift but NTP is reliable enough for our purposes.
|
||||
- **If `|ts - now| > 5 minutes`**, server logs a WARN but still accepts. Don't reject, don't lose data over clock skew.
|
||||
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
|
||||
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
|
||||
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
|
||||
|
||||
### Server expansion into `PluginMetric` rows
|
||||
|
||||
Per sample, the server writes:
|
||||
|
||||
| `metric_name` | `resource_name` | derivation |
|
||||
|---|---|---|
|
||||
| `cpu_pct` | `<host.name>` | `sample.cpu_pct` as-is |
|
||||
| `mem_used_pct` | `<host.name>` | `100 * (total - available) / total` |
|
||||
| `mem_available_bytes` | `<host.name>` | `sample.mem.available_bytes` |
|
||||
| `swap_used_bytes` | `<host.name>` | `sample.mem.swap_used_bytes` |
|
||||
| `load_1m` | `<host.name>` | `sample.load["1m"]` |
|
||||
| `load_5m` | `<host.name>` | `sample.load["5m"]` |
|
||||
| `load_15m` | `<host.name>` | `sample.load["15m"]` |
|
||||
| `uptime_secs` | `<host.name>` | `sample.uptime_secs` |
|
||||
| `disk_used_pct` | `<host.name>:<mount>` | `100 * used / total` per mount |
|
||||
| `disk_used_bytes` | `<host.name>:<mount>` | raw per mount |
|
||||
| `disk_total_bytes` | `<host.name>:<mount>` | raw per mount |
|
||||
| `disk_used_pct_worst` | `<host.name>` | `max(disk_used_pct across mounts)` — fleet table column |
|
||||
|
||||
All values are `float`. That's ~8 host-level metrics plus 3 per mount. A typical host with 3 mounts writes ~17 rows per 30s = ~34/min = ~49k/day. Well within `PluginMetric` and the existing cleanup path.
|
||||
|
||||
Plus one update to `HostAgentRegistration`:
|
||||
|
||||
```python
|
||||
reg.last_seen_at = max(sample.ts for sample in samples)
|
||||
reg.agent_version = payload.agent_version
|
||||
reg.kernel = payload.metadata.kernel
|
||||
reg.distro = payload.metadata.distro
|
||||
reg.arch = payload.metadata.arch
|
||||
```
|
||||
|
||||
(Only written if changed, to keep `updated_at` meaningful.)
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{ "ok": true, "samples_accepted": 3 }
|
||||
```
|
||||
|
||||
Failures:
|
||||
|
||||
```json
|
||||
{ "ok": false, "error": "invalid_token" } // 401
|
||||
{ "ok": false, "error": "malformed_payload", "detail": "missing 'samples'" } // 400
|
||||
```
|
||||
|
||||
Agent treats anything non-2xx as failure → ring buffer. Agent treats 401 specially (logs, continues trying so that token rotation + conf edit recovers without restart).
|
||||
|
||||
---
|
||||
|
||||
## Install script
|
||||
|
||||
### The one-liner (what the UI shows)
|
||||
|
||||
```
|
||||
curl -sSL 'https://roundtable.home.lan/plugins/host_agent/install.sh?token=a1b2c3...' | sudo sh
|
||||
```
|
||||
|
||||
The UI also offers a "review script before running" link that opens a modal showing the two-step form:
|
||||
|
||||
```
|
||||
curl -sSL '…' -o install.sh
|
||||
less install.sh
|
||||
sudo sh install.sh
|
||||
```
|
||||
|
||||
### Rendered script structure (Jinja template `install.sh.j2`)
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
# Roundtable host agent installer
|
||||
# Generated for: {{ host_name }} ({{ host_address }})
|
||||
# Roundtable URL: {{ url }}
|
||||
set -e
|
||||
|
||||
ROUNDTABLE_URL="{{ url }}"
|
||||
AGENT_TOKEN="{{ token }}"
|
||||
AGENT_VERSION="{{ agent_version }}"
|
||||
|
||||
AGENT_USER="roundtable-agent"
|
||||
AGENT_DIR="/usr/local/lib/roundtable-agent"
|
||||
CONF_FILE="/etc/roundtable-agent.conf"
|
||||
UNIT_FILE="/etc/systemd/system/roundtable-agent.service"
|
||||
|
||||
# ── preflight ────────────────────────────────────────────────────────────────
|
||||
[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; }
|
||||
command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; }
|
||||
command -v python3 >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; }
|
||||
|
||||
# Handle --uninstall
|
||||
if [ "${1:-}" = "--uninstall" ]; then
|
||||
systemctl disable --now roundtable-agent.service 2>/dev/null || true
|
||||
rm -f "$UNIT_FILE" "$CONF_FILE"
|
||||
rm -rf "$AGENT_DIR"
|
||||
systemctl daemon-reload
|
||||
# keep the system user — removing it risks orphaned files elsewhere
|
||||
echo "uninstalled"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── create system user ───────────────────────────────────────────────────────
|
||||
if ! id "$AGENT_USER" >/dev/null 2>&1; then
|
||||
useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER"
|
||||
fi
|
||||
|
||||
# ── drop the agent file ──────────────────────────────────────────────────────
|
||||
mkdir -p "$AGENT_DIR"
|
||||
curl -sSL "${ROUNDTABLE_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py"
|
||||
chmod 0755 "$AGENT_DIR/agent.py"
|
||||
chown root:root "$AGENT_DIR/agent.py"
|
||||
|
||||
# ── write config ─────────────────────────────────────────────────────────────
|
||||
cat > "$CONF_FILE" <<EOF
|
||||
url = $ROUNDTABLE_URL
|
||||
token = $AGENT_TOKEN
|
||||
interval_seconds = 30
|
||||
EOF
|
||||
chown "root:$AGENT_USER" "$CONF_FILE"
|
||||
chmod 0640 "$CONF_FILE"
|
||||
|
||||
# ── write systemd unit ───────────────────────────────────────────────────────
|
||||
cat > "$UNIT_FILE" <<EOF
|
||||
[Unit]
|
||||
Description=Roundtable host agent
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=$AGENT_USER
|
||||
ExecStart=/usr/bin/python3 $AGENT_DIR/agent.py
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
PrivateTmp=yes
|
||||
ReadOnlyPaths=/proc /sys
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now roundtable-agent.service
|
||||
|
||||
echo
|
||||
echo "Roundtable host agent $AGENT_VERSION installed and running."
|
||||
echo "Check status: systemctl status roundtable-agent"
|
||||
echo "Logs: journalctl -u roundtable-agent -f"
|
||||
```
|
||||
|
||||
### Design points
|
||||
|
||||
- **Agent binary served by Roundtable itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Roundtable ships.
|
||||
- **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
|
||||
- **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.
|
||||
- **Fail-fast on preflight.** Missing systemd or python3 → clear error, exit. No half-installed agent.
|
||||
- **Query-string token for install fetch is acceptable.** It's over the user's reverse-proxy TLS, and the token is fresh per-install. Runtime POSTs use the `Authorization` header.
|
||||
- **No auto-upgrade.** Upgrade = re-run the installer. Self-update implies trust decisions and rollback semantics that are YAGNI for home-lab scale.
|
||||
|
||||
---
|
||||
|
||||
## UI surfaces
|
||||
|
||||
### Dashboard widgets (`roundtable/core/widgets.py`)
|
||||
|
||||
- **`host_resources`** — table widget. One row per monitored host: name, CPU %, mem %, disk % (worst mount), load 1m, "last seen Xs ago" with red/yellow/green coloring. Fleet glance.
|
||||
- **`host_resource_history`** — chart widget for one host. CPU / mem / disk over a selectable time range (1h, 6h, 24h, 7d). Same pattern as every other history widget — Chart.js.
|
||||
|
||||
### Per-host detail page
|
||||
|
||||
`GET /plugins/host_agent/<host_id>/`:
|
||||
|
||||
- Current values as big numbers (CPU %, mem %, swap %, load 1m/5m/15m).
|
||||
- Per-mount storage breakdown (the metric that doesn't fit in a dashboard row).
|
||||
- Full history charts, reusing `widget_history.html` with a wider layout.
|
||||
- Metadata block: kernel, distro, arch, uptime (human-readable), agent version.
|
||||
- Agent status: last heartbeat timestamp, "live/stale" flag.
|
||||
- "Copy install command" button (for re-install / rotation) that fetches the current one-liner.
|
||||
|
||||
### Plugin settings page
|
||||
|
||||
List of registered hosts with their enable flags, an "Add host" button that opens the registration flow (creates/reuses `Host` row → creates `HostAgentRegistration` → generates token → displays the curl install one-liner), and a "rotate token" action per host.
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
| Failure | Who handles it | Response |
|
||||
|---|---|---|
|
||||
| Agent can't reach Roundtable | Agent | Push to ring buffer, exponential backoff (30→60→120→300s cap), log WARN. Resets on success. |
|
||||
| Roundtable rejects with 401 | Agent | Log ERROR with remediation hint, continue retry loop. Admin fixes via UI + conf edit; no restart needed. |
|
||||
| Roundtable rejects with 400 | Agent | Log ERROR, drop the sample (don't re-buffer), continue. Indicates agent/server version skew. |
|
||||
| Ring buffer full | Agent | Drop oldest, keep newest. DEBUG log (expected during outages). |
|
||||
| Config file missing or malformed | Agent | Log ERROR, exit non-zero. Systemd restarts after 10s. Repeated restarts are visible in journal. |
|
||||
| `/proc/stat` read fails between samples | Agent | Skip CPU for this cycle, still POST the rest. Partial samples allowed. |
|
||||
| Mount disappears between reads | Agent | Skip that mount for this sample. If permanent, its rows simply stop appearing. |
|
||||
| Ingest token not found | Server | 401 + `{"error": "invalid_token"}`. |
|
||||
| Ingest body malformed | Server | 400 + descriptive `detail`. |
|
||||
| Ingest writes fail mid-batch | Server | 500 + rollback. Agent retries whole batch on next cycle. |
|
||||
| Agent clock skew > 5 min | Server | Accept, log WARN. |
|
||||
| Host deleted from UI | Server | `HostAgentRegistration` removed; agent's next POST → 401 → admin re-adds or uninstalls. |
|
||||
|
||||
### Edge cases explicitly accepted, not handled
|
||||
|
||||
- Two agents sharing a token → last-write-wins on `last_seen_at`, metrics get stored under the same `resource_name`. Admin should never do this.
|
||||
- Agent clock running backward (NTP correction) → `recorded_at` values non-monotonic. Charts render fine; the DB doesn't care.
|
||||
- Hostname change on target → `HostAgentRegistration.host_id` is still the token's identity; `resource_name` tracks `Host.name`. Admin edits `Host.name` if they want the display to follow.
|
||||
|
||||
### Idempotency note
|
||||
|
||||
v1 doesn't enforce unique constraints on `PluginMetric` rows. Re-submitting the same sample (e.g., agent retries after a spurious 5xx) creates duplicate rows. Acceptable for v1; if duplicates become a UX issue, add a unique constraint on `(source_module, resource_name, metric_name, recorded_at)` later.
|
||||
|
||||
---
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **Unit tests**
|
||||
- Agent collectors against fixture `/proc/stat`, `/proc/meminfo`, `/proc/loadavg`, `/proc/uptime` content.
|
||||
- Config file parser: valid, invalid, comments, whitespace edge cases.
|
||||
- Ring buffer: fill, drain, overflow drops oldest.
|
||||
- Server ingest route: happy path, bad token, malformed body, clock skew warning, partial payload (missing CPU).
|
||||
- Plugin migration smoke test: `alembic upgrade head` in a temp DB, then `downgrade base`.
|
||||
- **Integration test**
|
||||
- Spin up the plugin's ingest route in-process (Quart test client).
|
||||
- Feed a real payload generated by the agent's `build_payload()` using mocked `/proc` content.
|
||||
- Assert `PluginMetric` rows land correctly and `HostAgentRegistration` is updated.
|
||||
- **Install script verification**
|
||||
- `sh -n install.sh` (syntax).
|
||||
- `shellcheck install.sh`.
|
||||
- Snapshot test of the rendered output against a committed fixture.
|
||||
- No live systemd-in-container test — high cost, low value for a one-time install script.
|
||||
- **Agent is importable as a Python module.** `main_loop()` is guarded by `if __name__ == "__main__":`, so everything else is unit-testable.
|
||||
|
||||
---
|
||||
|
||||
## Scope boundaries — deferred to follow-ups
|
||||
|
||||
- **Ansible-action alert type** (existing follow-up, Fable task 250) — how "host CPU pinned for 5 min" can run a playbook.
|
||||
- **Plugin synergy detection** (new follow-up, to be filed) — host_agent plugin noticing the Ansible plugin and offering to auto-deploy itself.
|
||||
- **Fleet-provision mode** — bootstrap registration secret for self-registering new hosts. Additive to per-host-token foundation.
|
||||
- **Event signals** — OOM killer, read-only FS, process crashes. Separate endpoint, not an extension.
|
||||
- **Non-systemd init systems** — OpenRC, runit, sysvinit. Install-script conditionals; no architectural change.
|
||||
- **macOS / BSD / Windows agents** — separate code paths.
|
||||
- **Self-upgrade mechanism** — re-run installer for now.
|
||||
- **Disk buffering** when RAM ring fills — drop-in upgrade, no protocol change.
|
||||
- **Network and per-process metrics** — additive; agent v2.
|
||||
- **`metadata_json` column on core `Host`** — rejected in favor of strict plugin isolation.
|
||||
|
||||
---
|
||||
|
||||
## Constraints worth being aware of
|
||||
|
||||
These are not blockers; they are feature boundaries inherent to the design.
|
||||
|
||||
1. **Stdlib-only Python is a ceiling.** Anything requiring a C library (NVMe SMART data, GPU stats, eBPF syscall tracing, `psutil`-class per-process detail) will either need shelling out to system tools or abandoning the one-file install. v1 metrics are all comfortably stdlib-reachable.
|
||||
2. **Interval push is not event push.** OOM killer, FS-read-only, process crash are events that lose meaning at 30s sampling. A future event-signal channel would be a *sibling* endpoint, not an extension of this design.
|
||||
3. **`PluginMetric` only stores floats.** String states and JSON blobs don't fit. All v1 metrics are float-able; discrete state metrics would need the same numeric-enum encoding the SNMP plugin already uses.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order (rough sketch — real plan comes from writing-plans)
|
||||
|
||||
1. Plugin scaffolding: `plugin.yaml`, `__init__.py`, empty routes blueprint, migration for `host_agent_registrations`.
|
||||
2. Agent v1 as a standalone script: all collectors + config parser + `build_payload()` working locally. Testable without any server at all.
|
||||
3. Server `/ingest` route: accepts payload, writes `PluginMetric`, updates `HostAgentRegistration`. Token auth.
|
||||
4. Agent ring buffer + backoff + retry logic on top of the working collectors.
|
||||
5. Install script template + `/install.sh` and `/agent.py` routes.
|
||||
6. Settings page: list, add-host flow, rotate-token, delete.
|
||||
7. Dashboard widgets: table + history chart. Register in `core/widgets.py`.
|
||||
8. Per-host detail page.
|
||||
9. `METRIC_CATALOG` entry in `roundtable/alerts/routes.py`.
|
||||
10. Scheduler: stale-agent marker.
|
||||
11. Tests at every stage; final integration test to tie it together.
|
||||
|
||||
The writing-plans skill will turn this into a proper stepwise plan with file-level specifics.
|
||||
Reference in New Issue
Block a user