bvandeusen/FabledSteward
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(host_agent): a root helper the agent can start, for updates, reboots and drains only (#5766)
CI / lint (push) Successful in 2s
Details
CI / integration (push) Successful in 1m22s
Details
CI / unit (push) Successful in 1m24s
Details
CI / publish (push) Successful in 1m12s
Details

Browse Source 
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.

The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.

The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.

Steward handing out jobs is #5767; the cadence is #5768.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-14 18:34:28 -04:00
co-authored by Claude Opus 5.5
parent 210a35c956
commit 6a68bd75d9
17 changed files with 851 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/host-agent-design.md
+58 -85
View File
@@ -58,9 +58,13 @@ Give Steward a fleet-glance view of remote Linux hosts — current resource usag
### File layout on the target host
```
/usr/local/lib/steward-agent/agent.py # the script, target ~300 lines
/usr/local/lib/steward-agent/agent.py # the agent (runs as steward-agent)
/usr/local/lib/steward-agent/maint.py # the maintenance helper (runs as root, #5766)
/etc/steward-agent.conf # key=value config, 0640 root:steward-agent
/etc/systemd/system/steward-agent.service # unit file
/etc/systemd/system/steward-agent.service # the agent's unit
/etc/systemd/system/steward-maint.{path,service} # starts the helper on a request
/var/lib/steward-agent/requests/ # the one place the agent writes (0700 steward-agent)
/var/lib/steward-maint/results/ # the helper's outcomes (root, readable)
# dedicated system user: steward-agent
```
@@ -128,6 +132,50 @@ One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer
Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses `/usr/lib/update-notifier/apt-check` (pending;security), or `apt list --upgradable` when that's missing. On dnf hosts it uses `dnf -C check-update`, `dnf -C updateinfo list --security`, and `needs-restarting -r` for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as `updates`. Every sample also reads Debian's `/var/run/reboot-required` (and `.pkgs`) as `reboot`. Steward records `updates_pending`, `updates_security` and `reboot_required` (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. `state["updates_due"] = True` asks for a check on the next sample; a maintenance job uses it once it has updated the host.
### A root helper for updates and reboots (1.17.0, #5766)
The agent can't update or reboot its host: it runs as `steward-agent` under
`NoNewPrivileges` and `ProtectSystem=strict`, so no sudo is possible, and that
is how it should stay. A separate helper, `maint.py`, runs as root, started by
systemd when the agent leaves it a request:
- The agent writes `<id>.json` = `{"id", "action", ["node"]}` into
`/var/lib/steward-agent/requests`, the one path its unit lets it write
(`ReadWritePaths=-…`; the `-` keeps an old host without the directory
starting). It writes a dot-file first and renames it, so the helper never
sees half a request.
- `steward-maint.path` (`PathExistsGlob=…/requests/*.json`) starts
`steward-maint.service`, a oneshot running `maint.py` as root.
- The helper opens each request with `O_NOFOLLOW`, wants a regular file under
4 KiB, and accepts only `update`, `reboot`, `drain` or `undrain`, an id of
letters, digits and dashes, and a node id in the swarm's own 25-character
form. It deletes every request, valid or not, and writes the outcome to
`/var/lib/steward-maint/results/<id>.json` (root's; kept a week). A request
whose id already has a result is not run again.
- **update**: `apt-get update` then `dist-upgrade` keeping existing config
files, with `NEEDRESTART_MODE=l` so needrestart only lists services rather
than restarting the Docker daemon under its containers; or
`dnf -y upgrade --refresh`. The result says how many packages changed and
whether a reboot is now required.
- **reboot**: writes `rebooting`, syncs, `systemctl reboot`. Any request after
it waits for the next boot.
- **drain / undrain**: `docker node update --availability drain|active`, on
the given node or this host's own. Only a manager can; a worker's helper says
so in its result.
The agent reports each result's new state once, as a `maintenance` event with
its next post that gets through (results older than a day are not news). When
an update finishes it re-checks what's waiting straight away rather than at
the next hourly check. Its metadata says `maint_helper: true` once the path
unit is installed and it can write its request directory; the host's Agent
panel shows that under Manage agent. **Update agent** installs the helper on a
host that has an older agent.
So a stolen agent token can make its host update, reboot or drain, and nothing
else. Steward handing a host a job and following it through the reboot is
#5767; the cadence that does it one host at a time is #5768.
### What happened to the host (1.14.0, #5756)
A post may carry `events`, said once each, with the next post that gets through:
@@ -362,92 +410,17 @@ sudo sh install.sh
### Rendered script structure (Jinja template `install.sh.j2`)
```sh
#!/bin/sh
# Steward host agent installer
# Generated for: {{ host_name }} ({{ host_address }})
# Steward URL: {{ url }}
set -e
STEWARD_URL="{{ url }}"
AGENT_TOKEN="{{ token }}"
AGENT_VERSION="{{ agent_version }}"
AGENT_USER="steward-agent"
AGENT_DIR="/usr/local/lib/steward-agent"
CONF_FILE="/etc/steward-agent.conf"
UNIT_FILE="/etc/systemd/system/steward-agent.service"
# ── preflight ────────────────────────────────────────────────────────────────
[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; }
command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; }
command -v python3 >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; }
# Handle --uninstall
if [ "${1:-}" = "--uninstall" ]; then
systemctl disable --now steward-agent.service 2>/dev/null || true
rm -f "$UNIT_FILE" "$CONF_FILE"
rm -rf "$AGENT_DIR"
systemctl daemon-reload
# keep the system user — removing it risks orphaned files elsewhere
echo "uninstalled"
exit 0
fi
# ── create system user ───────────────────────────────────────────────────────
if ! id "$AGENT_USER" >/dev/null 2>&1; then
useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER"
fi
# ── drop the agent file ──────────────────────────────────────────────────────
mkdir -p "$AGENT_DIR"
curl -sSL "${STEWARD_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py"
chmod 0755 "$AGENT_DIR/agent.py"
chown root:root "$AGENT_DIR/agent.py"
# ── write config ─────────────────────────────────────────────────────────────
cat > "$CONF_FILE" <<EOF
url = $STEWARD_URL
token = $AGENT_TOKEN
interval_seconds = 30
EOF
chown "root:$AGENT_USER" "$CONF_FILE"
chmod 0640 "$CONF_FILE"
# ── write systemd unit ───────────────────────────────────────────────────────
cat > "$UNIT_FILE" <<EOF
[Unit]
Description=Steward host agent
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=$AGENT_USER
ExecStart=/usr/bin/python3 $AGENT_DIR/agent.py
Restart=on-failure
RestartSec=10
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/proc /sys
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now steward-agent.service
echo
echo "Steward host agent $AGENT_VERSION installed and running."
echo "Check status: systemctl status steward-agent"
echo "Logs: journalctl -u steward-agent -f"
```
`plugins/host_agent/templates/install.sh.j2` is the script; it is not copied
here, because a copy drifts. It checks it runs as root with systemd and
python3, handles `--uninstall`, creates the `steward-agent` user, fetches
`agent.py`, writes the config, then fetches the maintenance helper and the
three units from `/plugins/host_agent/files/` (the same files the playbooks
install), creates the maintenance directories, and enables
`steward-maint.path` and the agent.
### Design points
- **One copy of every file a host installs.** The units and the maintenance helper live in `plugins/host_agent/files/` and are served at `/plugins/host_agent/files/<name>` (a fixed list; anything else is 404). install.sh and the install, provision and update playbooks all fetch them, so the unit can't differ between a host installed by hand and one installed by Ansible (#5766).
- **Agent binary served by Steward itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
- **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
- **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.