feat(host_agent): a root helper the agent can start, for updates, reboots and drains only (#5766)

#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.

The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.

The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.

Steward handing out jobs is #5767; the cadence is #5768.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plugins/host-agent-design.md

@@ -58,9 +58,13 @@ Give Steward a fleet-glance view of remote Linux hosts — current resource usag
 ### File layout on the target host
 
 ```
-/usr/local/lib/steward-agent/agent.py       # the script, target ~300 lines
+/usr/local/lib/steward-agent/agent.py       # the agent (runs as steward-agent)
+/usr/local/lib/steward-agent/maint.py       # the maintenance helper (runs as root, #5766)
 /etc/steward-agent.conf                      # key=value config, 0640 root:steward-agent
-/etc/systemd/system/steward-agent.service    # unit file
+/etc/systemd/system/steward-agent.service    # the agent's unit
+/etc/systemd/system/steward-maint.{path,service}  # starts the helper on a request
+/var/lib/steward-agent/requests/             # the one place the agent writes (0700 steward-agent)
+/var/lib/steward-maint/results/              # the helper's outcomes (root, readable)
                                                 # dedicated system user: steward-agent
 ```
 
@@ -128,6 +132,50 @@ One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer
 
 Once an hour, and when the agent starts, a background thread asks the package manager what it would upgrade, so a slow apt never holds up a sample. On apt hosts it uses `/usr/lib/update-notifier/apt-check` (pending;security), or `apt list --upgradable` when that's missing. On dnf hosts it uses `dnf -C check-update`, `dnf -C updateinfo list --security`, and `needs-restarting -r` for whether to reboot. None of this needs root or the network: the commands read lists the system's own timers already fetched. Each command has 120 s. The last answer rides every sample as `updates`. Every sample also reads Debian's `/var/run/reboot-required` (and `.pkgs`) as `reboot`. Steward records `updates_pending`, `updates_security` and `reboot_required` (1/0) for rules, and keeps the newest on the registration. The host page's Agent card leads with "Reboot required since …", then "N updates waiting (M security)" or "Up to date", with when the check ran. `state["updates_due"] = True` asks for a check on the next sample; a maintenance job uses it once it has updated the host.
 
+
+### A root helper for updates and reboots (1.17.0, #5766)
+
+The agent can't update or reboot its host: it runs as `steward-agent` under
+`NoNewPrivileges` and `ProtectSystem=strict`, so no sudo is possible, and that
+is how it should stay. A separate helper, `maint.py`, runs as root, started by
+systemd when the agent leaves it a request:
+
+- The agent writes `<id>.json` = `{"id", "action", ["node"]}` into
+  `/var/lib/steward-agent/requests`, the one path its unit lets it write
+  (`ReadWritePaths=-…`; the `-` keeps an old host without the directory
+  starting). It writes a dot-file first and renames it, so the helper never
+  sees half a request.
+- `steward-maint.path` (`PathExistsGlob=…/requests/*.json`) starts
+  `steward-maint.service`, a oneshot running `maint.py` as root.
+- The helper opens each request with `O_NOFOLLOW`, wants a regular file under
+  4 KiB, and accepts only `update`, `reboot`, `drain` or `undrain`, an id of
+  letters, digits and dashes, and a node id in the swarm's own 25-character
+  form. It deletes every request, valid or not, and writes the outcome to
+  `/var/lib/steward-maint/results/<id>.json` (root's; kept a week). A request
+  whose id already has a result is not run again.
+- **update**: `apt-get update` then `dist-upgrade` keeping existing config
+  files, with `NEEDRESTART_MODE=l` so needrestart only lists services rather
+  than restarting the Docker daemon under its containers; or
+  `dnf -y upgrade --refresh`. The result says how many packages changed and
+  whether a reboot is now required.
+- **reboot**: writes `rebooting`, syncs, `systemctl reboot`. Any request after
+  it waits for the next boot.
+- **drain / undrain**: `docker node update --availability drain|active`, on
+  the given node or this host's own. Only a manager can; a worker's helper says
+  so in its result.
+
+The agent reports each result's new state once, as a `maintenance` event with
+its next post that gets through (results older than a day are not news). When
+an update finishes it re-checks what's waiting straight away rather than at
+the next hourly check. Its metadata says `maint_helper: true` once the path
+unit is installed and it can write its request directory; the host's Agent
+panel shows that under Manage agent. **Update agent** installs the helper on a
+host that has an older agent.
+
+So a stolen agent token can make its host update, reboot or drain, and nothing
+else. Steward handing a host a job and following it through the reboot is
+#5767; the cadence that does it one host at a time is #5768.
+
 ### What happened to the host (1.14.0, #5756)
 
 A post may carry `events`, said once each, with the next post that gets through:
@@ -362,92 +410,17 @@ sudo sh install.sh
 
 ### Rendered script structure (Jinja template `install.sh.j2`)
 
-```sh
-#!/bin/sh
-# Steward host agent installer
-# Generated for: {{ host_name }}  ({{ host_address }})
-# Steward URL: {{ url }}
-set -e
-
-STEWARD_URL="{{ url }}"
-AGENT_TOKEN="{{ token }}"
-AGENT_VERSION="{{ agent_version }}"
-
-AGENT_USER="steward-agent"
-AGENT_DIR="/usr/local/lib/steward-agent"
-CONF_FILE="/etc/steward-agent.conf"
-UNIT_FILE="/etc/systemd/system/steward-agent.service"
-
-# ── preflight ────────────────────────────────────────────────────────────────
-[ "$(id -u)" = "0" ] || { echo "must run as root (use sudo)"; exit 1; }
-command -v systemctl >/dev/null 2>&1 || { echo "systemd not found — unsupported init system"; exit 1; }
-command -v python3   >/dev/null 2>&1 || { echo "python3 not found — install python3 first"; exit 1; }
-
-# Handle --uninstall
-if [ "${1:-}" = "--uninstall" ]; then
-    systemctl disable --now steward-agent.service 2>/dev/null || true
-    rm -f "$UNIT_FILE" "$CONF_FILE"
-    rm -rf "$AGENT_DIR"
-    systemctl daemon-reload
-    # keep the system user — removing it risks orphaned files elsewhere
-    echo "uninstalled"
-    exit 0
-fi
-
-# ── create system user ───────────────────────────────────────────────────────
-if ! id "$AGENT_USER" >/dev/null 2>&1; then
-    useradd --system --no-create-home --shell /usr/sbin/nologin "$AGENT_USER"
-fi
-
-# ── drop the agent file ──────────────────────────────────────────────────────
-mkdir -p "$AGENT_DIR"
-curl -sSL "${STEWARD_URL}/plugins/host_agent/agent.py" -o "$AGENT_DIR/agent.py"
-chmod 0755 "$AGENT_DIR/agent.py"
-chown root:root "$AGENT_DIR/agent.py"
-
-# ── write config ─────────────────────────────────────────────────────────────
-cat > "$CONF_FILE" <<EOF
-url = $STEWARD_URL
-token = $AGENT_TOKEN
-interval_seconds = 30
-EOF
-chown "root:$AGENT_USER" "$CONF_FILE"
-chmod 0640 "$CONF_FILE"
-
-# ── write systemd unit ───────────────────────────────────────────────────────
-cat > "$UNIT_FILE" <<EOF
-[Unit]
-Description=Steward host agent
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User=$AGENT_USER
-ExecStart=/usr/bin/python3 $AGENT_DIR/agent.py
-Restart=on-failure
-RestartSec=10
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-PrivateTmp=yes
-ReadOnlyPaths=/proc /sys
-
-[Install]
-WantedBy=multi-user.target
-EOF
-
-systemctl daemon-reload
-systemctl enable --now steward-agent.service
-
-echo
-echo "Steward host agent $AGENT_VERSION installed and running."
-echo "Check status:  systemctl status steward-agent"
-echo "Logs:          journalctl -u steward-agent -f"
-```
+`plugins/host_agent/templates/install.sh.j2` is the script; it is not copied
+here, because a copy drifts. It checks it runs as root with systemd and
+python3, handles `--uninstall`, creates the `steward-agent` user, fetches
+`agent.py`, writes the config, then fetches the maintenance helper and the
+three units from `/plugins/host_agent/files/` (the same files the playbooks
+install), creates the maintenance directories, and enables
+`steward-maint.path` and the agent.
 
 ### Design points
 
+- **One copy of every file a host installs.** The units and the maintenance helper live in `plugins/host_agent/files/` and are served at `/plugins/host_agent/files/<name>` (a fixed list; anything else is 404). install.sh and the install, provision and update playbooks all fetch them, so the unit can't differ between a host installed by hand and one installed by Ansible (#5766).
 - **Agent binary served by Steward itself.** `GET /plugins/host_agent/agent.py` serves the script bundled with the plugin, so version drift between install-script and agent is impossible — re-running the installer picks up whatever agent the currently-running Steward ships.
 - **Systemd hardening is cheap and correct.** `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `ReadOnlyPaths=/proc /sys`. The agent only needs read access to `/proc`, `/sys`, and mount points; denying everything else narrows blast radius.
 - **Uninstall is a first-class flag**, not a separate script. Same one-liner with `--uninstall` appended.