feat(ansible): host provisioning via steward managed SSH identity

Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).

- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
  (default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
  shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
  per-run user/password — never argv, never DB, never logged; drops the
  managed key when a bootstrap password is given; --user floor from the
  global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
  persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
  + /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
  user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

steward/ansible/bundled/host_agent/provision.yml

@@ -0,0 +1,143 @@
+---
+# First-contact provisioning: bring a fresh host under Steward management, then
+# install the host agent — all in one idempotent run.
+#
+# Run this ONCE per host with bootstrap credentials (an existing login + sudo,
+# typically password auth via the "Provision host" button). It:
+#   1. creates a dedicated `steward` login account,
+#   2. installs Steward's managed public key into its authorized_keys,
+#   3. grants it passwordless sudo,
+#   4. installs/updates the host agent.
+# After this, every future run (agent updates, maintenance) connects as
+# `steward` with the managed key — no operator credentials needed.
+#
+# Extra-vars (injected by the Provision card):
+#   steward_url, steward_token   — agent → Steward auth (per-host)
+#   steward_pubkey               — managed public key to authorize
+#   steward_user (default steward), agent_interval (default 30)
+- name: Provision host for Steward + install agent
+  hosts: all
+  become: true
+  gather_facts: false
+  vars:
+    steward_url: ""
+    steward_token: ""
+    steward_pubkey: ""
+    steward_user: steward
+    agent_interval: 30
+  tasks:
+    - name: Require provisioning vars
+      ansible.builtin.assert:
+        that:
+          - steward_url | length > 0
+          - steward_token | length > 0
+          - steward_pubkey | length > 0
+        fail_msg: "Pass steward_url, steward_token and steward_pubkey as extra-vars."
+
+    # ── Managed login account ────────────────────────────────────────────────
+    - name: Create the steward management account
+      ansible.builtin.user:
+        name: "{{ steward_user }}"
+        shell: /bin/sh
+        create_home: true
+        state: present
+
+    - name: Ensure ~/.ssh exists
+      ansible.builtin.file:
+        path: "/home/{{ steward_user }}/.ssh"
+        state: directory
+        owner: "{{ steward_user }}"
+        group: "{{ steward_user }}"
+        mode: "0700"
+
+    - name: Authorize the managed public key
+      ansible.builtin.lineinfile:
+        path: "/home/{{ steward_user }}/.ssh/authorized_keys"
+        line: "{{ steward_pubkey }}"
+        create: true
+        owner: "{{ steward_user }}"
+        group: "{{ steward_user }}"
+        mode: "0600"
+
+    - name: Grant passwordless sudo to the steward account
+      ansible.builtin.copy:
+        dest: "/etc/sudoers.d/steward"
+        content: "{{ steward_user }} ALL=(ALL) NOPASSWD:ALL\n"
+        owner: root
+        group: root
+        mode: "0440"
+        validate: "visudo -cf %s"
+
+    # ── Host agent (same as install.yml) ─────────────────────────────────────
+    - name: Create steward-agent system user
+      ansible.builtin.user:
+        name: steward-agent
+        system: true
+        shell: /usr/sbin/nologin
+        create_home: false
+
+    - name: Create agent directory
+      ansible.builtin.file:
+        path: /usr/local/lib/steward-agent
+        state: directory
+        mode: "0755"
+
+    - name: Download agent.py
+      ansible.builtin.get_url:
+        url: "{{ steward_url }}/plugins/host_agent/agent.py"
+        dest: /usr/local/lib/steward-agent/agent.py
+        mode: "0755"
+        force: true
+      notify: restart steward-agent
+
+    - name: Write agent config
+      ansible.builtin.copy:
+        dest: /etc/steward-agent.conf
+        owner: root
+        group: steward-agent
+        mode: "0640"
+        content: |
+          url = {{ steward_url }}
+          token = {{ steward_token }}
+          interval_seconds = {{ agent_interval }}
+      notify: restart steward-agent
+
+    - name: Install systemd unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/steward-agent.service
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Steward host agent
+          After=network-online.target
+          Wants=network-online.target
+
+          [Service]
+          Type=simple
+          User=steward-agent
+          Environment=STEWARD_AGENT_CONFIG=/etc/steward-agent.conf
+          ExecStart=/usr/bin/env python3 /usr/local/lib/steward-agent/agent.py
+          Restart=on-failure
+          RestartSec=5
+          NoNewPrivileges=true
+          ProtectSystem=strict
+          ProtectHome=true
+          PrivateTmp=true
+
+          [Install]
+          WantedBy=multi-user.target
+      notify: restart steward-agent
+
+    - name: Enable and start the agent
+      ansible.builtin.systemd:
+        name: steward-agent
+        enabled: true
+        state: started
+        daemon_reload: true
+
+  handlers:
+    - name: restart steward-agent
+      ansible.builtin.systemd:
+        name: steward-agent
+        state: restarted
+        daemon_reload: true

steward/ansible/executor.py

@@ -155,6 +155,37 @@ def build_credentials(creds: dict | None, tmpdir: str) -> tuple[list[str], list[
     return args, files
 
 
+def build_bootstrap(connection: dict | None, tmpdir: str) -> tuple[list[str], list[tuple[str, str]]]:
+    """Per-run connection override for first-contact provisioning.
+
+    Pure: returns (extra argv, files to write). The bootstrap user/password are
+    written to a temp vars file (``-e @file``) — never argv, never the DB — so
+    they don't leak via the process list or persist on the AnsibleRun row.
+    A bare connection password doubles as the become password unless one is
+    given explicitly. connection keys: user, password, become_password.
+    """
+    args: list[str] = []
+    files: list[tuple[str, str]] = []
+    connection = connection or {}
+    user = (connection.get("user") or "").strip()
+    password = connection.get("password") or ""
+    if not user and not password:
+        return args, files
+
+    lines: dict[str, str] = {}
+    if user:
+        lines["ansible_user"] = user
+    if password:
+        lines["ansible_password"] = password
+        lines["ansible_become_password"] = connection.get("become_password") or password
+    # JSON values are valid YAML and keep arbitrary characters safe.
+    content = "".join(f"{k}: {json.dumps(v)}\n" for k, v in lines.items())
+    path = os.path.join(tmpdir, "bootstrap.yml")
+    files.append((path, content))
+    args += ["-e", "@" + path]
+    return args, files
+
+
 def ansible_env(creds: dict | None, base_env) -> dict:
     """Return a copy of base_env with ANSIBLE_HOST_KEY_CHECKING set from creds."""
     env = dict(base_env)
@@ -250,12 +281,17 @@ async def start_run(
     source_path: str,
     params: dict | None = None,
     inventory_content: str | None = None,
+    connection: dict | None = None,
 ) -> None:
     """Execute ansible-playbook as a subprocess and update the DB run row.
 
     Respects a global concurrency cap (app._ansible_semaphore): if no slot is
     free the run shows as 'queued' until one frees. Supports cancellation, a
     persistent full-log artifact, and a parsed per-host result summary.
+
+    connection is an optional per-run SSH override (user/password) for
+    first-contact provisioning. It is applied to the subprocess only — it is
+    never persisted on the AnsibleRun row, never placed on argv, and not logged.
     """
     from steward.models.ansible import AnsibleRunStatus
 
@@ -328,14 +364,31 @@ async def start_run(
                 inv_target = inventory_path
             cmd = build_ansible_command(playbook_path, inv_target, params)
 
-            cred_args, cred_files = build_credentials(creds, tmpdir)
-            for cred_path, content in cred_files:
+            # First-contact provisioning supplies a password and a bootstrap
+            # user; the managed key isn't on the host yet, so don't offer it
+            # (avoids a failed key attempt before the password fallback).
+            conn = connection or {}
+            effective_creds = dict(creds)
+            if conn.get("password"):
+                effective_creds.pop("ssh_private_key", None)
+
+            cred_args, cred_files = build_credentials(effective_creds, tmpdir)
+            boot_args, boot_files = build_bootstrap(conn, tmpdir)
+            for cred_path, content in cred_files + boot_files:
                 with open(cred_path, "w", encoding="utf-8") as cf:
                     cf.write(content)
                 os.chmod(cred_path, 0o600)
 
+            # Steady-state floor: connect as the managed account unless a target
+            # var or the bootstrap override (extra-vars, higher precedence) wins.
+            user_args: list[str] = []
+            if not conn.get("user"):
+                ssh_user = (creds.get("ssh_user") or "").strip()
+                if ssh_user:
+                    user_args += ["--user", ssh_user]
+
             proc = await asyncio.create_subprocess_exec(
-                *cmd, *cred_args,
+                *cmd, *cred_args, *boot_args, *user_args,
                 stdout=asyncio.subprocess.PIPE,
                 stderr=asyncio.subprocess.STDOUT,
                 cwd=cwd,

steward/ansible/runner.py

@@ -25,6 +25,7 @@ async def trigger_run(
     params: dict | None = None,
     triggered_by: str | None = None,
     inventory_content: str | None = None,
+    connection: dict | None = None,
 ):
     """Resolve inventory for the scope, persist an AnsibleRun, and launch it.
 
@@ -32,6 +33,9 @@ async def trigger_run(
     If inventory_content is provided, it is used verbatim and scope resolution
     is skipped (the caller built a bespoke inventory — e.g. host_agent deploy
     injecting per-host tokens); inventory_scope is still recorded for display.
+    connection is an optional per-run SSH override (user/password) for
+    first-contact provisioning — passed to the executor but deliberately NOT
+    stored on the AnsibleRun row (params), so the password never lands in the DB.
     Returns (run, source, error): on success error is None; on failure run is
     None and error is a short human-readable reason.
     """
@@ -73,6 +77,7 @@ async def trigger_run(
         executor.start_run(
             app, run_id, playbook_path, inventory_path or "",
             source["path"], params or None, inventory_content,
+            connection=connection,
         )
     )
     task.add_done_callback(

steward/core/crypto.py

@@ -19,6 +19,8 @@ import os
 from pathlib import Path
 
 from cryptography.fernet import Fernet, InvalidToken
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
 
 logger = logging.getLogger(__name__)
 
@@ -91,3 +93,26 @@ def decrypt_secret(value: str) -> str:
     except InvalidToken:
         logger.error("Could not decrypt a stored secret (wrong/rotated key?)")
         return value
+
+
+def generate_ssh_keypair(comment: str = "steward-managed") -> tuple[str, str]:
+    """Mint a fresh ed25519 keypair for Steward's managed Ansible identity.
+
+    Returns (private_openssh_pem, public_openssh_line). ed25519 is small, fast,
+    and universally supported by modern OpenSSH. The private key is unencrypted
+    OpenSSH PEM (Steward stores it encrypted-at-rest itself); the public key is
+    the single-line ``ssh-ed25519 AAAA… comment`` form for authorized_keys.
+    """
+    key = Ed25519PrivateKey.generate()
+    private_pem = key.private_bytes(
+        encoding=serialization.Encoding.PEM,
+        format=serialization.PrivateFormat.OpenSSH,
+        encryption_algorithm=serialization.NoEncryption(),
+    ).decode("ascii")
+    public_line = key.public_key().public_bytes(
+        encoding=serialization.Encoding.OpenSSH,
+        format=serialization.PublicFormat.OpenSSH,
+    ).decode("ascii")
+    if comment:
+        public_line = f"{public_line} {comment}"
+    return private_pem, public_line

steward/core/settings.py

@@ -52,10 +52,17 @@ DEFAULTS: dict[str, Any] = {
     "webhook.template": _DEFAULT_WEBHOOK_TEMPLATE,
     "ansible.sources": [],
     # Ansible credentials — global, used by every run (manual + alert-triggered).
-    # Plaintext at rest, masked in the UI (encryption-at-rest tracked separately).
+    # ssh_private_key/become/vault are encrypted at rest (see SECRET_KEYS).
     "ansible.ssh_private_key": "",
     "ansible.become_password": "",
     "ansible.vault_password": "",
+    # Public half of the managed keypair — non-secret, displayed so it can be
+    # sprayed onto hosts (provisioning installs it into ~steward/.ssh).
+    "ansible.ssh_public_key": "",
+    # Default SSH login for steady-state runs — the dedicated account
+    # provisioning creates. Acts as a floor (--user); a target's ansible_user
+    # or a per-run bootstrap override still wins.
+    "ansible.ssh_user": "steward",
     "ansible.host_key_checking": False,
     # Max simultaneous playbook runs; extra runs queue. Applied at app start.
     "ansible.max_concurrent_runs": 3,
@@ -198,6 +205,8 @@ def to_ansible_cfg(settings: dict[str, Any]) -> dict:
         "ssh_private_key": settings.get("ansible.ssh_private_key", ""),
         "become_password": settings.get("ansible.become_password", ""),
         "vault_password": settings.get("ansible.vault_password", ""),
+        "ssh_public_key": settings.get("ansible.ssh_public_key", ""),
+        "ssh_user": settings.get("ansible.ssh_user", "steward"),
         "host_key_checking": settings.get("ansible.host_key_checking", False),
         "max_concurrent_runs": settings.get("ansible.max_concurrent_runs", 3),
     }

steward/settings/routes.py

@@ -178,11 +178,36 @@ async def ansible():
         ssh_key_set=bool(settings.get("ansible.ssh_private_key")),
         become_set=bool(settings.get("ansible.become_password")),
         vault_set=bool(settings.get("ansible.vault_password")),
+        ssh_public_key=settings.get("ansible.ssh_public_key", ""),
+        ssh_user=settings.get("ansible.ssh_user", "steward"),
         host_key_checking=settings.get("ansible.host_key_checking", False),
         max_concurrent_runs=settings.get("ansible.max_concurrent_runs", 3),
     )
 
 
+@settings_bp.post("/ansible/generate-key")
+@require_role(UserRole.admin)
+async def ansible_generate_key():
+    """Mint a fresh managed ed25519 keypair (private encrypted, public shown).
+
+    This is the reusable Steward identity: provisioning installs the public half
+    on every host, and steady-state runs authenticate with the private half.
+    Regenerating invalidates access to already-provisioned hosts until they are
+    re-provisioned, so the UI confirms before calling this.
+    """
+    from steward.core.crypto import generate_ssh_keypair
+
+    private_pem, public_line = generate_ssh_keypair()
+    async with current_app.db_sessionmaker() as db:
+        async with db.begin():
+            await set_setting(db, "ansible.ssh_private_key", private_pem)
+            await set_setting(db, "ansible.ssh_public_key", public_line)
+    await _reload_app_config()
+    await log_audit(current_app, session.get("user_id"), session.get("username", ""),
+                    "settings.saved", detail={"section": "ansible.managed_key"})
+    return redirect(url_for("settings.ansible"))
+
+
 @settings_bp.post("/ansible/credentials")
 @require_role(UserRole.admin)
 async def ansible_save_credentials():
@@ -203,6 +228,9 @@ async def ansible_save_credentials():
                 val = raw.strip("\n") if field == "ssh_private_key" else raw.strip()
                 if val:
                     await set_setting(db, key, val)
+            ssh_user = (form.get("ssh_user", "") or "").strip()
+            if ssh_user:
+                await set_setting(db, "ansible.ssh_user", ssh_user)
             await set_setting(db, "ansible.host_key_checking", "host_key_checking" in form)
             try:
                 mcr = max(1, min(50, int(form.get("max_concurrent_runs", 3))))

steward/templates/settings/ansible.html

@@ -21,6 +21,35 @@
   {# ── Credentials ──────────────────────────────────────────────────────────── #}
   {% set cfg_badge = '<span style="color:var(--green);font-size:0.72rem;">(configured)</span>' %}
   {% set not_set_badge = '<span style="color:var(--text-muted);font-size:0.72rem;">(not set)</span>' %}
+
+  {# ── Managed identity (provisioning) ───────────────────────────────────────── #}
+  <div style="margin-top:2.25rem;">
+    <div class="section-title" style="margin-bottom:0.75rem;">Managed SSH identity</div>
+    <p style="color:var(--text-muted);font-size:0.84rem;margin-bottom:1rem;">
+      Steward's reusable Ansible identity. Provisioning installs the <strong>public</strong>
+      key into each host's <code>{{ ssh_user }}</code> account; steady-state runs authenticate
+      with the private half (stored encrypted). Generate it once, then provision hosts from
+      <a href="/plugins/host_agent/settings/">Host Agents</a>.
+    </p>
+    {% if ssh_public_key %}
+    <div class="form-group">
+      <label>Public key <span style="color:var(--green);font-size:0.72rem;">(active)</span></label>
+      <textarea readonly rows="3" onclick="this.select();"
+                style="width:100%;font-family:ui-monospace,monospace;font-size:0.78rem;color:var(--text-muted);">{{ ssh_public_key }}</textarea>
+      <span style="color:var(--text-muted);font-size:0.78rem;">Click to select — add to a host's <code>authorized_keys</code> to authorize Steward manually.</span>
+    </div>
+    <form method="post" action="/settings/ansible/generate-key"
+          onsubmit="return confirm('Regenerate the managed key? Already-provisioned hosts will reject Steward until you re-provision them.');">
+      <button type="submit" class="btn btn-ghost btn-sm">Regenerate key</button>
+    </form>
+    {% else %}
+    <form method="post" action="/settings/ansible/generate-key">
+      <button type="submit" class="btn">Generate managed key</button>
+      <span style="color:var(--text-muted);font-size:0.78rem;margin-left:0.5rem;">ed25519 — created server-side, nothing to paste.</span>
+    </form>
+    {% endif %}
+  </div>
+
   <div style="margin-top:2.25rem;">
     <div class="section-title" style="margin-bottom:0.75rem;">Credentials</div>
     <p style="color:var(--text-muted);font-size:0.84rem;margin-bottom:1rem;">
@@ -56,6 +85,12 @@
           <input type="checkbox" name="clear_vault_password"> Clear</label>
         {% endif %}
       </div>
+      <div class="form-group">
+        <label>SSH user
+          <span style="color:var(--text-muted);font-size:0.78rem;font-weight:normal;">(default login for runs; a target's ansible_user overrides)</span>
+        </label>
+        <input type="text" name="ssh_user" value="{{ ssh_user }}" placeholder="steward" style="width:12rem;">
+      </div>
       <div class="form-group">
         <label style="display:flex;align-items:center;gap:0.5rem;font-weight:normal;">
           <input type="checkbox" name="host_key_checking" {% if host_key_checking %}checked{% endif %}>