Files

T

bvandeusen eb319d715e docs: Ansible inventory infrastructure implementation plan

2026-06-05 18:19:17 -04:00

73 KiB

Raw Blame History

Ansible Inventory Infrastructure Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add AnsibleTarget + AnsibleGroup models with M2M membership, DB-generated --list inventory, per-source PAT auth via GIT_ASKPASS, and a full CRUD UI so Ansible runs are driven by Steward's own inventory records rather than git-repo inventory files.

Architecture: Three new DB tables (ansible_targets, ansible_groups, ansible_target_groups) plus one new column (ansible_runs.inventory_scope) added via Alembic migrations. A pure generate_inventory() function in inventory_gen.py converts ORM objects to the Ansible --list JSON format. The existing browse.html run form gains a scope picker that replaces the repo-inventory select; create_run() fetches scope targets and passes the generated JSON as inventory_content to the existing executor. A new inventory_routes.py blueprint serves /ansible/inventory/ CRUD pages for groups and targets.

Tech Stack: Python 3.11+, SQLAlchemy (async/asyncpg), Alembic migrations, Quart, Jinja2 templates (no JS framework beyond existing HTMX), PyYAML (already in project via Ansible dependency).

File Map

Path	Action	Responsibility
`steward/migrations/versions/0015_ansible_inventory_targets.py`	Create	Alembic migration: `ansible_targets` table
`steward/migrations/versions/0016_ansible_inventory_groups.py`	Create	Alembic migration: `ansible_groups` + `ansible_target_groups` tables
`steward/migrations/versions/0017_ansible_run_scope.py`	Create	Alembic migration: `inventory_scope` column + make `inventory_path` nullable
`steward/models/ansible_inventory.py`	Create	`AnsibleTarget`, `AnsibleGroup`, `ansible_target_groups` association table
`steward/models/__init__.py`	Modify	Import/export new models so Alembic discovers them
`steward/models/ansible.py`	Modify	Add `inventory_scope` field + make `inventory_path` optional on `AnsibleRun`
`steward/ansible/inventory_gen.py`	Create	Pure `generate_inventory(targets)` + async `fetch_scope_targets(db, scope)`
`steward/ansible/sources.py`	Modify	Add `http_token` to returned source dict + GIT_ASKPASS in `git_pull()`
`steward/ansible/routes.py`	Modify	Update `create_run()` scope handling; update `index()` context
`steward/ansible/inventory_routes.py`	Create	Blueprint: CRUD for `/ansible/inventory/groups` + `/ansible/inventory/targets`
`steward/app.py`	Modify	Register `inventory_bp`
`steward/settings/routes.py`	Modify	Include `http_token` in `ansible_save_source()` + `ansible_add_source()`
`steward/templates/ansible/browse.html`	Modify	Replace inventory select with scope picker `<select>`
`steward/templates/ansible/run_detail.html`	Modify	Show `inventory_scope` label
`steward/templates/ansible/inventory/groups.html`	Create	Groups list page
`steward/templates/ansible/inventory/group_detail.html`	Create	Group detail + member management
`steward/templates/ansible/inventory/targets.html`	Create	Targets list page
`steward/templates/ansible/inventory/target_detail.html`	Create	Target detail + group assignment
`steward/templates/hosts/form.html`	Modify	Ansible section (edit-only) to link/show AnsibleTarget
`steward/templates/settings/_ansible_sources.html`	Modify	`http_token` password field in edit + add forms
`tests/core/test_inventory_gen.py`	Create	Unit tests for `generate_inventory()`
`tests/integration/test_ansible_inventory.py`	Create	Integration test: migration tables + fetch_scope_targets

Task 1: AnsibleTarget Model + Migration

Files:

Create: steward/migrations/versions/0015_ansible_inventory_targets.py
Create: steward/models/ansible_inventory.py
Modify: steward/models/__init__.py
Step 1.1: Write the migration

Create steward/migrations/versions/0015_ansible_inventory_targets.py:

"""Add ansible_targets table

Revision ID: 0015_ansible_inventory_targets
Revises: 0014_alert_ansible_action
Create Date: 2026-06-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa

revision: str = "0015_ansible_inventory_targets"
down_revision: Union[str, None] = "0014_alert_ansible_action"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    op.create_table(
        "ansible_targets",
        sa.Column("id", sa.String(36), primary_key=True),
        sa.Column("name", sa.String(128), nullable=False),
        sa.Column("address", sa.String(255), nullable=False),
        sa.Column("ansible_vars", sa.JSON(), nullable=False, server_default="{}"),
        sa.Column(
            "host_id",
            sa.String(36),
            sa.ForeignKey("hosts.id", ondelete="SET NULL"),
            nullable=True,
        ),
        sa.Column(
            "created_at",
            sa.DateTime(timezone=True),
            nullable=False,
            server_default=sa.text("now()"),
        ),
        sa.UniqueConstraint("name", name="uq_ansible_targets_name"),
    )


def downgrade() -> None:
    op.drop_table("ansible_targets")

Step 1.2: Create steward/models/ansible_inventory.py

from __future__ import annotations
import uuid
from datetime import datetime, timezone
from sqlalchemy import Column, DateTime, ForeignKey, JSON, String, Table
from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base

ansible_target_groups = Table(
    "ansible_target_groups",
    Base.metadata,
    Column(
        "target_id",
        String(36),
        ForeignKey("ansible_targets.id", ondelete="CASCADE"),
        primary_key=True,
    ),
    Column(
        "group_id",
        String(36),
        ForeignKey("ansible_groups.id", ondelete="CASCADE"),
        primary_key=True,
    ),
)


class AnsibleTarget(Base):
    __tablename__ = "ansible_targets"

    id: Mapped[str] = mapped_column(
        String(36), primary_key=True, default=lambda: str(uuid.uuid4())
    )
    name: Mapped[str] = mapped_column(String(128), nullable=False, unique=True)
    address: Mapped[str] = mapped_column(String(255), nullable=False)
    ansible_vars: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
    host_id: Mapped[str | None] = mapped_column(
        String(36), ForeignKey("hosts.id", ondelete="SET NULL"), nullable=True
    )
    created_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True),
        nullable=False,
        default=lambda: datetime.now(timezone.utc),
    )
    groups: Mapped[list["AnsibleGroup"]] = relationship(
        "AnsibleGroup",
        secondary=ansible_target_groups,
        back_populates="targets",
        lazy="select",
    )


class AnsibleGroup(Base):
    __tablename__ = "ansible_groups"

    id: Mapped[str] = mapped_column(
        String(36), primary_key=True, default=lambda: str(uuid.uuid4())
    )
    name: Mapped[str] = mapped_column(String(128), nullable=False, unique=True)
    ansible_vars: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
    created_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True),
        nullable=False,
        default=lambda: datetime.now(timezone.utc),
    )
    targets: Mapped[list["AnsibleTarget"]] = relationship(
        "AnsibleTarget",
        secondary=ansible_target_groups,
        back_populates="groups",
        lazy="select",
    )

Step 1.3: Register in steward/models/__init__.py

Add after the from .ansible import ... line:

from .ansible_inventory import AnsibleTarget, AnsibleGroup, ansible_target_groups

Add to __all__:

"AnsibleTarget", "AnsibleGroup", "ansible_target_groups",

Step 1.4: Run the migration to verify it applies cleanly

cd /home/bvandeusen/Nextcloud/Projects/FabledSteward
# Requires a running Postgres — if testing, use the integration test DB or a local instance
python -c "
import asyncio
from steward.app import create_app
app = asyncio.run(create_app.__wrapped__()) if hasattr(create_app, '__wrapped__') else None
print('model import OK')
"

Expected: model import OK with no ImportError.

Step 1.5: Commit

git add steward/migrations/versions/0015_ansible_inventory_targets.py \
        steward/models/ansible_inventory.py \
        steward/models/__init__.py
git commit -m "feat(ansible): AnsibleTarget model + 0015_ansible_inventory_targets migration"

Task 2: AnsibleGroup + Join Table Migration

Files:

Create: steward/migrations/versions/0016_ansible_inventory_groups.py
(model already created in Task 1)
Step 2.1: Write the migration

Create steward/migrations/versions/0016_ansible_inventory_groups.py:

"""Add ansible_groups and ansible_target_groups tables

Revision ID: 0016_ansible_inventory_groups
Revises: 0015_ansible_inventory_targets
Create Date: 2026-06-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa

revision: str = "0016_ansible_inventory_groups"
down_revision: Union[str, None] = "0015_ansible_inventory_targets"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    op.create_table(
        "ansible_groups",
        sa.Column("id", sa.String(36), primary_key=True),
        sa.Column("name", sa.String(128), nullable=False),
        sa.Column("ansible_vars", sa.JSON(), nullable=False, server_default="{}"),
        sa.Column(
            "created_at",
            sa.DateTime(timezone=True),
            nullable=False,
            server_default=sa.text("now()"),
        ),
        sa.UniqueConstraint("name", name="uq_ansible_groups_name"),
    )
    op.create_table(
        "ansible_target_groups",
        sa.Column(
            "target_id",
            sa.String(36),
            sa.ForeignKey("ansible_targets.id", ondelete="CASCADE"),
            nullable=False,
        ),
        sa.Column(
            "group_id",
            sa.String(36),
            sa.ForeignKey("ansible_groups.id", ondelete="CASCADE"),
            nullable=False,
        ),
        sa.PrimaryKeyConstraint("target_id", "group_id"),
    )


def downgrade() -> None:
    op.drop_table("ansible_target_groups")
    op.drop_table("ansible_groups")

Step 2.2: Verify the migration chain is unbroken

grep "down_revision" steward/migrations/versions/0015_ansible_inventory_targets.py
grep "down_revision" steward/migrations/versions/0016_ansible_inventory_groups.py

Expected:

down_revision: Union[str, None] = "0014_alert_ansible_action"
down_revision: Union[str, None] = "0015_ansible_inventory_targets"

Step 2.3: Commit

git add steward/migrations/versions/0016_ansible_inventory_groups.py
git commit -m "feat(ansible): AnsibleGroup + join table — 0016_ansible_inventory_groups migration"

Task 3: inventory_scope Column on AnsibleRun

Files:

Create: steward/migrations/versions/0017_ansible_run_scope.py
Modify: steward/models/ansible.py
Step 3.1: Write the migration

Create steward/migrations/versions/0017_ansible_run_scope.py:

"""Add inventory_scope to ansible_runs; make inventory_path nullable; backfill scope

Revision ID: 0017_ansible_run_scope
Revises: 0016_ansible_inventory_groups
Create Date: 2026-06-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa

revision: str = "0017_ansible_run_scope"
down_revision: Union[str, None] = "0016_ansible_inventory_groups"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
    # Add inventory_scope — default 'steward:all' so existing rows get a value.
    op.add_column(
        "ansible_runs",
        sa.Column(
            "inventory_scope",
            sa.String(255),
            nullable=True,
        ),
    )
    # Backfill: existing rows used repo inventory files — reconstruct the legacy scope string.
    op.execute(
        """
        UPDATE ansible_runs
        SET inventory_scope = 'repo:' || source_name || ':' || inventory_path
        WHERE inventory_path IS NOT NULL AND inventory_path != ''
        """
    )
    # Rows with empty inventory_path get 'steward:all'
    op.execute(
        "UPDATE ansible_runs SET inventory_scope = 'steward:all' WHERE inventory_scope IS NULL"
    )
    # Now make it non-nullable.
    op.alter_column("ansible_runs", "inventory_scope", nullable=False)
    # Make inventory_path nullable — new steward:* runs won't have a file path.
    op.alter_column("ansible_runs", "inventory_path", nullable=True)


def downgrade() -> None:
    op.drop_column("ansible_runs", "inventory_scope")
    op.alter_column("ansible_runs", "inventory_path", nullable=False)

Step 3.2: Update steward/models/ansible.py

Add inventory_scope field and make inventory_path optional. In the AnsibleRun class, change:

# Before:
inventory_path: Mapped[str] = mapped_column(String(512), nullable=False)

to:

# After:
inventory_path: Mapped[str | None] = mapped_column(String(512), nullable=True)
inventory_scope: Mapped[str] = mapped_column(
    String(255), nullable=False, default="steward:all"
)

The inventory_scope column goes after inventory_path in the class body (after line ~24, before source_name).

Step 3.3: Commit

git add steward/migrations/versions/0017_ansible_run_scope.py steward/models/ansible.py
git commit -m "feat(ansible): add inventory_scope to AnsibleRun — 0017_ansible_run_scope migration"

Task 4: Inventory Generation — Pure Function + Unit Tests

Files:

Create: steward/ansible/inventory_gen.py
Create: tests/core/test_inventory_gen.py
Step 4.1: Write failing unit tests

Create tests/core/test_inventory_gen.py:

"""Unit tests for generate_inventory().

Uses SimpleNamespace duck-typed objects so no DB or SQLAlchemy is required.
"""
from types import SimpleNamespace
import pytest
from steward.ansible.inventory_gen import generate_inventory


def _group(name, ansible_vars=None):
    g = SimpleNamespace()
    g.name = name
    g.ansible_vars = ansible_vars or {}
    return g


def _target(name, address, ansible_vars=None, groups=None):
    t = SimpleNamespace()
    t.name = name
    t.address = address
    t.ansible_vars = ansible_vars or {}
    t.groups = groups or []
    return t


def test_empty_targets():
    result = generate_inventory([])
    assert result == {"all": {"hosts": []}, "_meta": {"hostvars": {}}}


def test_single_target_no_groups():
    target = _target("webserver", "192.168.1.10", {"ansible_user": "ubuntu"})
    result = generate_inventory([target])
    assert result["all"]["hosts"] == ["webserver"]
    assert result["_meta"]["hostvars"]["webserver"]["ansible_host"] == "192.168.1.10"
    assert result["_meta"]["hostvars"]["webserver"]["ansible_user"] == "ubuntu"
    # No group keys other than 'all' and '_meta'
    assert set(result.keys()) == {"all", "_meta"}


def test_ansible_host_overrides_group_var():
    """ansible_host is always target.address, even if a group sets it to something else."""
    grp = _group("servers", {"ansible_host": "10.0.0.1"})
    target = _target("host1", "192.168.1.50", groups=[grp])
    result = generate_inventory([target])
    assert result["_meta"]["hostvars"]["host1"]["ansible_host"] == "192.168.1.50"


def test_group_vars_applied():
    grp = _group("webservers", {"http_port": 80, "ansible_user": "deploy"})
    target = _target("web1", "192.168.1.10", groups=[grp])
    result = generate_inventory([target])
    hostvars = result["_meta"]["hostvars"]["web1"]
    assert hostvars["http_port"] == 80
    assert hostvars["ansible_user"] == "deploy"
    assert result["webservers"]["hosts"] == ["web1"]
    assert result["webservers"]["vars"]["http_port"] == 80


def test_host_vars_override_group_vars():
    """Per-target ansible_vars beat group vars for the same key."""
    grp = _group("g1", {"ansible_user": "group_user"})
    target = _target("t1", "1.2.3.4", {"ansible_user": "host_user"}, groups=[grp])
    result = generate_inventory([target])
    assert result["_meta"]["hostvars"]["t1"]["ansible_user"] == "host_user"


def test_group_var_precedence_is_alphabetical():
    """When multiple groups set the same key, alphabetically-last group wins (Ansible default)."""
    g_a = _group("aaa", {"timeout": 10})
    g_b = _group("bbb", {"timeout": 20})
    target = _target("t1", "1.1.1.1", groups=[g_b, g_a])  # deliberately reversed order
    result = generate_inventory([target])
    # Groups sorted alphabetically: aaa then bbb → bbb wins
    assert result["_meta"]["hostvars"]["t1"]["timeout"] == 20


def test_multiple_targets_in_group():
    grp = _group("db", {"db_port": 5432})
    t1 = _target("db1", "10.0.0.1", groups=[grp])
    t2 = _target("db2", "10.0.0.2", groups=[grp])
    result = generate_inventory([t1, t2])
    assert set(result["db"]["hosts"]) == {"db1", "db2"}
    assert result["all"]["hosts"] == ["db1", "db2"]


def test_target_in_multiple_groups():
    g1 = _group("web", {"role": "web"})
    g2 = _group("prod", {"env": "production"})
    target = _target("web-prod-01", "192.168.1.1", groups=[g1, g2])
    result = generate_inventory([target])
    assert "web-prod-01" in result["web"]["hosts"]
    assert "web-prod-01" in result["prod"]["hosts"]
    assert result["_meta"]["hostvars"]["web-prod-01"]["role"] == "web"
    assert result["_meta"]["hostvars"]["web-prod-01"]["env"] == "production"

Step 4.2: Run tests to confirm they all fail with ImportError

cd /home/bvandeusen/Nextcloud/Projects/FabledSteward
pytest tests/core/test_inventory_gen.py -v 2>&1 | head -20

Expected: ModuleNotFoundError: No module named 'steward.ansible.inventory_gen'

Step 4.3: Create steward/ansible/inventory_gen.py

"""Ansible --list inventory generation from DB records."""
from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from steward.models.ansible_inventory import AnsibleTarget


def generate_inventory(targets: list) -> dict:
    """Build an Ansible --list JSON inventory from a list of AnsibleTarget objects.

    Accepts any objects with .name, .address, .ansible_vars (dict), .groups (list
    of objects with .name and .ansible_vars). Uses duck typing so unit tests can
    pass SimpleNamespace objects.

    Var precedence (lowest → highest, matching Ansible default):
      group vars (sorted alphabetically by group name) → target ansible_vars → ansible_host
    """
    inv: dict = {"all": {"hosts": []}, "_meta": {"hostvars": {}}}

    for target in targets:
        inv["all"]["hosts"].append(target.name)

        # Merge vars: group vars alphabetically, then host vars, then force ansible_host.
        merged: dict = {}
        for group in sorted(target.groups, key=lambda g: g.name):
            merged.update(group.ansible_vars or {})
        merged.update(target.ansible_vars or {})
        merged["ansible_host"] = target.address

        inv["_meta"]["hostvars"][target.name] = merged

        for group in target.groups:
            if group.name not in inv:
                inv[group.name] = {
                    "hosts": [],
                    "vars": dict(group.ansible_vars or {}),
                }
            inv[group.name]["hosts"].append(target.name)

    return inv


async def fetch_scope_targets(db, scope: str) -> list:
    """Query AnsibleTarget objects for a given inventory scope string.

    Scopes:
      steward:all              — every target
      steward:group:<uuid>     — targets in that group
      steward:target:<uuid>    — single target by ID

    Returns a list with groups pre-loaded (selectinload). Returns [] for
    repo:* scopes (caller handles those via legacy path).
    """
    from sqlalchemy import select
    from sqlalchemy.orm import selectinload
    from steward.models.ansible_inventory import AnsibleTarget, AnsibleGroup

    if scope == "steward:all":
        stmt = (
            select(AnsibleTarget)
            .options(selectinload(AnsibleTarget.groups))
            .order_by(AnsibleTarget.name)
        )
    elif scope.startswith("steward:group:"):
        group_id = scope[len("steward:group:"):]
        stmt = (
            select(AnsibleTarget)
            .join(AnsibleTarget.groups)
            .where(AnsibleGroup.id == group_id)
            .options(selectinload(AnsibleTarget.groups))
            .order_by(AnsibleTarget.name)
        )
    elif scope.startswith("steward:target:"):
        target_id = scope[len("steward:target:"):]
        stmt = (
            select(AnsibleTarget)
            .where(AnsibleTarget.id == target_id)
            .options(selectinload(AnsibleTarget.groups))
        )
    else:
        return []

    result = await db.execute(stmt)
    return list(result.scalars().all())

Step 4.4: Run tests to confirm they pass

pytest tests/core/test_inventory_gen.py -v

Expected: 8 PASSED.

Step 4.5: Commit

git add steward/ansible/inventory_gen.py tests/core/test_inventory_gen.py
git commit -m "feat(ansible): generate_inventory() + fetch_scope_targets() + unit tests"

Task 5: GIT_ASKPASS — Per-Source HTTP Token Auth

Files:

Modify: steward/ansible/sources.py
Modify: steward/settings/routes.py
Modify: steward/templates/settings/_ansible_sources.html
Step 5.1: Write a failing unit test

Create tests/core/test_git_askpass.py:

"""Unit tests for GIT_ASKPASS injection in git_pull()."""
import asyncio
import os
import stat
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest


def test_get_sources_includes_http_token(tmp_path):
    """get_sources() must pass http_token through from config."""
    from steward.ansible.sources import get_sources

    cfg = {
        "cache_dir": str(tmp_path),
        "sources": [
            {
                "name": "repo",
                "type": "git",
                "url": "https://git.example.com/org/repo.git",
                "http_token": "mytoken123",
            }
        ],
    }
    sources = get_sources(cfg)
    assert sources[0]["http_token"] == "mytoken123"


def test_get_sources_missing_http_token_defaults_empty(tmp_path):
    """http_token defaults to empty string when not in source config."""
    from steward.ansible.sources import get_sources

    cfg = {
        "cache_dir": str(tmp_path),
        "sources": [
            {"name": "r", "type": "git", "url": "https://example.com/r.git"}
        ],
    }
    sources = get_sources(cfg)
    assert sources[0]["http_token"] == ""


@pytest.mark.asyncio
async def test_git_pull_creates_askpass_when_token_set(tmp_path):
    """git_pull() sets GIT_ASKPASS env when source has http_token."""
    from steward.ansible.sources import git_pull

    # Simulate repo already cloned (has .git dir) so we hit the pull path.
    repo_path = tmp_path / "repo"
    (repo_path / ".git").mkdir(parents=True)

    source = {
        "name": "repo",
        "path": str(repo_path),
        "url": "https://git.example.com/org/repo.git",
        "branch": "main",
        "http_token": "secret-token",
    }

    captured_env = {}

    async def fake_subprocess(*args, **kwargs):
        captured_env.update(kwargs.get("env", {}))
        proc = MagicMock()
        proc.returncode = 0
        proc.communicate = AsyncMock(return_value=(b"", b""))
        return proc

    with patch("asyncio.create_subprocess_exec", side_effect=fake_subprocess):
        await git_pull(source)

    assert "GIT_ASKPASS" in captured_env
    askpass_path = captured_env["GIT_ASKPASS"]
    assert os.path.exists(askpass_path)
    mode = stat.S_IMODE(os.stat(askpass_path).st_mode)
    assert mode & stat.S_IXUSR  # executable

    content = Path(askpass_path).read_text()
    assert "secret-token" in content
    assert "oauth2" in content


@pytest.mark.asyncio
async def test_git_pull_no_askpass_without_token(tmp_path):
    """git_pull() does NOT set GIT_ASKPASS when http_token is empty."""
    from steward.ansible.sources import git_pull

    repo_path = tmp_path / "repo"
    (repo_path / ".git").mkdir(parents=True)

    source = {
        "name": "repo",
        "path": str(repo_path),
        "url": "https://git.example.com/org/repo.git",
        "branch": "main",
        "http_token": "",
    }

    captured_env = {}

    async def fake_subprocess(*args, **kwargs):
        captured_env.update(kwargs.get("env", {}))
        proc = MagicMock()
        proc.returncode = 0
        proc.communicate = AsyncMock(return_value=(b"", b""))
        return proc

    with patch("asyncio.create_subprocess_exec", side_effect=fake_subprocess):
        await git_pull(source)

    assert "GIT_ASKPASS" not in captured_env

Step 5.2: Run to confirm failures

pytest tests/core/test_git_askpass.py -v 2>&1 | tail -10

Expected: test_get_sources_includes_http_token FAILED (KeyError: http_token), asyncio tests FAILED.

Step 5.3: Update get_sources() in steward/ansible/sources.py

In get_sources(), inside the if src_type == "git": block, add:

"http_token": src.get("http_token", ""),

The full updated return dict for git sources becomes:

result.append({
    "name": src["name"],
    "type": src_type,
    "path": path,
    "url": src.get("url"),
    "branch": src.get("branch", "main"),
    "pull_interval_seconds": int(src.get("pull_interval_seconds", 3600)),
    "http_token": src.get("http_token", ""),  # ADD THIS LINE
})

For local sources, add "http_token": "" so callers can always rely on the key:

result.append({
    "name": src["name"],
    "type": src_type,
    "path": path,
    "url": src.get("url"),
    "branch": src.get("branch", "main"),
    "pull_interval_seconds": int(src.get("pull_interval_seconds", 3600)),
    "http_token": "",  # local sources don't use auth
})

Step 5.4: Update git_pull() in steward/ansible/sources.py

Replace the existing git_pull() function with:

async def git_pull(source: dict) -> None:
    """Clone the git repo if absent; pull if already present.

    When source['http_token'] is set, injects credentials via a temporary
    GIT_ASKPASS script so they never touch .git/config or process args.
    """
    import os
    import stat
    import tempfile

    path = Path(source["path"])
    token = source.get("http_token", "")

    env = None
    askpass_path = None
    if token:
        # Write a temp askpass script that echoes the token as the git password.
        fd, askpass_path = tempfile.mkstemp(prefix="steward-askpass-", suffix=".sh")
        try:
            script = (
                "#!/bin/sh\n"
                'case "$1" in\n'
                '  Username*) echo "oauth2" ;;\n'
                f'  Password*) echo "{token}" ;;\n'
                "esac\n"
            )
            os.write(fd, script.encode())
        finally:
            os.close(fd)
        os.chmod(askpass_path, stat.S_IRWXU)
        env = {**os.environ, "GIT_ASKPASS": askpass_path}

    try:
        if not (path / ".git").exists():
            path.mkdir(parents=True, exist_ok=True)
            proc = await asyncio.create_subprocess_exec(
                "git", "clone",
                "--branch", source["branch"],
                "--single-branch",
                source["url"], str(path),
                stdout=asyncio.subprocess.PIPE,
                stderr=asyncio.subprocess.PIPE,
                env=env,
            )
            _, stderr = await proc.communicate()
            if proc.returncode != 0:
                logger.error(
                    "git clone failed for %r: %s",
                    source["name"],
                    stderr.decode(errors="replace"),
                )
        else:
            proc = await asyncio.create_subprocess_exec(
                "git", "-C", str(path), "pull",
                stdout=asyncio.subprocess.PIPE,
                stderr=asyncio.subprocess.PIPE,
                env=env,
            )
            _, stderr = await proc.communicate()
            if proc.returncode != 0:
                logger.error(
                    "git pull failed for %r: %s",
                    source["name"],
                    stderr.decode(errors="replace"),
                )
    finally:
        if askpass_path and os.path.exists(askpass_path):
            os.unlink(askpass_path)

Step 5.5: Run tests to confirm they pass

pytest tests/core/test_git_askpass.py -v

Expected: 4 PASSED.

Step 5.6: Update ansible_save_source() in steward/settings/routes.py

In the ansible_save_source() function (line ~249), inside the if src_type == "git": block, add:

token = form.get("http_token", "").strip()
if token:
    entry["http_token"] = token
elif "http_token" in sources[idx]:
    # Preserve existing token if form submitted empty (password fields don't round-trip)
    entry["http_token"] = sources[idx]["http_token"]

Also find ansible_add_source() (around line ~212) and add the same token-saving logic in the if src_type == "git": block.

Step 5.7: Add http_token field to _ansible_sources.html

Inside the ansible-edit-git-fields-{{ idx }} div, after the pull_interval_seconds form-group, add:

<div class="form-group" style="margin:0;grid-column:1/-1;">
  <label style="font-size:0.8rem;">
    HTTP Token
    <span style="color:var(--text-muted);font-weight:normal;">(optional — Gitea PAT for private repos)</span>
  </label>
  <input type="password" name="http_token"
         placeholder="{% if src.http_token %}token saved — enter new to replace{% endif %}"
         autocomplete="new-password">
</div>

Inside the ansible-add-git-fields div, after the pull_interval_seconds form-group, add:

<div class="form-group" style="margin:0;grid-column:1/-1;">
  <label style="font-size:0.8rem;">HTTP Token
    <span style="color:var(--text-muted);font-weight:normal;">(optional)</span>
  </label>
  <input type="password" name="http_token" autocomplete="new-password">
</div>

Step 5.8: Commit

git add steward/ansible/sources.py \
        steward/settings/routes.py \
        steward/templates/settings/_ansible_sources.html \
        tests/core/test_git_askpass.py
git commit -m "feat(ansible): GIT_ASKPASS per-source HTTP token auth + settings UI"

Task 6: Wire Inventory Generation into create_run()

Files:

Modify: steward/ansible/routes.py
Modify: steward/templates/ansible/browse.html
Modify: steward/templates/ansible/run_detail.html
Step 6.1: Update index() in steward/ansible/routes.py

The index() route needs to pass inventory_scope label context to the run history table (for display). No test needed — it's a display change.

Change the index() route to also fetch target/group counts for displaying in the nav:

@ansible_bp.get("/")
@require_role(UserRole.viewer)
async def index():
    from sqlalchemy import func
    from steward.models.ansible_inventory import AnsibleTarget, AnsibleGroup

    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(AnsibleRun).order_by(AnsibleRun.started_at.desc()).limit(50)
        )
        runs = result.scalars().all()
        target_count = (await db.execute(select(func.count()).select_from(AnsibleTarget))).scalar()
    return await render_template("ansible/index.html", runs=runs, target_count=target_count)

Step 6.2: Update create_run() to use inventory_scope

Replace the existing create_run() implementation. The key changes are:

Accept inventory_scope from the form instead of inventory_path
Generate inventory_content from DB for steward:* scopes
Store inventory_scope on the AnsibleRun row

Full updated function (replaces the existing one at line ~108):

@ansible_bp.post("/runs")
@require_role(UserRole.operator)
async def create_run():
    import json as _json
    from steward.ansible.inventory_gen import fetch_scope_targets, generate_inventory

    form = await request.form
    playbook_path = form.get("playbook_path", "").strip()
    source_name = form.get("source_name", "").strip()
    inventory_scope = form.get("inventory_scope", "steward:all").strip()

    if not playbook_path:
        return "playbook_path is required", 400

    all_sources = _get_sources()
    source = next((s for s in all_sources if s["name"] == source_name), None)
    if source is None:
        return "Source not found", 404

    # Resolve inventory content for DB-backed scopes.
    inventory_content: str | None = None
    inventory_path: str | None = None
    if inventory_scope.startswith("steward:"):
        async with current_app.db_sessionmaker() as db:
            targets = await fetch_scope_targets(db, inventory_scope)
        inventory_content = _json.dumps(generate_inventory(targets))
    elif inventory_scope.startswith("repo:"):
        # repo:<source_name>:<relative_path>
        parts = inventory_scope.split(":", 2)
        inventory_path = parts[2] if len(parts) == 3 else ""
    else:
        return "Invalid inventory_scope", 400

    # Optional run parameters.
    extra_vars: list[str] = []
    for line in (form.get("extra_vars", "") or "").splitlines():
        line = line.strip()
        if not line:
            continue
        if "=" not in line:
            return f"Invalid extra var (expected key=value): {line!r}", 400
        extra_vars.append(line)
    limit = (form.get("limit", "") or "").strip()
    tags = (form.get("tags", "") or "").strip()
    check = "check" in form

    params: dict = {}
    if extra_vars:
        params["extra_vars"] = extra_vars
    if limit:
        params["limit"] = limit
    if tags:
        params["tags"] = tags
    if check:
        params["check"] = True
    params_or_none = params or None

    run_id = str(uuid.uuid4())
    now = datetime.now(timezone.utc)

    run = AnsibleRun(
        id=run_id,
        playbook_path=playbook_path,
        inventory_path=inventory_path,
        inventory_scope=inventory_scope,
        source_name=source_name,
        triggered_by=session["user_id"],
        status=AnsibleRunStatus.running,
        started_at=now,
        params=params_or_none,
    )

    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            db.add(run)

    asyncio.ensure_future(
        executor.start_run(
            current_app._get_current_object(),  # type: ignore[attr-defined]
            run_id,
            playbook_path=playbook_path,
            inventory_path=inventory_path or "",
            source_path=source["path"],
            params=params_or_none,
            inventory_content=inventory_content,
        )
    )
    return await render_template("ansible/run_started.html", run_id=run_id)

Step 6.3: Update browse() route to pass scope context

The browse() route needs to pass targets and groups for the scope picker. Update it:

@ansible_bp.get("/browse")
@require_role(UserRole.viewer)
async def browse():
    from steward.models.ansible_inventory import AnsibleTarget, AnsibleGroup

    all_sources = _get_sources()
    source_data = []
    for source in all_sources:
        playbooks = src_module.discover_playbooks(source["path"])
        inventories = src_module.discover_inventories(source["path"])
        source_data.append({
            "source": source,
            "playbooks": playbooks,
            "inventories": inventories,
        })

    async with current_app.db_sessionmaker() as db:
        targets = (await db.execute(
            select(AnsibleTarget).order_by(AnsibleTarget.name)
        )).scalars().all()
        groups = (await db.execute(
            select(AnsibleGroup).order_by(AnsibleGroup.name)
        )).scalars().all()

    return await render_template(
        "ansible/browse.html",
        source_data=source_data,
        targets=targets,
        groups=groups,
    )

Step 6.4: Update steward/templates/ansible/browse.html run form

Replace the existing "Inventory" form-group block (lines 73–85) with the scope picker. The existing block is:

            <div class="form-group">
                <label>Inventory</label>
                <select name="inventory_path">
                    {% for inv in sd.inventories %}
                    <option value="{{ inv }}">{{ inv }}</option>
                    {% endfor %}
                    <option value="">— Enter manually below —</option>
                </select>
            </div>
            <div class="form-group">
                <label>Or enter inventory path manually</label>
                <input type="text" id="manual-inv-{{ sd.source.name }}" placeholder="inventories/production/hosts">
            </div>

Replace with:

            <div class="form-group">
                <label>Run Against</label>
                <select name="inventory_scope">
                    {% if targets %}
                    <optgroup label="All Targets">
                        <option value="steward:all">All targets ({{ targets | length }})</option>
                    </optgroup>
                    {% if groups %}
                    <optgroup label="Group">
                        {% for grp in groups %}
                        <option value="steward:group:{{ grp.id }}">Group: {{ grp.name }}</option>
                        {% endfor %}
                    </optgroup>
                    {% endif %}
                    <optgroup label="Single Target">
                        {% for tgt in targets %}
                        <option value="steward:target:{{ tgt.id }}">{{ tgt.name }}</option>
                        {% endfor %}
                    </optgroup>
                    {% endif %}
                    {% if sd.inventories %}
                    <optgroup label="Repo Inventory File">
                        {% for inv in sd.inventories %}
                        <option value="repo:{{ sd.source.name }}:{{ inv }}">{{ sd.source.name }}/{{ inv }}</option>
                        {% endfor %}
                    </optgroup>
                    {% endif %}
                    {% if not targets and not sd.inventories %}
                    <option value="" disabled>No targets or inventory files found</option>
                    {% endif %}
                </select>
            </div>

Also update the "Execute" button's onclick — remove the manual-inv logic since there's no manual field anymore:

                <button type="submit" class="btn">Execute</button>

(The form can submit directly now.)

Step 6.5: Update run_detail.html to show scope label

In steward/templates/ansible/run_detail.html, find where run.inventory_path is displayed and add inventory_scope display. After the playbook path line, add:

<tr>
    <th>Scope</th>
    <td style="font-size:0.9rem;font-family:ui-monospace,monospace;">{{ run.inventory_scope }}</td>
</tr>

(Check the existing table structure in run_detail.html first — look for run.playbook_path and follow the same <tr> pattern.)

Step 6.6: Commit

git add steward/ansible/routes.py \
        steward/templates/ansible/browse.html \
        steward/templates/ansible/run_detail.html
git commit -m "feat(ansible): scope picker in run form + create_run() DB inventory generation"

Task 7: Inventory Blueprint — Groups CRUD

Files:

Create: steward/ansible/inventory_routes.py
Create: steward/templates/ansible/inventory/groups.html
Create: steward/templates/ansible/inventory/group_detail.html
Modify: steward/app.py
Step 7.1: Create steward/ansible/inventory_routes.py

"""CRUD routes for Ansible inventory: groups and targets."""
from __future__ import annotations
import uuid
import yaml

from quart import Blueprint, current_app, redirect, render_template, request, url_for
from sqlalchemy import select
from sqlalchemy.orm import selectinload

from steward.auth.middleware import require_role
from steward.models.ansible_inventory import AnsibleGroup, AnsibleTarget
from steward.models.users import UserRole

inventory_bp = Blueprint("ansible_inventory", __name__, url_prefix="/ansible/inventory")


def _parse_ansible_vars(raw: str) -> dict:
    """Parse a YAML/JSON string into a dict. Raises ValueError on invalid input."""
    raw = raw.strip()
    if not raw:
        return {}
    try:
        parsed = yaml.safe_load(raw)
    except yaml.YAMLError as exc:
        raise ValueError(f"Invalid YAML: {exc}") from exc
    if parsed is None:
        return {}
    if not isinstance(parsed, dict):
        raise ValueError("ansible_vars must be a YAML mapping (key: value pairs)")
    return parsed


@inventory_bp.get("/groups")
@require_role(UserRole.viewer)
async def groups_list():
    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(AnsibleGroup)
            .options(selectinload(AnsibleGroup.targets))
            .order_by(AnsibleGroup.name)
        )
        groups = result.scalars().all()
    return await render_template("ansible/inventory/groups.html", groups=groups)


@inventory_bp.post("/groups")
@require_role(UserRole.operator)
async def groups_create():
    form = await request.form
    name = form.get("name", "").strip()
    if not name:
        return "name is required", 400
    try:
        ansible_vars = _parse_ansible_vars(form.get("ansible_vars", ""))
    except ValueError as exc:
        return str(exc), 400

    group = AnsibleGroup(id=str(uuid.uuid4()), name=name, ansible_vars=ansible_vars)
    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            db.add(group)
    return redirect(url_for("ansible_inventory.groups_list"))


@inventory_bp.get("/groups/<group_id>")
@require_role(UserRole.viewer)
async def group_detail(group_id: str):
    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(AnsibleGroup)
            .where(AnsibleGroup.id == group_id)
            .options(selectinload(AnsibleGroup.targets))
        )
        group = result.scalar_one_or_none()
        if group is None:
            return "Group not found", 404
        all_targets = (
            await db.execute(select(AnsibleTarget).order_by(AnsibleTarget.name))
        ).scalars().all()
    member_ids = {t.id for t in group.targets}
    ansible_vars_yaml = yaml.dump(group.ansible_vars) if group.ansible_vars else ""
    return await render_template(
        "ansible/inventory/group_detail.html",
        group=group,
        all_targets=all_targets,
        member_ids=member_ids,
        ansible_vars_yaml=ansible_vars_yaml,
    )


@inventory_bp.post("/groups/<group_id>")
@require_role(UserRole.operator)
async def group_update(group_id: str):
    form = await request.form
    try:
        ansible_vars = _parse_ansible_vars(form.get("ansible_vars", ""))
    except ValueError as exc:
        return str(exc), 400

    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            result = await db.execute(
                select(AnsibleGroup)
                .where(AnsibleGroup.id == group_id)
                .options(selectinload(AnsibleGroup.targets))
            )
            group = result.scalar_one_or_none()
            if group is None:
                return "Group not found", 404

            name = form.get("name", group.name).strip()
            if name:
                group.name = name
            group.ansible_vars = ansible_vars

            # Update member list — form sends target_ids[] checkboxes.
            new_ids = set(form.getlist("target_ids"))
            all_targets = (
                await db.execute(select(AnsibleTarget))
            ).scalars().all()
            target_map = {t.id: t for t in all_targets}
            group.targets = [target_map[tid] for tid in new_ids if tid in target_map]

    return redirect(url_for("ansible_inventory.group_detail", group_id=group_id))


@inventory_bp.post("/groups/<group_id>/delete")
@require_role(UserRole.operator)
async def group_delete(group_id: str):
    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            result = await db.execute(
                select(AnsibleGroup).where(AnsibleGroup.id == group_id)
            )
            group = result.scalar_one_or_none()
            if group is not None:
                await db.delete(group)
    return redirect(url_for("ansible_inventory.groups_list"))

Step 7.2: Create steward/templates/ansible/inventory/groups.html

{% extends "base.html" %}
{% block title %}Inventory Groups — Steward{% endblock %}
{% block content %}
<div style="display:flex;align-items:center;justify-content:space-between;margin-bottom:1.5rem;">
    <h1 class="page-title" style="margin-bottom:0;">Inventory Groups</h1>
    <a href="/ansible/inventory/targets" class="btn btn-ghost btn-sm">Targets</a>
</div>

<div class="card" style="margin-bottom:1.5rem;">
    <h3 class="section-title">New Group</h3>
    <form method="post" action="/ansible/inventory/groups" style="display:flex;gap:0.75rem;align-items:flex-end;">
        <div class="form-group" style="flex:1;margin:0;">
            <label>Name</label>
            <input type="text" name="name" placeholder="webservers" required>
        </div>
        <button type="submit" class="btn">Create</button>
    </form>
</div>

{% if not groups %}
<div class="card">
    <p style="color:var(--text-muted);">No groups yet. Create one above.</p>
</div>
{% else %}
<div class="card-flush">
<table class="table">
    <thead>
        <tr><th>Name</th><th>Members</th><th></th></tr>
    </thead>
    <tbody>
    {% for grp in groups %}
    <tr>
        <td><strong>{{ grp.name }}</strong></td>
        <td style="color:var(--text-muted);font-size:0.9rem;">{{ grp.targets | length }}</td>
        <td class="td-actions">
            <a href="/ansible/inventory/groups/{{ grp.id }}" class="btn btn-sm btn-ghost">Edit</a>
            <form method="post" action="/ansible/inventory/groups/{{ grp.id }}/delete" style="display:inline;">
                <button type="submit" class="btn btn-sm btn-danger"
                        onclick="return confirm('Delete group {{ grp.name }}?')">×</button>
            </form>
        </td>
    </tr>
    {% endfor %}
    </tbody>
</table>
</div>
{% endif %}
{% endblock %}

Step 7.3: Create steward/templates/ansible/inventory/group_detail.html

{% extends "base.html" %}
{% block title %}Group: {{ group.name }} — Steward{% endblock %}
{% block content %}
<div style="display:flex;align-items:center;justify-content:space-between;margin-bottom:1.5rem;">
    <h1 class="page-title" style="margin-bottom:0;">Group: {{ group.name }}</h1>
    <a href="/ansible/inventory/groups" class="btn btn-ghost btn-sm">← Groups</a>
</div>

<form method="post" action="/ansible/inventory/groups/{{ group.id }}">
    <div class="card" style="margin-bottom:1.5rem;">
        <h3 class="section-title">Settings</h3>
        <div class="form-group">
            <label>Name</label>
            <input type="text" name="name" value="{{ group.name }}" required>
        </div>
        <div class="form-group">
            <label>
                Group Variables
                <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">
                    (YAML, applied to all members — per-target vars override these)
                </span>
            </label>
            <textarea name="ansible_vars" rows="8"
                      style="width:100%;font-family:ui-monospace,monospace;font-size:0.85rem;"
                      placeholder="ansible_user: deploy&#10;http_port: 80">{{ ansible_vars_yaml }}</textarea>
        </div>
    </div>

    <div class="card" style="margin-bottom:1.5rem;">
        <h3 class="section-title">Members</h3>
        {% if not all_targets %}
        <p style="color:var(--text-muted);font-size:0.9rem;">No targets exist yet.
            <a href="/ansible/inventory/targets">Add targets</a> first.</p>
        {% else %}
        <div style="display:grid;gap:0.4rem;margin-bottom:1rem;">
            {% for tgt in all_targets %}
            <label style="display:flex;align-items:center;gap:0.6rem;font-weight:normal;cursor:pointer;">
                <input type="checkbox" name="target_ids" value="{{ tgt.id }}"
                       {% if tgt.id in member_ids %}checked{% endif %}>
                <span>{{ tgt.name }}</span>
                <span style="color:var(--text-muted);font-size:0.82rem;">{{ tgt.address }}</span>
            </label>
            {% endfor %}
        </div>
        {% endif %}
    </div>

    <button type="submit" class="btn">Save Changes</button>
</form>
{% endblock %}

Step 7.4: Register blueprint in steward/app.py

After the existing from .ansible.routes import ansible_bp and app.register_blueprint(ansible_bp) lines, add:

from .ansible.inventory_routes import inventory_bp
app.register_blueprint(inventory_bp)

(These are inside the create_app() function, alongside the other blueprint registrations.)

Step 7.5: Run the app and verify /ansible/inventory/groups loads

cd /home/bvandeusen/Nextcloud/Projects/FabledSteward
docker compose up -d  # or start local dev server
# Navigate to http://localhost:5000/ansible/inventory/groups in browser

Expected: Groups list page renders with "No groups yet" message. No 500 errors.

Step 7.6: Commit

git add steward/ansible/inventory_routes.py \
        steward/app.py \
        steward/templates/ansible/inventory/groups.html \
        steward/templates/ansible/inventory/group_detail.html
git commit -m "feat(ansible): inventory groups CRUD routes + templates"

Task 8: Inventory Blueprint — Targets CRUD

Files:

Modify: steward/ansible/inventory_routes.py
Create: steward/templates/ansible/inventory/targets.html
Create: steward/templates/ansible/inventory/target_detail.html
Step 8.1: Add targets routes to steward/ansible/inventory_routes.py

Append to the existing inventory_routes.py file (after the group routes):

@inventory_bp.get("/targets")
@require_role(UserRole.viewer)
async def targets_list():
    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(AnsibleTarget)
            .options(selectinload(AnsibleTarget.groups))
            .order_by(AnsibleTarget.name)
        )
        targets = result.scalars().all()
    return await render_template("ansible/inventory/targets.html", targets=targets)


@inventory_bp.post("/targets")
@require_role(UserRole.operator)
async def targets_create():
    form = await request.form
    name = form.get("name", "").strip()
    address = form.get("address", "").strip()
    if not name or not address:
        return "name and address are required", 400
    try:
        ansible_vars = _parse_ansible_vars(form.get("ansible_vars", ""))
    except ValueError as exc:
        return str(exc), 400

    target = AnsibleTarget(
        id=str(uuid.uuid4()), name=name, address=address, ansible_vars=ansible_vars
    )
    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            db.add(target)
    return redirect(url_for("ansible_inventory.targets_list"))


@inventory_bp.get("/targets/<target_id>")
@require_role(UserRole.viewer)
async def target_detail(target_id: str):
    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(AnsibleTarget)
            .where(AnsibleTarget.id == target_id)
            .options(selectinload(AnsibleTarget.groups))
        )
        target = result.scalar_one_or_none()
        if target is None:
            return "Target not found", 404
        all_groups = (
            await db.execute(select(AnsibleGroup).order_by(AnsibleGroup.name))
        ).scalars().all()
    member_group_ids = {g.id for g in target.groups}
    ansible_vars_yaml = yaml.dump(target.ansible_vars) if target.ansible_vars else ""
    return await render_template(
        "ansible/inventory/target_detail.html",
        target=target,
        all_groups=all_groups,
        member_group_ids=member_group_ids,
        ansible_vars_yaml=ansible_vars_yaml,
    )


@inventory_bp.post("/targets/<target_id>")
@require_role(UserRole.operator)
async def target_update(target_id: str):
    form = await request.form
    try:
        ansible_vars = _parse_ansible_vars(form.get("ansible_vars", ""))
    except ValueError as exc:
        return str(exc), 400

    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            result = await db.execute(
                select(AnsibleTarget)
                .where(AnsibleTarget.id == target_id)
                .options(selectinload(AnsibleTarget.groups))
            )
            target = result.scalar_one_or_none()
            if target is None:
                return "Target not found", 404

            name = form.get("name", target.name).strip()
            address = form.get("address", target.address).strip()
            if name:
                target.name = name
            if address:
                target.address = address
            target.ansible_vars = ansible_vars

            new_group_ids = set(form.getlist("group_ids"))
            all_groups_result = await db.execute(select(AnsibleGroup))
            group_map = {g.id: g for g in all_groups_result.scalars().all()}
            target.groups = [group_map[gid] for gid in new_group_ids if gid in group_map]

    return redirect(url_for("ansible_inventory.target_detail", target_id=target_id))


@inventory_bp.post("/targets/<target_id>/delete")
@require_role(UserRole.operator)
async def target_delete(target_id: str):
    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            result = await db.execute(
                select(AnsibleTarget).where(AnsibleTarget.id == target_id)
            )
            target = result.scalar_one_or_none()
            if target is not None:
                await db.delete(target)
    return redirect(url_for("ansible_inventory.targets_list"))

Step 8.2: Create steward/templates/ansible/inventory/targets.html

{% extends "base.html" %}
{% block title %}Inventory Targets — Steward{% endblock %}
{% block content %}
<div style="display:flex;align-items:center;justify-content:space-between;margin-bottom:1.5rem;">
    <h1 class="page-title" style="margin-bottom:0;">Inventory Targets</h1>
    <a href="/ansible/inventory/groups" class="btn btn-ghost btn-sm">Groups</a>
</div>

<div class="card" style="margin-bottom:1.5rem;">
    <h3 class="section-title">New Target</h3>
    <form method="post" action="/ansible/inventory/targets" style="display:flex;gap:0.75rem;align-items:flex-end;">
        <div class="form-group" style="flex:1;margin:0;">
            <label>Name <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">(used as Ansible hostname)</span></label>
            <input type="text" name="name" placeholder="webserver-01" required>
        </div>
        <div class="form-group" style="flex:1;margin:0;">
            <label>Address <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">(IP or DNS)</span></label>
            <input type="text" name="address" placeholder="192.168.1.10" required>
        </div>
        <button type="submit" class="btn">Add</button>
    </form>
</div>

{% if not targets %}
<div class="card">
    <p style="color:var(--text-muted);">No targets yet. Add one above.</p>
</div>
{% else %}
<div class="card-flush">
<table class="table">
    <thead>
        <tr><th>Name</th><th>Address</th><th>Groups</th><th></th></tr>
    </thead>
    <tbody>
    {% for tgt in targets %}
    <tr>
        <td><strong>{{ tgt.name }}</strong></td>
        <td style="font-family:ui-monospace,monospace;font-size:0.88rem;">{{ tgt.address }}</td>
        <td style="font-size:0.82rem;color:var(--text-muted);">
            {% for grp in tgt.groups %}
            <span style="background:var(--bg-elevated);border-radius:3px;padding:0.1em 0.4em;margin-right:0.25rem;">{{ grp.name }}</span>
            {% endfor %}
        </td>
        <td class="td-actions">
            <a href="/ansible/inventory/targets/{{ tgt.id }}" class="btn btn-sm btn-ghost">Edit</a>
            <form method="post" action="/ansible/inventory/targets/{{ tgt.id }}/delete" style="display:inline;">
                <button type="submit" class="btn btn-sm btn-danger"
                        onclick="return confirm('Delete target {{ tgt.name }}?')">×</button>
            </form>
        </td>
    </tr>
    {% endfor %}
    </tbody>
</table>
</div>
{% endif %}
{% endblock %}

Step 8.3: Create steward/templates/ansible/inventory/target_detail.html

{% extends "base.html" %}
{% block title %}Target: {{ target.name }} — Steward{% endblock %}
{% block content %}
<div style="display:flex;align-items:center;justify-content:space-between;margin-bottom:1.5rem;">
    <h1 class="page-title" style="margin-bottom:0;">Target: {{ target.name }}</h1>
    <a href="/ansible/inventory/targets" class="btn btn-ghost btn-sm">← Targets</a>
</div>

<form method="post" action="/ansible/inventory/targets/{{ target.id }}">
    <div class="card" style="margin-bottom:1.5rem;">
        <h3 class="section-title">Connection</h3>
        <div class="form-group">
            <label>Name <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">(Ansible hostname — used in plays as <code>hosts: name</code>)</span></label>
            <input type="text" name="name" value="{{ target.name }}" required>
        </div>
        <div class="form-group">
            <label>Address <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">(becomes <code>ansible_host</code> in inventory)</span></label>
            <input type="text" name="address" value="{{ target.address }}" required>
        </div>
        <div class="form-group">
            <label>
                Target Variables
                <span style="color:var(--text-muted);font-weight:normal;font-size:0.82rem;">
                    (YAML — override group vars for this target only)
                </span>
            </label>
            <textarea name="ansible_vars" rows="8"
                      style="width:100%;font-family:ui-monospace,monospace;font-size:0.85rem;"
                      placeholder="ansible_user: ubuntu&#10;ansible_port: 22">{{ ansible_vars_yaml }}</textarea>
        </div>
    </div>

    <div class="card" style="margin-bottom:1.5rem;">
        <h3 class="section-title">Groups</h3>
        {% if not all_groups %}
        <p style="color:var(--text-muted);font-size:0.9rem;">No groups exist yet.
            <a href="/ansible/inventory/groups">Create a group</a> first.</p>
        {% else %}
        <div style="display:grid;gap:0.4rem;margin-bottom:1rem;">
            {% for grp in all_groups %}
            <label style="display:flex;align-items:center;gap:0.6rem;font-weight:normal;cursor:pointer;">
                <input type="checkbox" name="group_ids" value="{{ grp.id }}"
                       {% if grp.id in member_group_ids %}checked{% endif %}>
                <span>{{ grp.name }}</span>
            </label>
            {% endfor %}
        </div>
        {% endif %}
    </div>

    <button type="submit" class="btn">Save Changes</button>
</form>
{% endblock %}

Step 8.4: Verify targets CRUD works

Navigate to http://localhost:5000/ansible/inventory/targets in browser. Create a target, edit it, assign it to a group, verify it appears in the groups list as a member.

Step 8.5: Commit

git add steward/ansible/inventory_routes.py \
        steward/templates/ansible/inventory/targets.html \
        steward/templates/ansible/inventory/target_detail.html
git commit -m "feat(ansible): inventory targets CRUD routes + templates"

Task 9: Host Edit Form — Ansible Target Link Section

Files:

Modify: steward/templates/hosts/form.html
Modify: steward/hosts/routes.py
Step 9.1: Update edit_host() route in steward/hosts/routes.py

The edit_host() function currently (around line ~149) renders hosts/form.html with just the host. Extend it to also pass the linked Ansible target and all available targets:

@hosts_bp.get("/<host_id>/edit")
@require_role(UserRole.operator)
async def edit_host(host_id: str):
    from steward.models.ansible_inventory import AnsibleTarget
    from sqlalchemy.orm import selectinload

    async with current_app.db_sessionmaker() as db:
        result = await db.execute(
            select(Host).where(Host.id == host_id)
        )
        host = result.scalar_one_or_none()
        if host is None:
            return "Host not found", 404

        linked_target = None
        linked_result = await db.execute(
            select(AnsibleTarget)
            .where(AnsibleTarget.host_id == host_id)
            .options(selectinload(AnsibleTarget.groups))
        )
        linked_target = linked_result.scalar_one_or_none()

        all_targets = (
            await db.execute(
                select(AnsibleTarget)
                .where(AnsibleTarget.host_id.is_(None))
                .order_by(AnsibleTarget.name)
            )
        ).scalars().all()

    return await render_template(
        "hosts/form.html",
        host=host,
        probe_types=list(ProbeType),
        linked_target=linked_target,
        linkable_targets=all_targets,
    )

Note: all_targets shows only unlinked targets (those without a host_id) plus the currently linked target if any — so the user can re-select it. The form template handles this.

Also add a new route POST /<host_id>/ansible-link to handle the link action:

@hosts_bp.post("/<host_id>/ansible-link")
@require_role(UserRole.operator)
async def ansible_link(host_id: str):
    from steward.models.ansible_inventory import AnsibleTarget

    form = await request.form
    action = form.get("action", "")

    async with current_app.db_sessionmaker() as db:
        async with db.begin():
            if action == "unlink":
                # Clear host_id from the currently linked target.
                linked_result = await db.execute(
                    select(AnsibleTarget).where(AnsibleTarget.host_id == host_id)
                )
                linked = linked_result.scalar_one_or_none()
                if linked:
                    linked.host_id = None
            elif action == "link":
                target_id = form.get("target_id", "").strip()
                if target_id:
                    result = await db.execute(
                        select(AnsibleTarget).where(AnsibleTarget.id == target_id)
                    )
                    target = result.scalar_one_or_none()
                    if target:
                        # Clear any previous link for this host.
                        old_result = await db.execute(
                            select(AnsibleTarget).where(AnsibleTarget.host_id == host_id)
                        )
                        old = old_result.scalar_one_or_none()
                        if old and old.id != target_id:
                            old.host_id = None
                        target.host_id = host_id
            elif action == "create":
                # Create a new target pre-filled from host data.
                host_result = await db.execute(
                    select(Host).where(Host.id == host_id)
                )
                host = host_result.scalar_one_or_none()
                if host:
                    new_target = AnsibleTarget(
                        id=str(uuid.uuid4()),
                        name=host.name,
                        address=host.address,
                        host_id=host_id,
                    )
                    db.add(new_target)

    return redirect(f"/hosts/{host_id}/edit")

Add import uuid to the top of hosts/routes.py if not already present. Also add from sqlalchemy import select and from steward.models.hosts import Host, ProbeType imports (check what's already imported).

Step 9.2: Add Ansible section to steward/templates/hosts/form.html

At the end of the file, before {% endblock %}, add:

{% if host %}
<div class="card" style="margin-top:1.5rem;">
    <h3 class="section-title">Ansible Target</h3>
    {% if linked_target %}
    <div style="display:flex;align-items:center;gap:1rem;margin-bottom:1rem;padding:0.75rem;background:var(--bg-elevated);border-radius:4px;">
        <div style="flex:1;">
            <strong>{{ linked_target.name }}</strong>
            <span style="color:var(--text-muted);font-size:0.85rem;margin-left:0.5rem;">{{ linked_target.address }}</span>
            {% if linked_target.groups %}
            <div style="margin-top:0.25rem;">
                {% for grp in linked_target.groups %}
                <span style="font-size:0.78rem;background:var(--bg);border-radius:3px;padding:0.1em 0.4em;margin-right:0.2rem;color:var(--text-muted);">{{ grp.name }}</span>
                {% endfor %}
            </div>
            {% endif %}
        </div>
        <a href="/ansible/inventory/targets/{{ linked_target.id }}" class="btn btn-ghost btn-sm">Edit Target</a>
        <form method="post" action="/hosts/{{ host.id }}/ansible-link" style="display:inline;">
            <input type="hidden" name="action" value="unlink">
            <button type="submit" class="btn btn-ghost btn-sm">Unlink</button>
        </form>
    </div>
    {% else %}
    <p style="color:var(--text-muted);font-size:0.9rem;margin-bottom:1rem;">
        No Ansible target linked. Link an existing target or create a new one from this host's data.
    </p>
    <div style="display:flex;gap:0.75rem;flex-wrap:wrap;">
        {% if linkable_targets %}
        <form method="post" action="/hosts/{{ host.id }}/ansible-link" style="display:flex;gap:0.5rem;align-items:center;">
            <input type="hidden" name="action" value="link">
            <select name="target_id" style="flex:1;">
                {% for tgt in linkable_targets %}
                <option value="{{ tgt.id }}">{{ tgt.name }} ({{ tgt.address }})</option>
                {% endfor %}
            </select>
            <button type="submit" class="btn btn-sm">Link</button>
        </form>
        {% endif %}
        <form method="post" action="/hosts/{{ host.id }}/ansible-link">
            <input type="hidden" name="action" value="create">
            <button type="submit" class="btn btn-sm btn-ghost">Create Target from This Host</button>
        </form>
    </div>
    {% endif %}
</div>
{% endif %}

Step 9.3: Verify the Ansible section appears on host edit

Navigate to http://localhost:5000/hosts/<any-host-id>/edit. Verify the Ansible Target section appears at the bottom. Create a target from a host, verify it links and the target appears. Unlink and verify it clears.

Step 9.4: Commit

git add steward/hosts/routes.py steward/templates/hosts/form.html
git commit -m "feat(ansible): host edit page Ansible target link/create/unlink section"

Task 10: Integration Test — Migrations + fetch_scope_targets

Files:

Create: tests/integration/test_ansible_inventory.py
Step 10.1: Write integration tests

Create tests/integration/test_ansible_inventory.py:

"""Integration tests: ansible_targets / ansible_groups tables + fetch_scope_targets.

Requires a live Postgres DB. Run with:
  pytest tests/integration/test_ansible_inventory.py -m integration -v
"""
import pytest
import pytest_asyncio
from sqlalchemy import select, text


@pytest.mark.integration
@pytest.mark.asyncio
async def test_ansible_targets_table_exists(app):
    """Migration 0015 created the ansible_targets table."""
    async with app.db_sessionmaker() as db:
        result = await db.execute(
            text("SELECT column_name FROM information_schema.columns WHERE table_name = 'ansible_targets'")
        )
        columns = {row[0] for row in result.fetchall()}
    assert "id" in columns
    assert "name" in columns
    assert "address" in columns
    assert "ansible_vars" in columns
    assert "host_id" in columns


@pytest.mark.integration
@pytest.mark.asyncio
async def test_ansible_groups_table_exists(app):
    """Migration 0016 created ansible_groups and ansible_target_groups."""
    async with app.db_sessionmaker() as db:
        result = await db.execute(
            text("SELECT column_name FROM information_schema.columns WHERE table_name = 'ansible_groups'")
        )
        columns = {row[0] for row in result.fetchall()}
    assert "id" in columns
    assert "name" in columns
    assert "ansible_vars" in columns

    async with app.db_sessionmaker() as db:
        result = await db.execute(
            text("SELECT column_name FROM information_schema.columns WHERE table_name = 'ansible_target_groups'")
        )
        join_columns = {row[0] for row in result.fetchall()}
    assert "target_id" in join_columns
    assert "group_id" in join_columns


@pytest.mark.integration
@pytest.mark.asyncio
async def test_ansible_run_scope_column_exists(app):
    """Migration 0017 added inventory_scope to ansible_runs."""
    async with app.db_sessionmaker() as db:
        result = await db.execute(
            text("SELECT column_name FROM information_schema.columns WHERE table_name = 'ansible_runs'")
        )
        columns = {row[0] for row in result.fetchall()}
    assert "inventory_scope" in columns
    assert "inventory_path" in columns  # still present, now nullable


@pytest.mark.integration
@pytest.mark.asyncio
async def test_create_target_and_group_and_fetch(app):
    """Can create a target+group, assign membership, and fetch via fetch_scope_targets."""
    import uuid
    from steward.models.ansible_inventory import AnsibleTarget, AnsibleGroup
    from steward.ansible.inventory_gen import fetch_scope_targets, generate_inventory

    target_id = str(uuid.uuid4())
    group_id = str(uuid.uuid4())

    async with app.db_sessionmaker() as db:
        async with db.begin():
            grp = AnsibleGroup(
                id=group_id,
                name=f"test-group-{group_id[:8]}",
                ansible_vars={"env": "test"},
            )
            tgt = AnsibleTarget(
                id=target_id,
                name=f"test-target-{target_id[:8]}",
                address="10.0.99.1",
                ansible_vars={"ansible_user": "ubuntu"},
            )
            tgt.groups.append(grp)
            db.add(grp)
            db.add(tgt)

    # Fetch all
    async with app.db_sessionmaker() as db:
        all_targets = await fetch_scope_targets(db, "steward:all")
    names = [t.name for t in all_targets]
    assert f"test-target-{target_id[:8]}" in names

    # Fetch by group
    async with app.db_sessionmaker() as db:
        group_targets = await fetch_scope_targets(db, f"steward:group:{group_id}")
    assert any(t.id == target_id for t in group_targets)

    # Fetch single target
    async with app.db_sessionmaker() as db:
        single = await fetch_scope_targets(db, f"steward:target:{target_id}")
    assert len(single) == 1
    assert single[0].id == target_id

    # Generate inventory
    async with app.db_sessionmaker() as db:
        targets = await fetch_scope_targets(db, f"steward:group:{group_id}")
    inv = generate_inventory(targets)
    target_name = f"test-target-{target_id[:8]}"
    group_name = f"test-group-{group_id[:8]}"
    assert target_name in inv["all"]["hosts"]
    assert target_name in inv[group_name]["hosts"]
    assert inv["_meta"]["hostvars"][target_name]["ansible_host"] == "10.0.99.1"
    assert inv["_meta"]["hostvars"][target_name]["ansible_user"] == "ubuntu"
    assert inv["_meta"]["hostvars"][target_name]["env"] == "test"

    # Cleanup
    async with app.db_sessionmaker() as db:
        async with db.begin():
            result = await db.execute(select(AnsibleTarget).where(AnsibleTarget.id == target_id))
            t = result.scalar_one_or_none()
            if t:
                await db.delete(t)
            result = await db.execute(select(AnsibleGroup).where(AnsibleGroup.id == group_id))
            g = result.scalar_one_or_none()
            if g:
                await db.delete(g)

Step 10.2: Run integration tests

pytest tests/integration/test_ansible_inventory.py -m integration -v

Expected: 4 PASSED (requires running Postgres with migrations applied).

Step 10.3: Commit

git add tests/integration/test_ansible_inventory.py
git commit -m "test(ansible): integration tests for inventory tables + fetch_scope_targets"

Self-Review Checklist

Spec coverage: All 8 spec sections are covered:
- §1 Data Model → Tasks 1–3
- §2 PAT/HTTP Token → Task 5
- §3 Inventory Generation → Task 4
- §4 Run Form → Task 6
- §5 UI Surfaces → Tasks 7–9
- §6 Repo integration → Task 6 (scope fallback preserved)
- §7 Migrations → Tasks 1–3
- §8 Out of scope → not implemented ✓
No placeholders: All code blocks are complete.
Type consistency: AnsibleTarget, AnsibleGroup, ansible_target_groups defined in Task 1 and referenced consistently throughout. fetch_scope_targets(db, scope) / generate_inventory(targets) defined in Task 4 and called in Task 6.
Migration chain: 0015 → 0016 → 0017, each down_revision points to the previous.
Session pattern: All routes use async with current_app.db_sessionmaker() as db: matching existing code.
Import pattern: New models added to steward/models/__init__.py for Alembic discovery.
get_sources() http_token: Added for both git and local sources (empty string for local).

73 KiB Raw Blame History Unescape Escape

Ansible Inventory Infrastructure Implementation Plan

File Map

Task 1: AnsibleTarget Model + Migration

Task 2: AnsibleGroup + Join Table Migration

Task 3: inventory_scope Column on AnsibleRun

Task 4: Inventory Generation — Pure Function + Unit Tests

Task 5: GIT_ASKPASS — Per-Source HTTP Token Auth

Task 6: Wire Inventory Generation into create_run()

Task 7: Inventory Blueprint — Groups CRUD

Task 8: Inventory Blueprint — Targets CRUD

Task 9: Host Edit Form — Ansible Target Link Section

Task 10: Integration Test — Migrations + fetch_scope_targets

Self-Review Checklist

73 KiB

Raw Blame History