FabledSteward/docs/architecture.md

# Architecture

Steward is a single Quart (async Python) process. There are no separate worker processes, no message brokers, and no build pipeline for the frontend. All monitoring, scheduling, and request handling runs on a single asyncio event loop.

---

## Startup Sequence

`create_app()` in `steward/app.py` runs these steps synchronously before the event loop starts:

| Step | What happens |
|---|---|
| 1 | Bootstrap config loaded (`database_url`, `secret_key`, `plugin_dir`) |
| 2 | SQLAlchemy async engine and session factory attached to `app` |
| 3 | Core Alembic migrations applied (creates `app_settings` table among others) |
| 4 | All settings loaded from `app_settings` DB table into `app.config` |
| 5 | Each enabled plugin's Alembic migrations applied |
| 6 | Alert pipeline initialised (`init_alerts(app)` stores app ref for deferred notifications) |
| 7 | Core blueprints registered (auth, dashboard, hosts, ping, dns, alerts, ansible, settings) |
| 8 | Core scheduled tasks registered into `app._task_registry` |
| 9 | Plugins loaded via `load_plugins(app)` — blueprints and tasks appended |
| 10 | `/health` endpoint registered |
| 11 | `before_serving` hook starts the scheduler as an `asyncio.create_task()` |

The two-phase migration approach (step 3 core → step 4 load settings → step 5 plugins) exists because plugin enabling is stored in the settings DB, which requires the core schema to exist first.

---

## Request Routing

All routes are Quart Blueprints registered in `app.py`:

| Prefix | Blueprint | Module |
|---|---|---|
| `/auth/` | `auth_bp` | `steward/auth/routes.py` |
| `/` | `dashboard_bp` | `steward/dashboard/routes.py` |
| `/hosts/` | `hosts_bp` | `steward/hosts/routes.py` |
| `/ping/` | `ping_bp` | `steward/ping/routes.py` |
| `/dns/` | `dns_bp` | `steward/dns/routes.py` |
| `/alerts/` | `alerts_bp` | `steward/alerts/routes.py` |
| `/ansible/` | `ansible_bp` | `steward/ansible/routes.py` |
| `/settings/` | `settings_bp` | `steward/settings/routes.py` |
| `/plugins/<name>/` | plugin blueprint | `plugins/<name>/routes.py` |

Plugin blueprints are mounted automatically by `load_plugins()` using the plugin directory name as the URL prefix.

`/ai-access/` (`ai_access_bp`, `steward/ai_access/routes.py`) is where a signed-in user manages their API tokens.

### `/mcp`: outside the blueprint pipeline

`mount_mcp(app)` (`steward/mcp/server.py`) wraps `app.asgi_app`. A request to
`/mcp` is handed to the MCP SDK's streamable-HTTP app before Quart sees it, so
no blueprint, session cookie or `before_request` hook runs for it. Each
request then goes through these steps:

1. **Database gate.** If the database is unreachable, the request gets a 503
   with a readable message.
2. **Bearer resolution.** `api_tokens.resolve_token()` turns the token into a
   `Caller` (user, effective role, token id). A missing, revoked or expired
   token gets a 401.
3. **Caller context.** The `Caller` and the app are set in ContextVars
   (`steward/mcp/_context.py`). Tools read them through `current_caller()` and
   `mcp_app()`, because Quart's `current_app` and `session` do not exist there.
4. **Role gate.** `StewardMCPServer.refusal()` hides and refuses tools above
   the caller's role, and plugin tools whose plugin is disabled.

The SDK's session manager runs in Quart's `before_serving`/`after_serving`
lifecycle. The server is stateless: no MCP session outlives a request. See
[MCP server](core/mcp.md) for the security model and the tools.

---

## Scheduler

`steward/core/scheduler.py` exports two things:

- `ScheduledTask` dataclass — holds a `name`, `coro_factory` (zero-argument callable returning a coroutine), `interval_seconds`, and optional `run_on_startup` flag
- `start_scheduler(tasks)` — async function that loops every second, calling `asyncio.create_task()` for each task whose interval has elapsed

Tasks are never awaited serially — each runs as a fully independent asyncio task. Exceptions inside tasks are caught and logged; they do not crash other tasks or the app.

Core tasks registered in `app._task_registry`:
- `ping_monitor` — runs every `monitors.poll_interval_seconds` (default 60s), `run_on_startup=True`
- `dns_monitor` — same interval, `run_on_startup=True`
- `data_cleanup` — runs hourly, `run_on_startup=False`
- `ansible_git_pull_<name>` — one per configured Git source, interval from source config

---

## Metric Store

Every plugin and monitor records into one store keyed by (source, resource, metric):
- `plugin_metrics` holds raw samples for `metrics.retention.raw_days` (7 by default).
- `plugin_metric_latest` holds each key's newest value.
- `data_cleanup` rolls raw samples older than the raw window into `plugin_metrics_hourly`, kept for `metrics.retention.rollup_days` (90 by default).

Each rolled-up hour keeps three figures:
- `value_avg`: the hour's average.
- `value_max`: its peak.
- `value_p95`: the level it held for all but its briefest 5%, about 3 minutes.

A two-minute spike raises the peak and leaves the p95 low; an hour spent at that level raises both. Learned normals use this to tell a recurring spike from a held level. Hours rolled up before the p95 was kept have it NULL; read those as their average.

### Normals and anomalies

A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).

Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, opens an anomaly (`steward/core/anomalies.py`):
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
- `data_cleanup` prunes reviewed and lapsed rows at `data.retention_days`; a row still to review is never pruned.

What learns a normal, and the floor each departure must clear (Settings → Thresholds → Normals, `norms.floor.*`; a floor may be a function, `floor_setting()`, read at each pass):

| Kind | Metric | Floor (default) |
|---|---|---|
| Host CPU | `host_agent` / `cpu_pct` | 10 points |
| Host memory | `host_agent` / `mem_used_pct` | 5 points |
| Host load per core | `host_agent` / `load_pct` (load_1m ÷ cores × 100, recorded at ingest) | 10 points |
| Container CPU | `docker` / `cpu_pct` (100% is one core) | 10 points of one core |
| Container memory | `docker` / `mem_pct` (of its limit, or of host RAM when it has none) | 5 points |
| Monitor response time | `<type>` / `response_ms` | 100 ms |
| Router 4xx share, p95 | `traefik` / `error_rate_4xx_pct`, `latency_p95_ms` | 2 points, 50 ms |

Host metrics learn with `parts=False`: the host learns, its cores (`<host>:core<N>`) don't. Disk space, a monitor that is down, an agent gone quiet and a router's 5xx share are failures and keep their fixed cutoffs.

Pages grade by the normal through `anomalies.graded()` (hosts' `host_grades` and `worst_reading(normals=)`, `docker.service.usage(thresholds=, norms=)`, `monitors.service.latency_grade`): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages list their anomalies with `_anomaly_card.html`.

---

## Database Session Pattern

`app.db_sessionmaker` is an SQLAlchemy `async_sessionmaker`. All DB access follows this pattern:

```python
async with current_app.db_sessionmaker() as session:
    async with session.begin():
        # queries and writes here
        # transaction commits on exit, rolls back on exception
```

Sessions are never shared across request boundaries or between tasks.

---

## Config Two-Layer Design

There are exactly two config layers:

1. **Bootstrap** (`config.yaml` + env vars) — `database_url`, `secret_key`, `plugin_dir` only. Read once at startup before the event loop.
2. **App settings** (`app_settings` DB table) — everything else: SMTP, webhooks, Ansible sources, monitor intervals, ping thresholds, plugin config. Read at startup via `load_settings_sync()` and written via the Settings UI at runtime.

This means the only file you must touch to get the app running is the database URL. Everything else can be configured through the web UI.

---

## Frontend Approach

No JavaScript framework, no build step. The frontend is:

- **Jinja2 templates** rendered server-side (`steward/templates/`)
- **HTMX** for live-updating fragments (dashboard widgets, ping pills, DNS status)
- A single CSS design system in `steward/templates/base.html` using CSS custom properties

Live-updating widgets use HTMX polling:
```html
<div hx-get="/ping/rows" hx-trigger="load, every 30s" hx-swap="innerHTML">
```

Fragment endpoints (`/ping/rows`, `/dns/rows`, `/plugins/traefik/widget`) return partial HTML that HTMX swaps in without a full page reload.