bvandeusenandClaude Opus 5.5
cbbf186f17
plugin_metrics_hourly gains value_p95 beside value_avg and value_max. A two-minute backup spike raises the hour's peak and leaves its p95 low; an hour spent at that level raises both. Learned normals (milestone 513) need that difference to call a recurring spike usual without calling the same level held at another hour usual too. Hours rolled up before this keep a NULL p95 and read as their average. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
146 lines
6.7 KiB
Markdown
146 lines
6.7 KiB
Markdown
# Architecture
|
|
|
|
Steward is a single Quart (async Python) process. There are no separate worker processes, no message brokers, and no build pipeline for the frontend. All monitoring, scheduling, and request handling runs on a single asyncio event loop.
|
|
|
|
---
|
|
|
|
## Startup Sequence
|
|
|
|
`create_app()` in `steward/app.py` runs these steps synchronously before the event loop starts:
|
|
|
|
| Step | What happens |
|
|
|---|---|
|
|
| 1 | Bootstrap config loaded (`database_url`, `secret_key`, `plugin_dir`) |
|
|
| 2 | SQLAlchemy async engine and session factory attached to `app` |
|
|
| 3 | Core Alembic migrations applied (creates `app_settings` table among others) |
|
|
| 4 | All settings loaded from `app_settings` DB table into `app.config` |
|
|
| 5 | Each enabled plugin's Alembic migrations applied |
|
|
| 6 | Alert pipeline initialised (`init_alerts(app)` stores app ref for deferred notifications) |
|
|
| 7 | Core blueprints registered (auth, dashboard, hosts, ping, dns, alerts, ansible, settings) |
|
|
| 8 | Core scheduled tasks registered into `app._task_registry` |
|
|
| 9 | Plugins loaded via `load_plugins(app)` — blueprints and tasks appended |
|
|
| 10 | `/health` endpoint registered |
|
|
| 11 | `before_serving` hook starts the scheduler as an `asyncio.create_task()` |
|
|
|
|
The two-phase migration approach (step 3 core → step 4 load settings → step 5 plugins) exists because plugin enabling is stored in the settings DB, which requires the core schema to exist first.
|
|
|
|
---
|
|
|
|
## Request Routing
|
|
|
|
All routes are Quart Blueprints registered in `app.py`:
|
|
|
|
| Prefix | Blueprint | Module |
|
|
|---|---|---|
|
|
| `/auth/` | `auth_bp` | `steward/auth/routes.py` |
|
|
| `/` | `dashboard_bp` | `steward/dashboard/routes.py` |
|
|
| `/hosts/` | `hosts_bp` | `steward/hosts/routes.py` |
|
|
| `/ping/` | `ping_bp` | `steward/ping/routes.py` |
|
|
| `/dns/` | `dns_bp` | `steward/dns/routes.py` |
|
|
| `/alerts/` | `alerts_bp` | `steward/alerts/routes.py` |
|
|
| `/ansible/` | `ansible_bp` | `steward/ansible/routes.py` |
|
|
| `/settings/` | `settings_bp` | `steward/settings/routes.py` |
|
|
| `/plugins/<name>/` | plugin blueprint | `plugins/<name>/routes.py` |
|
|
|
|
Plugin blueprints are mounted automatically by `load_plugins()` using the plugin directory name as the URL prefix.
|
|
|
|
`/ai-access/` (`ai_access_bp`, `steward/ai_access/routes.py`) is where a signed-in user manages their API tokens.
|
|
|
|
### `/mcp`: outside the blueprint pipeline
|
|
|
|
`mount_mcp(app)` (`steward/mcp/server.py`) wraps `app.asgi_app`. A request to
|
|
`/mcp` is handed to the MCP SDK's streamable-HTTP app before Quart sees it, so
|
|
no blueprint, session cookie or `before_request` hook runs for it. Each
|
|
request then goes through these steps:
|
|
|
|
1. **Database gate.** If the database is unreachable, the request gets a 503
|
|
with a readable message.
|
|
2. **Bearer resolution.** `api_tokens.resolve_token()` turns the token into a
|
|
`Caller` (user, effective role, token id). A missing, revoked or expired
|
|
token gets a 401.
|
|
3. **Caller context.** The `Caller` and the app are set in ContextVars
|
|
(`steward/mcp/_context.py`). Tools read them through `current_caller()` and
|
|
`mcp_app()`, because Quart's `current_app` and `session` do not exist there.
|
|
4. **Role gate.** `StewardMCPServer.refusal()` hides and refuses tools above
|
|
the caller's role, and plugin tools whose plugin is disabled.
|
|
|
|
The SDK's session manager runs in Quart's `before_serving`/`after_serving`
|
|
lifecycle. The server is stateless: no MCP session outlives a request. See
|
|
[MCP server](core/mcp.md) for the security model and the tools.
|
|
|
|
---
|
|
|
|
## Scheduler
|
|
|
|
`steward/core/scheduler.py` exports two things:
|
|
|
|
- `ScheduledTask` dataclass — holds a `name`, `coro_factory` (zero-argument callable returning a coroutine), `interval_seconds`, and optional `run_on_startup` flag
|
|
- `start_scheduler(tasks)` — async function that loops every second, calling `asyncio.create_task()` for each task whose interval has elapsed
|
|
|
|
Tasks are never awaited serially — each runs as a fully independent asyncio task. Exceptions inside tasks are caught and logged; they do not crash other tasks or the app.
|
|
|
|
Core tasks registered in `app._task_registry`:
|
|
- `ping_monitor` — runs every `monitors.poll_interval_seconds` (default 60s), `run_on_startup=True`
|
|
- `dns_monitor` — same interval, `run_on_startup=True`
|
|
- `data_cleanup` — runs hourly, `run_on_startup=False`
|
|
- `ansible_git_pull_<name>` — one per configured Git source, interval from source config
|
|
|
|
---
|
|
|
|
## Metric Store
|
|
|
|
Every plugin and monitor records into one store keyed by (source, resource, metric):
|
|
- `plugin_metrics` holds raw samples for `metrics.retention.raw_days` (7 by default).
|
|
- `plugin_metric_latest` holds each key's newest value.
|
|
- `data_cleanup` rolls raw samples older than the raw window into `plugin_metrics_hourly`, kept for `metrics.retention.rollup_days` (90 by default).
|
|
|
|
Each rolled-up hour keeps three figures:
|
|
- `value_avg`: the hour's average.
|
|
- `value_max`: its peak.
|
|
- `value_p95`: the level it held for all but its briefest 5%, about 3 minutes.
|
|
|
|
A two-minute spike raises the peak and leaves the p95 low; an hour spent at that level raises both. Learned normals use this to tell a recurring spike from a held level. Hours rolled up before the p95 was kept have it NULL; read those as their average.
|
|
|
|
---
|
|
|
|
## Database Session Pattern
|
|
|
|
`app.db_sessionmaker` is an SQLAlchemy `async_sessionmaker`. All DB access follows this pattern:
|
|
|
|
```python
|
|
async with current_app.db_sessionmaker() as session:
|
|
async with session.begin():
|
|
# queries and writes here
|
|
# transaction commits on exit, rolls back on exception
|
|
```
|
|
|
|
Sessions are never shared across request boundaries or between tasks.
|
|
|
|
---
|
|
|
|
## Config Two-Layer Design
|
|
|
|
There are exactly two config layers:
|
|
|
|
1. **Bootstrap** (`config.yaml` + env vars) — `database_url`, `secret_key`, `plugin_dir` only. Read once at startup before the event loop.
|
|
2. **App settings** (`app_settings` DB table) — everything else: SMTP, webhooks, Ansible sources, monitor intervals, ping thresholds, plugin config. Read at startup via `load_settings_sync()` and written via the Settings UI at runtime.
|
|
|
|
This means the only file you must touch to get the app running is the database URL. Everything else can be configured through the web UI.
|
|
|
|
---
|
|
|
|
## Frontend Approach
|
|
|
|
No JavaScript framework, no build step. The frontend is:
|
|
|
|
- **Jinja2 templates** rendered server-side (`steward/templates/`)
|
|
- **HTMX** for live-updating fragments (dashboard widgets, ping pills, DNS status)
|
|
- A single CSS design system in `steward/templates/base.html` using CSS custom properties
|
|
|
|
Live-updating widgets use HTMX polling:
|
|
```html
|
|
<div hx-get="/ping/rows" hx-trigger="load, every 30s" hx-swap="innerHTML">
|
|
```
|
|
|
|
Fragment endpoints (`/ping/rows`, `/dns/rows`, `/plugins/traefik/widget`) return partial HTML that HTMX swaps in without a full page reload.
|