bvandeusenandClaude Opus 5.5
5569f9934e
#5740 "A metric that departs from its normal is recorded as an anomaly to review, and is findable after it ends", milestone 513. - Every minute each learned key's last 15 minutes is judged against its hour's normal: above the usual hold x1.25 and by the metric's floor opens an occurrence, which ends when the level comes back. - metric_anomalies holds one row per item to review; a repeat while it still waits joins it as an occurrence (metric_anomaly_spans). - An ended row nobody reviews within anomalies.review_days (14, Settings -> Thresholds) lapses; an ongoing one never does. - Learning skips every hour an occurrence covers unless its anomaly was marked normal: no normal is learned without review. - Alerts -> Anomalies lists those to review first, then the reviewed and lapsed under day dividers; marking is operator/admin and audited. - The Overview widget and get_overview warn of open ones and count those to review; the host page shows its own; the weekly report lists the week's and the lapsed; MCP list_anomalies and review_anomaly. - Reviewed and lapsed rows are pruned at data.retention_days, and kept in backups like alert history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
7.7 KiB
Markdown
156 lines
7.7 KiB
Markdown
# Architecture
|
||
|
||
Steward is a single Quart (async Python) process. There are no separate worker processes, no message brokers, and no build pipeline for the frontend. All monitoring, scheduling, and request handling runs on a single asyncio event loop.
|
||
|
||
---
|
||
|
||
## Startup Sequence
|
||
|
||
`create_app()` in `steward/app.py` runs these steps synchronously before the event loop starts:
|
||
|
||
| Step | What happens |
|
||
|---|---|
|
||
| 1 | Bootstrap config loaded (`database_url`, `secret_key`, `plugin_dir`) |
|
||
| 2 | SQLAlchemy async engine and session factory attached to `app` |
|
||
| 3 | Core Alembic migrations applied (creates `app_settings` table among others) |
|
||
| 4 | All settings loaded from `app_settings` DB table into `app.config` |
|
||
| 5 | Each enabled plugin's Alembic migrations applied |
|
||
| 6 | Alert pipeline initialised (`init_alerts(app)` stores app ref for deferred notifications) |
|
||
| 7 | Core blueprints registered (auth, dashboard, hosts, ping, dns, alerts, ansible, settings) |
|
||
| 8 | Core scheduled tasks registered into `app._task_registry` |
|
||
| 9 | Plugins loaded via `load_plugins(app)` — blueprints and tasks appended |
|
||
| 10 | `/health` endpoint registered |
|
||
| 11 | `before_serving` hook starts the scheduler as an `asyncio.create_task()` |
|
||
|
||
The two-phase migration approach (step 3 core → step 4 load settings → step 5 plugins) exists because plugin enabling is stored in the settings DB, which requires the core schema to exist first.
|
||
|
||
---
|
||
|
||
## Request Routing
|
||
|
||
All routes are Quart Blueprints registered in `app.py`:
|
||
|
||
| Prefix | Blueprint | Module |
|
||
|---|---|---|
|
||
| `/auth/` | `auth_bp` | `steward/auth/routes.py` |
|
||
| `/` | `dashboard_bp` | `steward/dashboard/routes.py` |
|
||
| `/hosts/` | `hosts_bp` | `steward/hosts/routes.py` |
|
||
| `/ping/` | `ping_bp` | `steward/ping/routes.py` |
|
||
| `/dns/` | `dns_bp` | `steward/dns/routes.py` |
|
||
| `/alerts/` | `alerts_bp` | `steward/alerts/routes.py` |
|
||
| `/ansible/` | `ansible_bp` | `steward/ansible/routes.py` |
|
||
| `/settings/` | `settings_bp` | `steward/settings/routes.py` |
|
||
| `/plugins/<name>/` | plugin blueprint | `plugins/<name>/routes.py` |
|
||
|
||
Plugin blueprints are mounted automatically by `load_plugins()` using the plugin directory name as the URL prefix.
|
||
|
||
`/ai-access/` (`ai_access_bp`, `steward/ai_access/routes.py`) is where a signed-in user manages their API tokens.
|
||
|
||
### `/mcp`: outside the blueprint pipeline
|
||
|
||
`mount_mcp(app)` (`steward/mcp/server.py`) wraps `app.asgi_app`. A request to
|
||
`/mcp` is handed to the MCP SDK's streamable-HTTP app before Quart sees it, so
|
||
no blueprint, session cookie or `before_request` hook runs for it. Each
|
||
request then goes through these steps:
|
||
|
||
1. **Database gate.** If the database is unreachable, the request gets a 503
|
||
with a readable message.
|
||
2. **Bearer resolution.** `api_tokens.resolve_token()` turns the token into a
|
||
`Caller` (user, effective role, token id). A missing, revoked or expired
|
||
token gets a 401.
|
||
3. **Caller context.** The `Caller` and the app are set in ContextVars
|
||
(`steward/mcp/_context.py`). Tools read them through `current_caller()` and
|
||
`mcp_app()`, because Quart's `current_app` and `session` do not exist there.
|
||
4. **Role gate.** `StewardMCPServer.refusal()` hides and refuses tools above
|
||
the caller's role, and plugin tools whose plugin is disabled.
|
||
|
||
The SDK's session manager runs in Quart's `before_serving`/`after_serving`
|
||
lifecycle. The server is stateless: no MCP session outlives a request. See
|
||
[MCP server](core/mcp.md) for the security model and the tools.
|
||
|
||
---
|
||
|
||
## Scheduler
|
||
|
||
`steward/core/scheduler.py` exports two things:
|
||
|
||
- `ScheduledTask` dataclass — holds a `name`, `coro_factory` (zero-argument callable returning a coroutine), `interval_seconds`, and optional `run_on_startup` flag
|
||
- `start_scheduler(tasks)` — async function that loops every second, calling `asyncio.create_task()` for each task whose interval has elapsed
|
||
|
||
Tasks are never awaited serially — each runs as a fully independent asyncio task. Exceptions inside tasks are caught and logged; they do not crash other tasks or the app.
|
||
|
||
Core tasks registered in `app._task_registry`:
|
||
- `ping_monitor` — runs every `monitors.poll_interval_seconds` (default 60s), `run_on_startup=True`
|
||
- `dns_monitor` — same interval, `run_on_startup=True`
|
||
- `data_cleanup` — runs hourly, `run_on_startup=False`
|
||
- `ansible_git_pull_<name>` — one per configured Git source, interval from source config
|
||
|
||
---
|
||
|
||
## Metric Store
|
||
|
||
Every plugin and monitor records into one store keyed by (source, resource, metric):
|
||
- `plugin_metrics` holds raw samples for `metrics.retention.raw_days` (7 by default).
|
||
- `plugin_metric_latest` holds each key's newest value.
|
||
- `data_cleanup` rolls raw samples older than the raw window into `plugin_metrics_hourly`, kept for `metrics.retention.rollup_days` (90 by default).
|
||
|
||
Each rolled-up hour keeps three figures:
|
||
- `value_avg`: the hour's average.
|
||
- `value_max`: its peak.
|
||
- `value_p95`: the level it held for all but its briefest 5%, about 3 minutes.
|
||
|
||
A two-minute spike raises the peak and leaves the p95 low; an hour spent at that level raises both. Learned normals use this to tell a recurring spike from a held level. Hours rolled up before the p95 was kept have it NULL; read those as their average.
|
||
|
||
### Normals and anomalies
|
||
|
||
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
|
||
|
||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, opens an anomaly (`steward/core/anomalies.py`):
|
||
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
|
||
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
|
||
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
|
||
- `data_cleanup` prunes reviewed and lapsed rows at `data.retention_days`; a row still to review is never pruned.
|
||
|
||
---
|
||
|
||
## Database Session Pattern
|
||
|
||
`app.db_sessionmaker` is an SQLAlchemy `async_sessionmaker`. All DB access follows this pattern:
|
||
|
||
```python
|
||
async with current_app.db_sessionmaker() as session:
|
||
async with session.begin():
|
||
# queries and writes here
|
||
# transaction commits on exit, rolls back on exception
|
||
```
|
||
|
||
Sessions are never shared across request boundaries or between tasks.
|
||
|
||
---
|
||
|
||
## Config Two-Layer Design
|
||
|
||
There are exactly two config layers:
|
||
|
||
1. **Bootstrap** (`config.yaml` + env vars) — `database_url`, `secret_key`, `plugin_dir` only. Read once at startup before the event loop.
|
||
2. **App settings** (`app_settings` DB table) — everything else: SMTP, webhooks, Ansible sources, monitor intervals, ping thresholds, plugin config. Read at startup via `load_settings_sync()` and written via the Settings UI at runtime.
|
||
|
||
This means the only file you must touch to get the app running is the database URL. Everything else can be configured through the web UI.
|
||
|
||
---
|
||
|
||
## Frontend Approach
|
||
|
||
No JavaScript framework, no build step. The frontend is:
|
||
|
||
- **Jinja2 templates** rendered server-side (`steward/templates/`)
|
||
- **HTMX** for live-updating fragments (dashboard widgets, ping pills, DNS status)
|
||
- A single CSS design system in `steward/templates/base.html` using CSS custom properties
|
||
|
||
Live-updating widgets use HTMX polling:
|
||
```html
|
||
<div hx-get="/ping/rows" hx-trigger="load, every 30s" hx-swap="innerHTML">
|
||
```
|
||
|
||
Fragment endpoints (`/ping/rows`, `/dns/rows`, `/plugins/traefik/widget`) return partial HTML that HTMX swaps in without a full page reload.
|