FabledSteward/docs/architecture.md

Architecture

Steward is a single Quart (async Python) process. There are no separate worker processes, no message brokers, and no build pipeline for the frontend. All monitoring, scheduling, and request handling runs on a single asyncio event loop.

---

Startup Sequence

create_app() in steward/app.py runs these steps synchronously before the event loop starts:

Step	What happens
1	Bootstrap config loaded (database_url, secret_key, plugin_dir)
2	SQLAlchemy async engine and session factory attached to app
3	Core Alembic migrations applied (creates app_settings table among others)
4	All settings loaded from app_settings DB table into app.config
5	Each enabled plugin's Alembic migrations applied
6	Alert pipeline initialised (init_alerts(app) stores app ref for deferred notifications)
7	Core blueprints registered (auth, dashboard, hosts, ping, dns, alerts, ansible, settings)
8	Core scheduled tasks registered into app._task_registry
9	Plugins loaded via load_plugins(app) — blueprints and tasks appended
10	/health endpoint registered
11	before_serving hook starts the scheduler as an asyncio.create_task()

The two-phase migration approach (step 3 core → step 4 load settings → step 5 plugins) exists because plugin enabling is stored in the settings DB, which requires the core schema to exist first.

---

Request Routing

All routes are Quart Blueprints registered in app.py:

Prefix	Blueprint	Module
/auth/	auth_bp	steward/auth/routes.py
/	dashboard_bp	steward/dashboard/routes.py
/hosts/	hosts_bp	steward/hosts/routes.py
/ping/	ping_bp	steward/ping/routes.py
/dns/	dns_bp	steward/dns/routes.py
/alerts/	alerts_bp	steward/alerts/routes.py
/ansible/	ansible_bp	steward/ansible/routes.py
/settings/	settings_bp	steward/settings/routes.py
/plugins/<name>/	plugin blueprint	plugins/<name>/routes.py

Plugin blueprints are mounted automatically by load_plugins() using the plugin directory name as the URL prefix.

/ai-access/ (ai_access_bp, steward/ai_access/routes.py) is where a signed-in user manages their API tokens.

/mcp: outside the blueprint pipeline

mount_mcp(app) (steward/mcp/server.py) wraps app.asgi_app. A request to /mcp is handed to the MCP SDK's streamable-HTTP app before Quart sees it, so no blueprint, session cookie or before_request hook runs for it. Each request then goes through these steps:

1. Database gate. If the database is unreachable, the request gets a 503 with a readable message.
2. Bearer resolution. api_tokens.resolve_token() turns the token into a Caller (user, effective role, token id). A missing, revoked or expired token gets a 401.
3. Caller context. The Caller and the app are set in ContextVars (steward/mcp/_context.py). Tools read them through current_caller() and mcp_app(), because Quart's current_app and session do not exist there.
4. Role gate. StewardMCPServer.refusal() hides and refuses tools above the caller's role, and plugin tools whose plugin is disabled.

The SDK's session manager runs in Quart's before_serving/after_serving lifecycle. The server is stateless: no MCP session outlives a request. See MCP server for the security model and the tools.

---

Scheduler

steward/core/scheduler.py exports two things:

• ScheduledTask dataclass — holds a name, coro_factory (zero-argument callable returning a coroutine), interval_seconds, and optional run_on_startup flag
• start_scheduler(tasks) — async function that loops every second, calling asyncio.create_task() for each task whose interval has elapsed

Tasks are never awaited serially — each runs as a fully independent asyncio task. Exceptions inside tasks are caught and logged; they do not crash other tasks or the app.

Core tasks registered in app._task_registry:

• ping_monitor — runs every monitors.poll_interval_seconds (default 60s), run_on_startup=True
• dns_monitor — same interval, run_on_startup=True
• data_cleanup — runs hourly, run_on_startup=False
• ansible_git_pull_<name> — one per configured Git source, interval from source config

---

Metric Store

Every plugin and monitor records into one store keyed by (source, resource, metric):

• plugin_metrics holds raw samples for metrics.retention.raw_days (7 by default).
• plugin_metric_latest holds each key's newest value.
• data_cleanup rolls raw samples older than the raw window into plugin_metrics_hourly, kept for metrics.retention.rollup_days (90 by default).

Each rolled-up hour keeps three figures:

• value_avg: the hour's average.
• value_max: its peak.
• value_p95: the level it held for all but its briefest 5%, about 3 minutes.

A two-minute spike raises the peak and leaves the p95 low; an hour spent at that level raises both. Learned normals use this to tell a recurring spike from a held level. Hours rolled up before the p95 was kept have it NULL; read those as their average.

Normals and anomalies

A metric a plugin registers with learn_normal() gets a normal per hour of the day, then per hour of the week once three weeks of history exist (metric_norms, relearned hourly by metric_norms; steward/core/norms.py).

Every minute, metric_anomalies averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, opens an anomaly (steward/core/anomalies.py):

• metric_anomalies holds one row per item to review. A repeat while the row still waits joins it as another occurrence (metric_anomaly_spans).
• A row is to_review until someone marks it normal or real (Alerts → Anomalies, or MCP review_anomaly). An ended row nobody reviews within anomalies.review_days (14) becomes lapsed.
• Learning skips every hour an occurrence covers unless its anomaly was marked normal, so a new level is never learned without review.
• data_cleanup prunes reviewed and lapsed rows at data.retention_days; a row still to review is never pruned.

What learns a normal, and the floor each departure must clear (Settings → Thresholds → Normals, norms.floor.*; a floor may be a function, floor_setting(), read at each pass):

Kind	Metric	Floor (default)
Host CPU	host_agent / cpu_pct	10 points
Host memory	host_agent / mem_used_pct	5 points
Host load per core	host_agent / load_pct (load_1m ÷ cores × 100, recorded at ingest)	10 points
Container CPU	docker / cpu_pct (100% is one core)	10 points of one core
Container memory	docker / mem_pct (of its limit, or of host RAM when it has none)	5 points
Monitor response time	<type> / response_ms	100 ms
Router 4xx share, p95	traefik / error_rate_4xx_pct, latency_p95_ms	2 points, 50 ms

Host metrics learn with parts=False: the host learns, its cores (<host>:core<N>) don't. Disk space, a monitor that is down, an agent gone quiet and a router's 5xx share are failures and keep their fixed cutoffs (a router's shares are judged only over enough requests).

Pages grade by the normal through anomalies.graded() (hosts' host_grades and worst_reading(normals=), docker.service.usage(thresholds=, norms=), monitors.service.latency_grade): a reading past its cutoff but within its normal reads healthy and says what is usual at this hour. The host, monitor, container and Traefik pages list their anomalies with _anomaly_card.html.

---

Database Session Pattern

app.db_sessionmaker is an SQLAlchemy async_sessionmaker. All DB access follows this pattern:

async with current_app.db_sessionmaker() as session:
    async with session.begin():
        # queries and writes here
        # transaction commits on exit, rolls back on exception

Sessions are never shared across request boundaries or between tasks.

---

Config Two-Layer Design

There are exactly two config layers:

1. Bootstrap (config.yaml + env vars) — database_url, secret_key, plugin_dir only. Read once at startup before the event loop.
2. App settings (app_settings DB table) — everything else: SMTP, webhooks, Ansible sources, monitor intervals, ping thresholds, plugin config. Read at startup via load_settings_sync() and written via the Settings UI at runtime.

This means the only file you must touch to get the app running is the database URL. Everything else can be configured through the web UI.

---

Frontend Approach

No JavaScript framework, no build step. The frontend is:

• Jinja2 templates rendered server-side (steward/templates/)
• HTMX for live-updating fragments (dashboard widgets, ping pills, DNS status)
• A single CSS design system in steward/templates/base.html using CSS custom properties

Live-updating widgets use HTMX polling:

<div hx-get="/ping/rows" hx-trigger="load, every 30s" hx-swap="innerHTML">

Fragment endpoints (/ping/rows, /dns/rows, /plugins/traefik/widget) return partial HTML that HTMX swaps in without a full page reload.