bvandeusen/FabledSteward
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: rename to FabledScryer, multi-dashboard system, plugin management, branding

Browse Source 
- Rename package fablednetmon → fabledscryer throughout
- Multi-dashboard: ownership, per-user defaults, HTMX edit (add/remove/reorder)
- Read-only share tokens scoped to individual dashboards
- Dashboard edit is HTMX-driven (no page reloads)
- Plugin management system: remote catalog, download/install, hot-reload, in-app restart
- plugin_index.py: fetch/cache remote index.yaml; default URL → bvandeusen/fabledscryer-plugins
- plugin_manager.py: download_and_install_plugin, hot_reload_plugin, restart_app
  - ZIP extraction handles GitHub archive formats (name-v1.0.0/, name-main/)
- Settings split into tabbed sections: General, Notifications, Ansible, Plugins
- Plugins tab: catalog browser (HTMX), install/activate/update/restart actions
- UI/branding: dark palette (#07071a), crystal ball SVG logo, animated star field,
  Libertinus Serif applied to headings, nav, labels, and section titles
- Widget registry (core/widgets.py) for dashboard plugin integration
- UPS widget.html (dashboard card) and settings/_tabs.html include
- Migrations 0005–0008: dashboards, is_default, ownership, share tokens
- docs/plugins/: writing-a-plugin.md updated with publishing guide,
  index.yaml.example template for fabledscryer-plugins repo

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-03-22 18:27:56 -04:00
parent 165a202ba4
commit 230b542015
121 changed files with 4820 additions and 715 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/core/alerting.md
+138
View File
@@ -0,0 +1,138 @@
# Alerting
Alert rules evaluate every metric that flows through `record_metric()`. There is no separate polling process — evaluation is inline with every metric write.
---
## How It Works
When any monitor or plugin calls `record_metric()`:
1. The metric value is written to the `plugin_metrics` table
2. All enabled `AlertRule` rows matching `(source_module, resource_name, metric_name)` are loaded
3. Each matching rule is evaluated against the new value
4. If a state transition occurs, an `AlertEvent` row is written
5. Notification I/O is deferred outside the transaction via `asyncio.create_task()`
**Key function:** `fabledscryer/core/alerts.py` → `record_metric(session, source_module, resource_name, metric_name, value)`
`record_metric()` must always be called inside an active transaction:
```python
async with session.begin():
await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)
```
Notifications are sent via `asyncio.create_task()` after the function returns, so no network I/O blocks the DB transaction.
---
## Alert State Machine
Each `AlertRule` has one associated `AlertState` row. The state transitions are:
```
inactive ──(breached)──► pending ──(consecutive count met)──► FIRING
│ │
└──(recovered)──► inactive (no notification) │
│
FIRING ──(recovered)──► RESOLVED ──► inactive
FIRING ──(acknowledged)──► ACKNOWLEDGED
ACKNOWLEDGED ──(recovered)──► RESOLVED ──► inactive
ACKNOWLEDGED ──(re-breached)──► FIRING
```
`RESOLVED` is transient — the evaluator writes the event and immediately sets state back to `inactive` within the same transaction. `RESOLVED` never persists as a final state in the DB.
### State Definitions
| State | Meaning |
|---|---|
| `inactive` | Threshold not breached |
| `pending` | Threshold breached but consecutive count not yet met; no notification sent |
| `firing` | Consecutive count met; FIRING notification sent |
| `acknowledged` | Operator acknowledged; suppresses repeat notifications; auto-clears on recovery |
| `resolved` | Transient — notification sent, immediately transitions to `inactive` |
---
## Creating Alert Rules
Alert rules are created in the UI at `/alerts/`. Each rule requires:
- **Name** — human-readable label
- **Source module** — `ping`, `dns`, `traefik`, or any plugin name
- **Resource name** — host name, router name, etc. (must match exactly what the monitor writes)
- **Metric name** — the metric key (e.g. `response_time_ms`, `up`, `error_rate_5xx_pct`)
- **Operator** — `>`, `<`, `>=`, `<=`, `==`, `!=`
- **Threshold** — numeric value
- **Consecutive failures required** — how many consecutive breaches before FIRING (default 1)
### Available Metrics by Module
| `source_module` | `metric_name` | Description |
|---|---|---|
| `ping` | `response_time_ms` | Probe latency (0.0 if down) |
| `ping` | `up` | 1.0 = up, 0.0 = down |
| `dns` | `resolved` | 1.0 = resolved, 0.0 = failed |
| `dns` | `ip_changed` | 1.0 = IP changed from previous result |
| `traefik` | `request_rate` | Requests per second |
| `traefik` | `error_rate_4xx_pct` | 4xx errors as % of requests |
| `traefik` | `error_rate_5xx_pct` | 5xx errors as % of requests |
| `traefik` | `latency_p50_ms` | Approximate p50 latency (ms) |
| `traefik` | `latency_p95_ms` | Approximate p95 latency (ms) |
| `traefik` | `latency_p99_ms` | Approximate p99 latency (ms) |
---
## Notifications
Notifications fire on `FIRING` and `RESOLVED` transitions. All configured channels receive every notification; there is no per-rule channel routing.
### Email
Configured at `/settings/` under SMTP. Settings:
| Setting key | Description |
|---|---|
| `smtp.host` | SMTP server |
| `smtp.port` | Port (default 587) |
| `smtp.tls` | STARTTLS (default true) |
| `smtp.username` | Login |
| `smtp.password` | Password |
| `smtp.recipients` | List of email addresses |
Email is skipped if `smtp.host` is empty.
### Webhook
Configured at `/settings/` under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always `application/json`. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.
| Template variable | Type | Description |
|---|---|---|
| `alert.rule_name` | str | Alert rule name |
| `alert.state` | str | `FIRING` or `RESOLVED` |
| `alert.metric` | str | Metric name |
| `alert.value` | float | Current value |
| `alert.threshold` | float | Configured threshold |
| `alert.resource` | str | Resource name |
| `alert.source_module` | str | `ping`, `dns`, `traefik`, etc. |
| `alert.timestamp` | str | ISO 8601 UTC |
Default template (Discord-compatible):
```json
{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}
```
Webhook is skipped if `webhook.url` is empty.
---
## Data Models
Defined in `fabledscryer/models/alerts.py`:
- **`alert_rules`** — one row per configured rule
- **`alert_states`** — one row per rule, tracks current state and consecutive failure count
- **`alert_events`** — append-only log of all state transitions and notification outcomes
- **`plugin_metrics`** — all metric values written by any monitor or plugin
docs/core/ansible.md
+143
View File
@@ -0,0 +1,143 @@
# Ansible Integration
Fabled Scryer can browse, trigger, and stream output from Ansible playbooks directly from the web UI. Runs execute as asyncio tasks inside the same process — no Celery, no external workers.
---
## Playbook Sources
Sources are configured at `/settings/` under Ansible. Two source types are supported:
### Local Filesystem
Points to a directory on the host that already contains playbooks.
```yaml
# In app_settings (configured via UI)
ansible.sources:
- name: "homelab"
type: local
path: "/opt/playbooks"
```
### Git Repository
The app clones or pulls the repo into a local cache directory on a configurable schedule. All execution uses the local cache.
```yaml
ansible.sources:
- name: "infra-repo"
type: git
url: "https://github.com/example/infra.git"
branch: "main"
pull_interval_seconds: 300
cache_path: "/data/playbook_cache/infra-repo"
```
Git sources register a `ScheduledTask` named `ansible_git_pull_<name>` that runs `git pull` on the configured interval.
---
## Inventory Discovery
The app discovers inventory files within the root of the playbook source directory (non-recursive). It looks for files named:
- `hosts`
- `inventory`
- `inventory.yml`
- `inventory.ini`
A manual relative path can also be entered in the UI (e.g. `inventories/production/hosts`) for inventories in subdirectories.
---
## Triggering a Run
From the UI at `/ansible/`, you can:
1. Browse available playbooks across all sources
2. View playbook contents before running
3. Select an inventory file
4. Trigger a run (requires `operator` or `viewer` role — execution is restricted to `operator`/`admin`)
The run flow:
1. UI submits `POST /ansible/runs` with `playbook_path`, `source_name`, and `inventory_path`
2. An `AnsibleRun` row is created with `status = running`
3. Playbook execution starts as `asyncio.create_task()`
4. The response returns the `run_id` and an HTMX partial that wires the SSE subscription
5. The browser connects to `GET /ansible/runs/<run_id>/stream` to receive live output
---
## SSE Streaming
Run output is streamed to the browser via Server-Sent Events (SSE) at:
```
GET /ansible/runs/<run_id>/stream
```
Each output line is sent as:
```
event: output
data: <line of stdout/stderr>
```
Run completion:
```
event: done
data: success|failed|interrupted
```
Output is flushed to the `ansible_runs.output` DB column every 50 lines or every 5 seconds (whichever comes first) and always on completion. This means partial output survives a process crash.
If a client connects after the run has already completed, the endpoint immediately sends `event: done` with the final status and closes the stream. To view stored output, use the run history view at `GET /ansible/runs/<run_id>`.
---
## Run Lifecycle
| Status | Description |
|---|---|
| `running` | Execution in progress |
| `success` | Playbook exited 0 |
| `failed` | Playbook exited non-zero |
| `interrupted` | App restarted while run was in progress |
On startup, the app marks any runs still in `running` state as `interrupted` (see `_mark_interrupted_runs()` in `app.py`).
Output stored in the DB is capped at 1 MB. If truncated, `[output truncated]` is appended to the DB column, but the live SSE stream continues unaffected.
---
## Data Model
`ansible_runs` table (defined in `fabledscryer/models/ansible.py`):
| Column | Type | Description |
|---|---|---|
| `id` | UUID | Primary key |
| `playbook_path` | str | Relative path to playbook |
| `inventory_path` | str | Inventory path used |
| `source_name` | str | Name of the playbook source |
| `triggered_by` | FK → users | |
| `status` | enum | `running`, `success`, `failed`, `interrupted` |
| `started_at` | timestamp UTC | |
| `finished_at` | timestamp UTC | Null if still running |
| `output` | text | Captured stdout/stderr (capped at 1 MB) |
Runs older than `data.retention_days` (default 90) are pruned by the `data_cleanup` scheduled task.
---
## Source Files
| File | Purpose |
|---|---|
| `fabledscryer/ansible/sources.py` | Source discovery, git pull logic |
| `fabledscryer/ansible/executor.py` | Subprocess execution and output streaming |
| `fabledscryer/ansible/routes.py` | HTTP routes (browse, trigger, stream, history) |
| `fabledscryer/models/ansible.py` | `AnsibleRun` model |
docs/core/configuration.md
+103
View File
@@ -0,0 +1,103 @@
# Configuration
Fabled Scryer uses a two-layer configuration system. Only the bare minimum needed to boot lives in files or environment variables. Everything else is stored in the database and managed through the Settings UI.
---
## Bootstrap Config (File / Env Vars)
These three values are read at startup from `config.yaml` and/or environment variables:
| Key | Env var | Default | Description |
|---|---|---|---|
| `database.url` | `FABLEDSCRYER_DATABASE_URL` | — | PostgreSQL async URL. **Required.** |
| `secret_key` | `FABLEDSCRYER_SECRET_KEY` | auto-generated | Flask/Quart session signing key. Auto-generated and saved to `/data/secret.key` if not set. |
| `plugin_dir` | `FABLEDSCRYER_PLUGIN_DIR` | `plugins` | Path to the plugins directory. |
**Resolution order for `database_url`:** env var `FABLEDSCRYER_DATABASE_URL` → env var `FABLEDSCRYER_DATABASE__URL` (legacy double-underscore) → `database.url` in `config.yaml`.
**Resolution order for `secret_key`:** env var `FABLEDSCRYER_SECRET_KEY` → `secret_key` in `config.yaml` → `/data/secret.key` file → auto-generate and write to `/data/secret.key`.
### Minimal config.yaml
```yaml
database:
url: "postgresql+asyncpg://user:password@localhost/fabledscryer"
```
### Minimal env-only setup (Docker)
```bash
FABLEDSCRYER_DATABASE_URL=postgresql+asyncpg://user:password@db/fabledscryer
```
A `.env` file is loaded automatically if present.
---
## App Settings (Database-backed)
All runtime settings are stored in the `app_settings` table and editable through the web UI at `/settings/`. They are loaded into `app.config` at startup.
### All Settings and Defaults
| Key | Default | Description |
|---|---|---|
| `session.lifetime_hours` | `8` | How long a login session lasts |
| `data.retention_days` | `90` | How many days of ping/DNS/metric/ansible history to keep |
| `monitors.poll_interval_seconds` | `60` | How often ping and DNS checks run |
| `smtp.host` | `""` | SMTP server hostname |
| `smtp.port` | `587` | SMTP server port |
| `smtp.tls` | `true` | Use STARTTLS |
| `smtp.username` | `""` | SMTP login username |
| `smtp.password` | `""` | SMTP login password |
| `smtp.recipients` | `[]` | List of email addresses to notify |
| `webhook.url` | `""` | Webhook POST destination URL |
| `webhook.template` | see below | Jinja2 JSON template for webhook body |
| `ansible.sources` | `[]` | List of playbook source definitions |
| `ping.threshold.good_ms` | `50` | Latency below this is shown green in the ping UI |
| `ping.threshold.warn_ms` | `200` | Latency below this is shown yellow; above is orange |
Default webhook template:
```
{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}
```
### Reading and Writing Settings in Code
```python
from fabledscryer.core.settings import get_setting, set_setting
# Read
async with current_app.db_sessionmaker() as session:
host = await get_setting(session, "smtp.host")
# Write (must be inside a transaction)
async with current_app.db_sessionmaker() as session:
async with session.begin():
await set_setting(session, "smtp.host", "mail.example.com")
```
`get_setting()` returns the stored value or the default from `DEFAULTS` if the key has never been set.
### The DEFAULTS Dict
`fabledscryer/core/settings.py` contains the `DEFAULTS` dict — the canonical list of all recognised settings and their default values. Add new settings here to make them recognised by `get_all_settings()` and the settings UI.
---
## app.config Keys
After startup, `app.config` contains these additional keys (in addition to standard Quart keys):
| Key | Type | Source |
|---|---|---|
| `DATABASE_URL` | str | Bootstrap |
| `PLUGIN_DIR` | str | Bootstrap |
| `SESSION_LIFETIME_HOURS` | int | DB settings |
| `DATA_RETENTION_DAYS` | int | DB settings |
| `MONITORS_POLL_INTERVAL` | int | DB settings |
| `SMTP` | dict | DB settings, via `to_smtp_cfg()` |
| `WEBHOOK` | dict | DB settings, via `to_webhook_cfg()` |
| `ANSIBLE` | dict | DB settings, via `to_ansible_cfg()` |
| `PLUGINS` | dict | DB settings + plugin.yaml defaults, keyed by plugin name |
docs/core/monitors.md
+104
View File
@@ -0,0 +1,104 @@
# Core Monitors
Fabled Scryer ships two built-in monitors: Ping and DNS. Both run as asyncio scheduled tasks on the same event loop as the web server, with no separate processes.
---
## Ping Monitor
**Source:** `fabledscryer/monitors/ping.py`
**Scheduler task:** `ping_monitor` in `fabledscryer/app.py`
**Interval:** `monitors.poll_interval_seconds` (default 60s), runs on startup
### How It Works
On each tick, the scheduler fetches all hosts with `ping_enabled = true` and calls `ping_check(host, session)` for each.
`ping_check()` probes the host using:
- **ICMP** if `host.probe_type == "icmp"` — uses the system `ping` binary (`/bin/ping` or equivalent). Requires `iputils-ping` in Docker.
- **TCP** if `host.probe_type == "tcp"` (default) — attempts an async TCP connection to `host.address:host.probe_port` (default port 80).
Each probe writes a `PingResult` row and calls `record_metric()`:
| `source_module` | `resource_name` | `metric_name` | `value` |
|---|---|---|---|
| `ping` | `host.name` | `response_time_ms` | measured latency, or `0.0` if down |
| `ping` | `host.name` | `up` | `1.0` if up, `0.0` if down |
**Alert rule note:** Because `response_time_ms` is recorded as `0.0` when a host is down, a latency rule (e.g. `response_time_ms > 500`) will not fire on complete outages. Use a separate rule on `up == 0.0` to detect host down events.
### Data Model
`ping_results` table (defined in `fabledscryer/models/monitors.py`):
| Column | Type | Description |
|---|---|---|
| `id` | UUID | Primary key |
| `host_id` | FK → hosts | |
| `probed_at` | timestamp UTC | When the probe ran |
| `status` | enum `up`/`down` | Result |
| `response_time_ms` | float | Null if down |
Old rows are pruned by the `data_cleanup` task (default: 90 days).
### UI
- **Dashboard widget** — live-updating via HTMX polling (`/ping/rows`)
- **`/ping/` page** — full page with 30-pill history per host and threshold settings form
- **Hosts list** — shows latest ping status dot and latency
---
## DNS Monitor
**Source:** `fabledscryer/monitors/dns.py`
**Scheduler task:** `dns_monitor` in `fabledscryer/app.py`
**Interval:** `monitors.poll_interval_seconds` (default 60s), runs on startup
### How It Works
On each tick, the scheduler fetches all hosts with `dns_enabled = true` and calls `dns_check(host, session)` for each.
`dns_check()` resolves `host.address` using the system resolver. If `host.dns_expected_ip` is set, the check passes only if at least one returned A/AAAA record exactly matches that string. If `dns_expected_ip` is null, any successful resolution counts as a pass.
Each check writes a `DnsResult` row and calls `record_metric()`:
| `source_module` | `resource_name` | `metric_name` | `value` |
|---|---|---|---|
| `dns` | `host.name` | `resolved` | `1.0` if resolved, `0.0` if failed |
| `dns` | `host.name` | `ip_changed` | `1.0` if IP changed from last successful result, `0.0` otherwise |
### Data Model
`dns_results` table (defined in `fabledscryer/models/monitors.py`):
| Column | Type | Description |
|---|---|---|
| `id` | UUID | Primary key |
| `host_id` | FK → hosts | |
| `resolved_at` | timestamp UTC | When the check ran |
| `status` | enum `resolved`/`failed` | Result |
| `resolved_ip` | str | First returned A/AAAA record; null if failed |
`ip_changed` is computed by comparing `resolved_ip` of the current result against the most recent prior `resolved` result for the same host.
### UI
- **Dashboard widget** — live-updating via HTMX polling (`/dns/rows`)
- **`/dns/` page** — full page showing all DNS-enabled hosts with status, resolved IP, and timestamp
- **Hosts list** — shows latest DNS status dot
---
## Host Configuration
Hosts are managed at `/hosts/`. Both monitors are configured per-host:
| Field | Description |
|---|---|
| `ping_enabled` | Enable ping probing for this host |
| `probe_type` | `tcp` (default) or `icmp` |
| `probe_port` | TCP port to connect to (default 80; ignored for ICMP) |
| `dns_enabled` | Enable DNS resolution checks |
| `dns_expected_ip` | If set, the resolved IP must match this string exactly |
| `poll_interval_seconds` | Per-host override for the global poll interval; null uses global |