feat: rename to FabledScryer, multi-dashboard system, plugin management, branding
- Rename package fablednetmon → fabledscryer throughout - Multi-dashboard: ownership, per-user defaults, HTMX edit (add/remove/reorder) - Read-only share tokens scoped to individual dashboards - Dashboard edit is HTMX-driven (no page reloads) - Plugin management system: remote catalog, download/install, hot-reload, in-app restart - plugin_index.py: fetch/cache remote index.yaml; default URL → bvandeusen/fabledscryer-plugins - plugin_manager.py: download_and_install_plugin, hot_reload_plugin, restart_app - ZIP extraction handles GitHub archive formats (name-v1.0.0/, name-main/) - Settings split into tabbed sections: General, Notifications, Ansible, Plugins - Plugins tab: catalog browser (HTMX), install/activate/update/restart actions - UI/branding: dark palette (#07071a), crystal ball SVG logo, animated star field, Libertinus Serif applied to headings, nav, labels, and section titles - Widget registry (core/widgets.py) for dashboard plugin integration - UPS widget.html (dashboard card) and settings/_tabs.html include - Migrations 0005–0008: dashboards, is_default, ownership, share tokens - docs/plugins/: writing-a-plugin.md updated with publishing guide, index.yaml.example template for fabledscryer-plugins repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,138 @@
|
||||
# Alerting
|
||||
|
||||
Alert rules evaluate every metric that flows through `record_metric()`. There is no separate polling process — evaluation is inline with every metric write.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
When any monitor or plugin calls `record_metric()`:
|
||||
|
||||
1. The metric value is written to the `plugin_metrics` table
|
||||
2. All enabled `AlertRule` rows matching `(source_module, resource_name, metric_name)` are loaded
|
||||
3. Each matching rule is evaluated against the new value
|
||||
4. If a state transition occurs, an `AlertEvent` row is written
|
||||
5. Notification I/O is deferred outside the transaction via `asyncio.create_task()`
|
||||
|
||||
**Key function:** `fabledscryer/core/alerts.py` → `record_metric(session, source_module, resource_name, metric_name, value)`
|
||||
|
||||
`record_metric()` must always be called inside an active transaction:
|
||||
|
||||
```python
|
||||
async with session.begin():
|
||||
await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)
|
||||
```
|
||||
|
||||
Notifications are sent via `asyncio.create_task()` after the function returns, so no network I/O blocks the DB transaction.
|
||||
|
||||
---
|
||||
|
||||
## Alert State Machine
|
||||
|
||||
Each `AlertRule` has one associated `AlertState` row. The state transitions are:
|
||||
|
||||
```
|
||||
inactive ──(breached)──► pending ──(consecutive count met)──► FIRING
|
||||
│ │
|
||||
└──(recovered)──► inactive (no notification) │
|
||||
│
|
||||
FIRING ──(recovered)──► RESOLVED ──► inactive
|
||||
FIRING ──(acknowledged)──► ACKNOWLEDGED
|
||||
ACKNOWLEDGED ──(recovered)──► RESOLVED ──► inactive
|
||||
ACKNOWLEDGED ──(re-breached)──► FIRING
|
||||
```
|
||||
|
||||
`RESOLVED` is transient — the evaluator writes the event and immediately sets state back to `inactive` within the same transaction. `RESOLVED` never persists as a final state in the DB.
|
||||
|
||||
### State Definitions
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `inactive` | Threshold not breached |
|
||||
| `pending` | Threshold breached but consecutive count not yet met; no notification sent |
|
||||
| `firing` | Consecutive count met; FIRING notification sent |
|
||||
| `acknowledged` | Operator acknowledged; suppresses repeat notifications; auto-clears on recovery |
|
||||
| `resolved` | Transient — notification sent, immediately transitions to `inactive` |
|
||||
|
||||
---
|
||||
|
||||
## Creating Alert Rules
|
||||
|
||||
Alert rules are created in the UI at `/alerts/`. Each rule requires:
|
||||
|
||||
- **Name** — human-readable label
|
||||
- **Source module** — `ping`, `dns`, `traefik`, or any plugin name
|
||||
- **Resource name** — host name, router name, etc. (must match exactly what the monitor writes)
|
||||
- **Metric name** — the metric key (e.g. `response_time_ms`, `up`, `error_rate_5xx_pct`)
|
||||
- **Operator** — `>`, `<`, `>=`, `<=`, `==`, `!=`
|
||||
- **Threshold** — numeric value
|
||||
- **Consecutive failures required** — how many consecutive breaches before FIRING (default 1)
|
||||
|
||||
### Available Metrics by Module
|
||||
|
||||
| `source_module` | `metric_name` | Description |
|
||||
|---|---|---|
|
||||
| `ping` | `response_time_ms` | Probe latency (0.0 if down) |
|
||||
| `ping` | `up` | 1.0 = up, 0.0 = down |
|
||||
| `dns` | `resolved` | 1.0 = resolved, 0.0 = failed |
|
||||
| `dns` | `ip_changed` | 1.0 = IP changed from previous result |
|
||||
| `traefik` | `request_rate` | Requests per second |
|
||||
| `traefik` | `error_rate_4xx_pct` | 4xx errors as % of requests |
|
||||
| `traefik` | `error_rate_5xx_pct` | 5xx errors as % of requests |
|
||||
| `traefik` | `latency_p50_ms` | Approximate p50 latency (ms) |
|
||||
| `traefik` | `latency_p95_ms` | Approximate p95 latency (ms) |
|
||||
| `traefik` | `latency_p99_ms` | Approximate p99 latency (ms) |
|
||||
|
||||
---
|
||||
|
||||
## Notifications
|
||||
|
||||
Notifications fire on `FIRING` and `RESOLVED` transitions. All configured channels receive every notification; there is no per-rule channel routing.
|
||||
|
||||
### Email
|
||||
|
||||
Configured at `/settings/` under SMTP. Settings:
|
||||
|
||||
| Setting key | Description |
|
||||
|---|---|
|
||||
| `smtp.host` | SMTP server |
|
||||
| `smtp.port` | Port (default 587) |
|
||||
| `smtp.tls` | STARTTLS (default true) |
|
||||
| `smtp.username` | Login |
|
||||
| `smtp.password` | Password |
|
||||
| `smtp.recipients` | List of email addresses |
|
||||
|
||||
Email is skipped if `smtp.host` is empty.
|
||||
|
||||
### Webhook
|
||||
|
||||
Configured at `/settings/` under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always `application/json`. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.
|
||||
|
||||
| Template variable | Type | Description |
|
||||
|---|---|---|
|
||||
| `alert.rule_name` | str | Alert rule name |
|
||||
| `alert.state` | str | `FIRING` or `RESOLVED` |
|
||||
| `alert.metric` | str | Metric name |
|
||||
| `alert.value` | float | Current value |
|
||||
| `alert.threshold` | float | Configured threshold |
|
||||
| `alert.resource` | str | Resource name |
|
||||
| `alert.source_module` | str | `ping`, `dns`, `traefik`, etc. |
|
||||
| `alert.timestamp` | str | ISO 8601 UTC |
|
||||
|
||||
Default template (Discord-compatible):
|
||||
```json
|
||||
{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}
|
||||
```
|
||||
|
||||
Webhook is skipped if `webhook.url` is empty.
|
||||
|
||||
---
|
||||
|
||||
## Data Models
|
||||
|
||||
Defined in `fabledscryer/models/alerts.py`:
|
||||
|
||||
- **`alert_rules`** — one row per configured rule
|
||||
- **`alert_states`** — one row per rule, tracks current state and consecutive failure count
|
||||
- **`alert_events`** — append-only log of all state transitions and notification outcomes
|
||||
- **`plugin_metrics`** — all metric values written by any monitor or plugin
|
||||
Reference in New Issue
Block a user