perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)
The dashboard's host widgets took 18-60 s. Every "latest reading" query (hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host) searched plugin_metrics for the newest row per metric. Postgres has no skip scan, so each one read every raw sample in the 7-day window. - plugin_metric_latest: one row per (source, resource, metric). record_metrics upserts it in the same transaction, moving forward only and in sorted key order. Migration 0030 seeds it by stepping the existing index. - Retention drops a latest row once its sample leaves the raw window, so a resource that stopped reporting disappears when its history does, as before. Renames carry it along. - A quiet host keeps its last values, marked stale. A time-bounded query would have shown it as missing. - Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points, instead of fetching 6 h raw and drawing the last 120 samples. - Dashboard and share-view cells: "Loading…" until the first answer; a 20 s deadline per fetch; "Couldn't load (why)" with Retry now and an automatic retry for cells that don't poll; "as of HH:MM" in the header, and "couldn't refresh · as of" when a refresh fails over shown data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -8,7 +8,7 @@ Alert rules evaluate every metric that flows through `record_metric()`. There is
|
||||
|
||||
When any monitor or plugin calls `record_metric()`:
|
||||
|
||||
1. The metric value is written to the `plugin_metrics` table
|
||||
1. The metric value is written to the `plugin_metrics` table, and `plugin_metric_latest` moves to it if it is the newest sample of that metric
|
||||
2. All enabled `AlertRule` rows matching `(source_module, resource_name, metric_name)` are loaded
|
||||
3. Each matching rule is evaluated against the new value
|
||||
4. If a state transition occurs, an `AlertEvent` row is written
|
||||
@@ -136,3 +136,4 @@ Defined in `steward/models/alerts.py`:
|
||||
- **`alert_states`** — one row per rule, tracks current state and consecutive failure count
|
||||
- **`alert_events`** — append-only log of all state transitions and notification outcomes
|
||||
- **`plugin_metrics`** — all metric values written by any monitor or plugin
|
||||
- **`plugin_metric_latest`** — the newest sample of each (source, resource, metric). Pages that show "what it is now" read this instead of searching history; retention drops a row once its sample leaves the raw window
|
||||
|
||||
@@ -89,6 +89,7 @@ Quick reference for where key functions, models, and entry points live in the co
|
||||
| `AlertState` | `steward/models/alerts.py` | `alert_states` |
|
||||
| `AlertEvent` | `steward/models/alerts.py` | `alert_events` |
|
||||
| `PluginMetric` | `steward/models/metrics.py` | `plugin_metrics` |
|
||||
| `PluginMetricLatest` | `steward/models/metrics.py` | `plugin_metric_latest` |
|
||||
| `AnsibleRun` | `steward/models/ansible.py` | `ansible_runs` |
|
||||
| `User` | `steward/models/users.py` | `users` |
|
||||
| `AppSetting` | `steward/models/settings.py` | `app_settings` |
|
||||
|
||||
Reference in New Issue
Block a user