perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m14s
CI / unit (push) Successful in 1m33s
CI / publish (push) Skipped

The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.

- plugin_metric_latest: one row per (source, resource, metric).
  record_metrics upserts it in the same transaction, moving forward only and
  in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
  resource that stopped reporting disappears when its history does, as before.
  Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
  would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
  instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
  deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
  retry for cells that don't poll; "as of HH:MM" in the header, and
  "couldn't refresh · as of" when a refresh fails over shown data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 23:06:34 -04:00
co-authored by Claude Opus 5.5
parent ba140ab109
commit b8f73270e7
20 changed files with 487 additions and 202 deletions
+7 -23
View File
@@ -9,7 +9,7 @@ from __future__ import annotations
from datetime import datetime, timedelta, timezone
from sqlalchemy import func, or_, select
from sqlalchemy import func, select
from steward.core.time_range import bucket_seconds
from steward.models.metrics import PluginMetric, PluginMetricHourly
@@ -27,30 +27,14 @@ HISTORY_METRICS = (
async def _latest_metrics_for_host(session, host_name: str) -> dict[str, dict[str, float]]:
"""{resource_name: {metric: value}} — latest sample per (resource, metric) for
a host + its sub-resources.
DISTINCT ON picks the newest row per group in one index-ordered pass over
ix_plugin_metrics_module_resource_metric_recorded, instead of a GROUP-BY-max
subquery self-joined back to the table (two passes over the whole history).
a host + its sub-resources, from plugin_metric_latest (#5208). A host whose
agent has gone quiet keeps its last values until they age out of the raw
window; the caller decides staleness from the registration's last_seen_at.
"""
rows = (await session.execute(
select(PluginMetric)
.where(
PluginMetric.source_module == SOURCE_MODULE,
or_(
PluginMetric.resource_name == host_name,
PluginMetric.resource_name.like(host_name + ":%"),
),
)
.distinct(PluginMetric.resource_name, PluginMetric.metric_name)
.order_by(
PluginMetric.resource_name,
PluginMetric.metric_name,
PluginMetric.recorded_at.desc(),
)
)).scalars().all()
from steward.core.metric_series import latest_values
out: dict[str, dict[str, float]] = {}
for r in rows:
for r in await latest_values(session, SOURCE_MODULE, host_name, include_children=True):
out.setdefault(r.resource_name, {})[r.metric_name] = r.value
return out