perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)

The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.

- plugin_metric_latest: one row per (source, resource, metric).
  record_metrics upserts it in the same transaction, moving forward only and
  in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
  resource that stopped reporting disappears when its history does, as before.
  Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
  would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
  instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
  deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
  retry for cells that don't poll; "as of HH:MM" in the header, and
  "couldn't refresh · as of" when a refresh fails over shown data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/core/alerts.py

@@ -18,7 +18,7 @@ from steward.models.alerts import (
     MaintenanceWindow,
 )
 from steward.core.formatting import human_duration
-from steward.models.metrics import PluginMetric
+from steward.models.metrics import PluginMetric, PluginMetricLatest
 
 if TYPE_CHECKING:
     from quart import Quart
@@ -155,6 +155,34 @@ def _after_commit(session: AsyncSession, fn, *args) -> None:
     info[_AFTER_COMMIT].append((fn, args))
 
 
+LATEST_CHUNK = 1000
+
+
+async def _keep_latest(session: AsyncSession, rows: list[PluginMetric]) -> None:
+    """Move each key's row in plugin_metric_latest to the newest sample in
+    `rows`. A row only moves forward, so a backlog arriving after newer samples
+    leaves it alone. Keys go in sorted order, so two batches sharing keys lock
+    them in the same order and cannot deadlock."""
+    from sqlalchemy.dialects.postgresql import insert as pg_insert
+
+    newest: dict[tuple[str, str, str], PluginMetric] = {}
+    for row in rows:
+        key = (row.source_module, row.resource_name, row.metric_name)
+        if key not in newest or row.recorded_at > newest[key].recorded_at:
+            newest[key] = row
+    values = [{"source_module": k[0], "resource_name": k[1], "metric_name": k[2],
+               "value": newest[k].value, "recorded_at": newest[k].recorded_at}
+              for k in sorted(newest)]
+    # Five parameters a row; chunks keep a statement well under asyncpg's 32767.
+    for i in range(0, len(values), LATEST_CHUNK):
+        stmt = pg_insert(PluginMetricLatest).values(values[i:i + LATEST_CHUNK])
+        await session.execute(stmt.on_conflict_do_update(
+            index_elements=["source_module", "resource_name", "metric_name"],
+            set_={"value": stmt.excluded.value, "recorded_at": stmt.excluded.recorded_at},
+            where=PluginMetricLatest.recorded_at <= stmt.excluded.recorded_at,
+        ))
+
+
 async def record_metrics(session: AsyncSession, rows: list[PluginMetric], phases=None) -> None:
     """Write metric rows and evaluate the alert rules that watch them.
 
@@ -182,6 +210,7 @@ async def record_metrics(session: AsyncSession, rows: list[PluginMetric], phases
             row.recorded_at = now
     session.add_all(rows)
     await session.flush()
+    await _keep_latest(session, rows)
     mark("flush")
 
     by_key: dict[tuple[str, str, str], list[PluginMetric]] = {}