perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)

The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.

- plugin_metric_latest: one row per (source, resource, metric).
  record_metrics upserts it in the same transaction, moving forward only and
  in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
  resource that stopped reporting disappears when its history does, as before.
  Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
  would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
  instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
  deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
  retry for cells that don't poll; "as of HH:MM" in the header, and
  "couldn't refresh · as of" when a refresh fails over shown data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/migrations/versions/0030_plugin_metric_latest.py

@@ -0,0 +1,55 @@
+"""The newest sample of each metric, kept beside plugin_metrics (#5208)
+
+Revision ID: 0030_plugin_metric_latest
+Revises: 0029_alert_no_data
+Create Date: 2026-10-07
+"""
+from typing import Sequence, Union
+from alembic import op
+import sqlalchemy as sa
+
+revision: str = "0030_plugin_metric_latest"
+down_revision: Union[str, None] = "0029_alert_no_data"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "plugin_metric_latest",
+        sa.Column("source_module", sa.String(64), primary_key=True),
+        sa.Column("resource_name", sa.String(255), primary_key=True),
+        sa.Column("metric_name", sa.String(128), primary_key=True),
+        sa.Column("value", sa.Float, nullable=False),
+        sa.Column("recorded_at", sa.DateTime(timezone=True), nullable=False),
+    )
+    # Seed from history without a full scan: step through the distinct keys of
+    # the (source, resource, metric, recorded_at) index one probe at a time,
+    # then take each key's newest row with a backward probe of the same index.
+    op.execute("""
+        WITH RECURSIVE k AS (
+            (SELECT source_module, resource_name, metric_name FROM plugin_metrics
+             ORDER BY source_module, resource_name, metric_name LIMIT 1)
+            UNION ALL
+            SELECT n.source_module, n.resource_name, n.metric_name
+            FROM k, LATERAL (
+                SELECT p.source_module, p.resource_name, p.metric_name FROM plugin_metrics p
+                WHERE (p.source_module, p.resource_name, p.metric_name)
+                    > (k.source_module, k.resource_name, k.metric_name)
+                ORDER BY p.source_module, p.resource_name, p.metric_name LIMIT 1
+            ) n
+        )
+        INSERT INTO plugin_metric_latest
+            (source_module, resource_name, metric_name, value, recorded_at)
+        SELECT k.source_module, k.resource_name, k.metric_name, l.value, l.recorded_at
+        FROM k, LATERAL (
+            SELECT p.value, p.recorded_at FROM plugin_metrics p
+            WHERE p.source_module = k.source_module AND p.resource_name = k.resource_name
+              AND p.metric_name = k.metric_name
+            ORDER BY p.recorded_at DESC LIMIT 1
+        ) l
+    """)
+
+
+def downgrade() -> None:
+    op.drop_table("plugin_metric_latest")