perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)
The dashboard's host widgets took 18-60 s. Every "latest reading" query (hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host) searched plugin_metrics for the newest row per metric. Postgres has no skip scan, so each one read every raw sample in the 7-day window. - plugin_metric_latest: one row per (source, resource, metric). record_metrics upserts it in the same transaction, moving forward only and in sorted key order. Migration 0030 seeds it by stepping the existing index. - Retention drops a latest row once its sample leaves the raw window, so a resource that stopped reporting disappears when its history does, as before. Renames carry it along. - A quiet host keeps its last values, marked stale. A time-bounded query would have shown it as missing. - Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points, instead of fetching 6 h raw and drawing the last 120 samples. - Dashboard and share-view cells: "Loading…" until the first answer; a 20 s deadline per fetch; "Couldn't load (why)" with Retry now and an automatic retry for cells that don't poll; "as of HH:MM" in the header, and "couldn't refresh · as of" when a refresh fails over shown data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -44,6 +44,7 @@ def test_diagnostic_tools_over_a_seeded_fleet(app):
|
||||
)
|
||||
from steward.models.ansible import AnsibleRun, AnsibleRunStatus
|
||||
from steward.models.hosts import Host
|
||||
from steward.core.alerts import _keep_latest
|
||||
from steward.models.metrics import PluginMetric
|
||||
from steward.models.monitors import Monitor, MonitorResult
|
||||
from steward.models.users import User, UserRole
|
||||
@@ -77,16 +78,21 @@ def test_diagnostic_tools_over_a_seeded_fleet(app):
|
||||
s.add(MonitorResult(monitor_id=web_id, checked_at=now - timedelta(minutes=1),
|
||||
is_up=True, status_code=200, response_ms=40.0,
|
||||
tls_expires_at=now + timedelta(days=5)))
|
||||
for i, v in enumerate((10.0, 20.0, 95.0)):
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="cpu_pct", value=v,
|
||||
recorded_at=now - timedelta(minutes=40 - i * 20, seconds=30)))
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="mem_used_pct", value=50.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
s.add(PluginMetric(source_module="host_agent", resource_name=host_name + ":/",
|
||||
metric_name="disk_used_pct", value=41.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
samples = [PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="cpu_pct", value=v,
|
||||
recorded_at=now - timedelta(minutes=40 - i * 20, seconds=30))
|
||||
for i, v in enumerate((10.0, 20.0, 95.0))]
|
||||
samples.append(PluginMetric(source_module="host_agent", resource_name=host_name,
|
||||
metric_name="mem_used_pct", value=50.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
samples.append(PluginMetric(source_module="host_agent",
|
||||
resource_name=host_name + ":/",
|
||||
metric_name="disk_used_pct", value=41.0,
|
||||
recorded_at=now - timedelta(seconds=30)))
|
||||
s.add_all(samples)
|
||||
# What record_metrics keeps beside the rows, without evaluating
|
||||
# the rule this seed sets up by hand.
|
||||
await _keep_latest(s, samples)
|
||||
s.add(AlertRule(id=rule_id, name=f"cpu hot {tag}", source_module="host_agent",
|
||||
resource_name=host_name, metric_name="cpu_pct",
|
||||
operator=AlertOperator.gt, threshold=90.0,
|
||||
|
||||
Reference in New Issue
Block a user