4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	b8f73270e7	perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208) ... The dashboard's host widgets took 18-60 s. Every "latest reading" query (hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host) searched plugin_metrics for the newest row per metric. Postgres has no skip scan, so each one read every raw sample in the 7-day window. - plugin_metric_latest: one row per (source, resource, metric). record_metrics upserts it in the same transaction, moving forward only and in sorted key order. Migration 0030 seeds it by stepping the existing index. - Retention drops a latest row once its sample leaves the raw window, so a resource that stopped reporting disappears when its history does, as before. Renames carry it along. - A quiet host keeps its last values, marked stale. A time-bounded query would have shown it as missing. - Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points, instead of fetching 6 h raw and drawing the last 120 samples. - Dashboard and share-view cells: "Loading…" until the first answer; a 20 s deadline per fetch; "Couldn't load (why)" with Retry now and an automatic retry for cells that don't poll; "as of HH:MM" in the header, and "couldn't refresh · as of" when a refresh fails over shown data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-06 23:06:34 -04:00
bvandeusenandClaude Opus 4.8	8af297670e	feat(metrics): roll plugin_metrics up to hourly to bound storage ... plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a short raw window is kept at full resolution, with hourly averages archived longer. - PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/ count per source/resource/metric/hour, unique bucket constraint + lookup index). - steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of raw older than the hour-aligned raw window, idempotent pg upsert into hourly, delete the rolled raw, prune hourly beyond the rollup window. - cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days; _run_metrics_retention drives the rollup with windows read live from settings. - Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the Thresholds & Retention page (new "Host metrics retention" card). - Chart read: _history_for_host merges the hourly rollup (older part of the range) with raw date_bin (recent part, capped ≤1h), so 30d charts keep working — recent at full resolution, older at hourly. Route passes raw_days from settings. - Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history merges hourly + raw) against Postgres. Speed was already handled by the indexes + SQL aggregation; this is the storage lever (raw window ~10x smaller). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC	2026-06-20 21:08:30 -04:00
bvandeusenandClaude Opus 4.8	a78af23793	refactor(host_agent): extract metric-query helpers to a model-free module ... The previous commit's integration test failed at collection-time import: importing plugins.host_agent.routes (which imports the host_agent ORM models at top level) double-registers host_agent_registrations against the app-loaded plugin's metadata ("Table already defined"). Move the two pure read helpers (_latest_metrics_for_host, _history_for_host) plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py, which imports only the core PluginMetric model — no plugin models. routes.py imports them back. The integration test now imports from metrics_query and no longer trips the loader's registration guard. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC	2026-06-19 23:32:32 -04:00
bvandeusenandClaude Opus 4.8	aff0c36d37	perf(host_agent): aggregate metric history in SQL; DISTINCT ON for latest ... Follow-on to the plugin_metrics indexes. plugin_metrics is already retention- bounded (core cleanup prunes > data.retention_days, default 90d) and charts top out at 30d, so the cost wasn't growth — it was the read path shipping raw rows to Python. - _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120 buckets) instead of fetching every raw sample (a 30d range was hundreds of thousands of rows) and downsampling in Python. Uses the new (source_module, resource_name, recorded_at) index. - _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY recorded_at DESC — newest row per group in one index-ordered pass, replacing the GROUP-BY-max subquery self-joined back to the whole history. - Integration test validates both against Postgres. Deliberately not a materialized raw→hourly rollup: these query-side changes deliver the speed; a rollup would additionally cut storage and remains a future option if scale demands it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC	2026-06-19 23:23:45 -04:00