feat(metrics): roll plugin_metrics up to hourly to bound storage
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
"""Unit tests for the plugin_metrics rollup cutoff helpers (no DB)."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from steward.core.metrics_retention import _hour_floor, _rollup_cutoff
|
||||
|
||||
|
||||
def test_hour_floor_drops_sub_hour():
|
||||
dt = datetime(2026, 6, 20, 15, 42, 9, 123456, tzinfo=timezone.utc)
|
||||
assert _hour_floor(dt) == datetime(2026, 6, 20, 15, 0, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def test_rollup_cutoff_is_hour_aligned_and_offset():
|
||||
now = datetime(2026, 6, 20, 15, 42, 9, tzinfo=timezone.utc)
|
||||
cutoff = _rollup_cutoff(now, 7)
|
||||
assert (cutoff.minute, cutoff.second, cutoff.microsecond) == (0, 0, 0)
|
||||
# 7 whole days back, then floored to the hour.
|
||||
assert cutoff == datetime(2026, 6, 13, 15, 0, 0, 0, tzinfo=timezone.utc)
|
||||
Reference in New Issue
Block a user