Commit Graph
4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 3f9aae7f46 feat(host_agent): NFS delay per mount, read from /proc/self/mountstats (#5996)
CI / unit (push) Failing after 1m32s
CI / integration (push) Successful in 1m31s
CI / publish (push) Skipped
CI / lint (push) Successful in 2s
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.

Agent 1.22.0 (which also carries the backlog order from 41fc807).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:46:14 -04:00
bvandeusenandClaude Opus 5.5 b8f73270e7 perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m14s
CI / unit (push) Successful in 1m33s
CI / publish (push) Skipped
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.

- plugin_metric_latest: one row per (source, resource, metric).
  record_metrics upserts it in the same transaction, moving forward only and
  in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
  resource that stopped reporting disappears when its history does, as before.
  Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
  would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
  instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
  deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
  retry for cells that don't poll; "as of HH:MM" in the header, and
  "couldn't refresh · as of" when a refresh fails over shown data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:06:34 -04:00
bvandeusenandClaude Opus 4.8 8af297670e feat(metrics): roll plugin_metrics up to hourly to bound storage
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 47s
CI / integration (push) Successful in 2m18s
CI / publish (push) Successful in 1m6s
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.

- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
  count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
  raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
  delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
  _run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
  Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
  with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
  recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
  merges hourly + raw) against Postgres.

Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-20 21:08:30 -04:00
bvandeusenandClaude Opus 4.8 a78af23793 refactor(host_agent): extract metric-query helpers to a model-free module
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 48s
CI / integration (push) Successful in 2m23s
CI / publish (push) Successful in 1m6s
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").

Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
2026-06-19 23:32:32 -04:00