feat(host_agent): how far each host's clock is from Steward's (milestone 490)
CI / lint (push) Successful in 5s
CI / integration (push) Successful in 1m8s
CI / unit (push) Successful in 1m34s
CI / publish (push) Successful in 2m27s

- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
  moment the post arrived: clock_offset_s / clock_checked_at on the
  registration (host_agent_003), and a clock_off_secs metric (either way)
  that alert rules can watch. Replaces the old skew log, which measured the
  newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
  off by about the same amount, says Steward's own clock is the likely one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 08:00:33 -04:00
co-authored by Claude Opus 5.5
parent 766eef04c5
commit 96c1ea9407
15 changed files with 217 additions and 11 deletions
+4
View File
@@ -97,6 +97,9 @@ DEFAULTS: dict[str, Any] = {
# many container deaths within an hour count as crash-looping.
"thresholds.tls_warn_days": 14, "thresholds.tls_crit_days": 3,
"thresholds.crash_loop_deaths": 3,
# Seconds a host's clock may be from Steward's before it is called out
# (milestone 490), either way.
"thresholds.clock_warn_s": 30, "thresholds.clock_crit_s": 300,
# Docker time-series retention (rule 25 — tunable, no restart). Raw 30s
# samples are heavy, so keep a short raw window then roll up to hourly
# averages kept much longer; lifecycle events are light, keep a month.
@@ -448,6 +451,7 @@ def to_thresholds_cfg(settings: dict[str, Any]) -> dict:
"latency": {"warn": g("ping.threshold.good_ms", 50), "crit": g("ping.threshold.warn_ms", 200), "dir": "high"},
"uptime": {"warn": g("thresholds.uptime_warn", 99.0), "crit": g("thresholds.uptime_crit", 95.0), "dir": "low"},
"tls_days": {"warn": g("thresholds.tls_warn_days", 14), "crit": g("thresholds.tls_crit_days", 3), "dir": "low"},
"clock": {"warn": g("thresholds.clock_warn_s", 30), "crit": g("thresholds.clock_crit_s", 300), "dir": "high"},
"crash_loop": {"warn": g("thresholds.crash_loop_deaths", 3), "crit": g("thresholds.crash_loop_deaths", 3), "dir": "high"},
}