feat(host_agent): how far each host's clock is from Steward's (milestone 490)
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the moment the post arrived: clock_offset_s / clock_checked_at on the registration (host_agent_003), and a clock_off_secs metric (either way) that alert rules can watch. Replaces the old skew log, which measured the newest sample and was off by up to an interval. - Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical. - Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning. - Hosts list: names hosts whose clock is off; when every reporting host is off by about the same amount, says Steward's own clock is the likely one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -97,6 +97,9 @@ DEFAULTS: dict[str, Any] = {
|
||||
# many container deaths within an hour count as crash-looping.
|
||||
"thresholds.tls_warn_days": 14, "thresholds.tls_crit_days": 3,
|
||||
"thresholds.crash_loop_deaths": 3,
|
||||
# Seconds a host's clock may be from Steward's before it is called out
|
||||
# (milestone 490), either way.
|
||||
"thresholds.clock_warn_s": 30, "thresholds.clock_crit_s": 300,
|
||||
# Docker time-series retention (rule 25 — tunable, no restart). Raw 30s
|
||||
# samples are heavy, so keep a short raw window then roll up to hourly
|
||||
# averages kept much longer; lifecycle events are light, keep a month.
|
||||
@@ -448,6 +451,7 @@ def to_thresholds_cfg(settings: dict[str, Any]) -> dict:
|
||||
"latency": {"warn": g("ping.threshold.good_ms", 50), "crit": g("ping.threshold.warn_ms", 200), "dir": "high"},
|
||||
"uptime": {"warn": g("thresholds.uptime_warn", 99.0), "crit": g("thresholds.uptime_crit", 95.0), "dir": "low"},
|
||||
"tls_days": {"warn": g("thresholds.tls_warn_days", 14), "crit": g("thresholds.tls_crit_days", 3), "dir": "low"},
|
||||
"clock": {"warn": g("thresholds.clock_warn_s", 30), "crit": g("thresholds.clock_crit_s", 300), "dir": "high"},
|
||||
"crash_loop": {"warn": g("thresholds.crash_loop_deaths", 3), "crit": g("thresholds.crash_loop_deaths", 3), "dir": "high"},
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user