feat(host_agent): how far each host's clock is from Steward's (milestone 490)
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the moment the post arrived: clock_offset_s / clock_checked_at on the registration (host_agent_003), and a clock_off_secs metric (either way) that alert rules can watch. Replaces the old skew log, which measured the newest sample and was off by up to an interval. - Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical. - Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning. - Hosts list: names hosts whose clock is off; when every reporting host is off by about the same amount, says Steward's own clock is the likely one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -285,6 +285,7 @@ METRICS: dict[str, tuple[str, str]] = {
|
||||
"load_15m": ("Load (15 min)", "runnable processes, averaged over 15 minutes"),
|
||||
"uptime_secs": ("Uptime", "time since the host booted"),
|
||||
"seconds_since_seen": ("Time since the agent reported", "over 3 minutes means the agent is offline"),
|
||||
"clock_off_secs": ("Clock off by", "how far the host's clock is from Steward's, either way"),
|
||||
}
|
||||
|
||||
# Where a rule's metric comes from, as the picker and the rules list say it.
|
||||
|
||||
@@ -97,6 +97,9 @@ DEFAULTS: dict[str, Any] = {
|
||||
# many container deaths within an hour count as crash-looping.
|
||||
"thresholds.tls_warn_days": 14, "thresholds.tls_crit_days": 3,
|
||||
"thresholds.crash_loop_deaths": 3,
|
||||
# Seconds a host's clock may be from Steward's before it is called out
|
||||
# (milestone 490), either way.
|
||||
"thresholds.clock_warn_s": 30, "thresholds.clock_crit_s": 300,
|
||||
# Docker time-series retention (rule 25 — tunable, no restart). Raw 30s
|
||||
# samples are heavy, so keep a short raw window then roll up to hourly
|
||||
# averages kept much longer; lifecycle events are light, keep a month.
|
||||
@@ -448,6 +451,7 @@ def to_thresholds_cfg(settings: dict[str, Any]) -> dict:
|
||||
"latency": {"warn": g("ping.threshold.good_ms", 50), "crit": g("ping.threshold.warn_ms", 200), "dir": "high"},
|
||||
"uptime": {"warn": g("thresholds.uptime_warn", 99.0), "crit": g("thresholds.uptime_crit", 95.0), "dir": "low"},
|
||||
"tls_days": {"warn": g("thresholds.tls_warn_days", 14), "crit": g("thresholds.tls_crit_days", 3), "dir": "low"},
|
||||
"clock": {"warn": g("thresholds.clock_warn_s", 30), "crit": g("thresholds.clock_crit_s", 300), "dir": "high"},
|
||||
"crash_loop": {"warn": g("thresholds.crash_loop_deaths", 3), "crit": g("thresholds.crash_loop_deaths", 3), "dir": "high"},
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user