feat(alerts): the host's temperature and throughput can drive alert rules (#5787)
temp_c_max was recorded on every ingest, but host_agent didn't offer it to rules, so "host running hot" couldn't be watched. The catalog guard didn't see the row(...) helper ingest builds its rows with. Now offered: temp_c_max, temp_c per sensor, and network and disk throughput per host, interface and disk. A new guard holds every literal row(...) name to the catalog, except NOT_ALERTABLE, where each exclusion is given a reason. temp_c_max reads in °C. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -78,7 +78,18 @@ def get_alert_metrics() -> dict[str, list[str]]:
|
||||
reboot_required is 1 when the host needs a reboot (agent ≥ 1.16.0):
|
||||
"security updates waiting" is updates_security > 0. maintenance_failed
|
||||
comes from the scheduler (#5768): 1 when the host's latest maintenance job
|
||||
failed or timed out, 0 once one succeeds: "maintenance failed" is > 0."""
|
||||
failed or timed out, 0 once one succeeds: "maintenance failed" is > 0.
|
||||
temp_c_max is the host's hottest real thermometer, in °C (#5787); sensors
|
||||
with nothing wired to them are left out (#5759, #5783): "running hot" is
|
||||
> 85. temp_c is one sensor, resource "<host>:temp:<label>". net_rx_bps /
|
||||
net_tx_bps and disk_read_bps / disk_write_bps are bytes per second, for
|
||||
the whole host ("<host>") or one interface ("<host>:net:<iface>") or disk
|
||||
("<host>:diskio:<device>").
|
||||
|
||||
Recorded but not offered (NOT_ALERTABLE): disk_used_bytes and
|
||||
disk_total_bytes, where disk_used_pct is the figure to watch (and docker
|
||||
uses disk_total_bytes for something else), and mem_cached_bytes /
|
||||
mem_buffers_bytes, parts of what mem_available_bytes already counts."""
|
||||
return {"host_agent": [
|
||||
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
|
||||
"disk_used_pct_worst", "disk_used_pct", "inodes_used_pct", "load_1m", "load_5m", "load_15m",
|
||||
@@ -86,4 +97,11 @@ def get_alert_metrics() -> dict[str, list[str]]:
|
||||
"uptime_secs", "seconds_since_seen", "clock_off_secs", "mounts_stalled",
|
||||
"agent_rss_bytes", "agent_cpu_pct",
|
||||
"updates_pending", "updates_security", "reboot_required", "maintenance_failed",
|
||||
"temp_c_max", "temp_c", "net_rx_bps", "net_tx_bps", "disk_read_bps", "disk_write_bps",
|
||||
]}
|
||||
|
||||
|
||||
# Metrics host_agent records that no rule is offered, each for the reason
|
||||
# get_alert_metrics gives; tests/core/test_metric_catalog.py holds every other
|
||||
# recorded name to the catalog.
|
||||
NOT_ALERTABLE = frozenset({"disk_used_bytes", "disk_total_bytes", "mem_cached_bytes", "mem_buffers_bytes"})
|
||||
|
||||
Reference in New Issue
Block a user