feat(swarm): services short, nodes down and managers lost are alertable (#5788)
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each manager's swarm report the docker plugin now records three figures on resource "swarm": swarm_services_short, swarm_nodes_down and swarm_managers_unreachable. Drained nodes are left out of the last two, since a drain is deliberate. The figures are recorded at most once every 30 s whichever manager reported, and are offered to alert rules. Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents fall back to the node's status. A maintenance job on a swarm node also keeps a window over docker/swarm, so its drain and reboot hold those alerts until the job ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -83,6 +83,10 @@ def get_alert_metrics() -> dict[str, list[str]]:
|
||||
"""What ingest.py records per container (resource "<host>/<container>"):
|
||||
restart_count is cumulative (alert on crash-looping); is_healthy is 1.0 /
|
||||
0.0 from the container's HEALTHCHECK, only for containers that define one.
|
||||
Per host (resource "<host>"), what Docker keeps on its disk (#5304)."""
|
||||
Per host (resource "<host>"), what Docker keeps on its disk (#5304).
|
||||
Swarm-wide (resource "swarm"), from any manager's report: services short
|
||||
of their replicas, nodes down and managers unreachable, drained nodes
|
||||
aside (#5788, swarm_health.py)."""
|
||||
from .disk_view import DISK_METRICS
|
||||
return {"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy", *DISK_METRICS]}
|
||||
from .swarm_health import SWARM_METRICS
|
||||
return {"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy", *DISK_METRICS, *SWARM_METRICS]}
|
||||
|
||||
Reference in New Issue
Block a user