feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m45s
CI / publish (push) Successful in 17s

On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
+6 -2
View File
@@ -83,6 +83,10 @@ def get_alert_metrics() -> dict[str, list[str]]:
"""What ingest.py records per container (resource "<host>/<container>"):
restart_count is cumulative (alert on crash-looping); is_healthy is 1.0 /
0.0 from the container's HEALTHCHECK, only for containers that define one.
Per host (resource "<host>"), what Docker keeps on its disk (#5304)."""
Per host (resource "<host>"), what Docker keeps on its disk (#5304).
Swarm-wide (resource "swarm"), from any manager's report: services short
of their replicas, nodes down and managers unreachable, drained nodes
aside (#5788, swarm_health.py)."""
from .disk_view import DISK_METRICS
return {"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy", *DISK_METRICS]}
from .swarm_health import SWARM_METRICS
return {"docker": ["cpu_pct", "mem_pct", "restart_count", "is_healthy", *DISK_METRICS, *SWARM_METRICS]}