feat(swarm): services short, nodes down and managers lost are alertable (#5788)
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each manager's swarm report the docker plugin now records three figures on resource "swarm": swarm_services_short, swarm_nodes_down and swarm_managers_unreachable. Drained nodes are left out of the last two, since a drain is deliberate. The figures are recorded at most once every 30 s whichever manager reported, and are offered to alert rules. Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents fall back to the node's status. A maintenance job on a swarm node also keeps a window over docker/swarm, so its drain and reboot hold those alerts until the job ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,7 +22,7 @@ from collections import deque
|
||||
from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
|
||||
from datetime import datetime, timezone
|
||||
|
||||
AGENT_VERSION = "1.18.0"
|
||||
AGENT_VERSION = "1.19.0"
|
||||
|
||||
# Default path to the local Docker Engine socket. Overridable via the
|
||||
# `docker_socket` config key; collection is silently skipped if it's absent or
|
||||
@@ -1170,7 +1170,10 @@ def _swarm_services(services: list, tasks: list) -> list:
|
||||
|
||||
|
||||
def _swarm_nodes(nodes: list) -> list:
|
||||
"""Normalise node role / availability / status (+ manager leader flag)."""
|
||||
"""Normalise node role / availability / status (+ manager leader flag and,
|
||||
from 1.19.0, whether the swarm can reach a manager: "reachable",
|
||||
"unreachable" or "unknown"; "" for a worker), so Steward can count the
|
||||
managers the swarm has lost (#5788)."""
|
||||
out = []
|
||||
for n in (nodes or []):
|
||||
spec = n.get("Spec") or {}
|
||||
@@ -1184,6 +1187,7 @@ def _swarm_nodes(nodes: list) -> list:
|
||||
"availability": (spec.get("Availability") or "active").lower(),
|
||||
"status": (status.get("State") or "unknown").lower(),
|
||||
"leader": bool(mgr.get("Leader", False)),
|
||||
"reachability": (mgr.get("Reachability") or "").lower(),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
Reference in New Issue
Block a user