feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m45s
CI / publish (push) Successful in 17s

On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
+6 -2
View File
@@ -22,7 +22,7 @@ from collections import deque
from concurrent.futures import ThreadPoolExecutor, wait as wait_futures
from datetime import datetime, timezone
AGENT_VERSION = "1.18.0"
AGENT_VERSION = "1.19.0"
# Default path to the local Docker Engine socket. Overridable via the
# `docker_socket` config key; collection is silently skipped if it's absent or
@@ -1170,7 +1170,10 @@ def _swarm_services(services: list, tasks: list) -> list:
def _swarm_nodes(nodes: list) -> list:
"""Normalise node role / availability / status (+ manager leader flag)."""
"""Normalise node role / availability / status (+ manager leader flag and,
from 1.19.0, whether the swarm can reach a manager: "reachable",
"unreachable" or "unknown"; "" for a worker), so Steward can count the
managers the swarm has lost (#5788)."""
out = []
for n in (nodes or []):
spec = n.get("Spec") or {}
@@ -1184,6 +1187,7 @@ def _swarm_nodes(nodes: list) -> list:
"availability": (spec.get("Availability") or "active").lower(),
"status": (status.get("State") or "unknown").lower(),
"leader": bool(mgr.get("Leader", False)),
"reachability": (mgr.get("Reachability") or "").lower(),
})
return out