feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m45s
CI / publish (push) Successful in 17s

On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
@@ -298,16 +298,18 @@ def test_swarm_nodes_normalises_fields():
nodes = [
{"ID": "n1", "Description": {"Hostname": "mgr"},
"Spec": {"Role": "manager", "Availability": "active"},
"Status": {"State": "ready"}, "ManagerStatus": {"Leader": True}},
"Status": {"State": "ready"}, "ManagerStatus": {"Leader": True, "Reachability": "Reachable"}},
{"ID": "n2", "Description": {"Hostname": "wrk"},
"Spec": {"Role": "worker", "Availability": "drain"},
"Status": {"State": "down"}},
]
out = {n["node_id"]: n for n in a._swarm_nodes(nodes)}
assert out["n1"] == {"node_id": "n1", "hostname": "mgr", "role": "manager",
"availability": "active", "status": "ready", "leader": True}
"availability": "active", "status": "ready", "leader": True,
"reachability": "reachable"}
assert out["n2"] == {"node_id": "n2", "hostname": "wrk", "role": "worker",
"availability": "drain", "status": "down", "leader": False}
"availability": "drain", "status": "down", "leader": False,
"reachability": ""}
def test_collect_swarm_skips_on_worker(monkeypatch):