feat(swarm): services short, nodes down and managers lost are alertable (#5788)
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each manager's swarm report the docker plugin now records three figures on resource "swarm": swarm_services_short, swarm_nodes_down and swarm_managers_unreachable. Drained nodes are left out of the last two, since a drain is deliberate. The figures are recorded at most once every 30 s whichever manager reported, and are offered to alert rules. Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents fall back to the node's status. A maintenance job on a swarm node also keeps a window over docker/swarm, so its drain and reboot hold those alerts until the job ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -298,16 +298,18 @@ def test_swarm_nodes_normalises_fields():
|
||||
nodes = [
|
||||
{"ID": "n1", "Description": {"Hostname": "mgr"},
|
||||
"Spec": {"Role": "manager", "Availability": "active"},
|
||||
"Status": {"State": "ready"}, "ManagerStatus": {"Leader": True}},
|
||||
"Status": {"State": "ready"}, "ManagerStatus": {"Leader": True, "Reachability": "Reachable"}},
|
||||
{"ID": "n2", "Description": {"Hostname": "wrk"},
|
||||
"Spec": {"Role": "worker", "Availability": "drain"},
|
||||
"Status": {"State": "down"}},
|
||||
]
|
||||
out = {n["node_id"]: n for n in a._swarm_nodes(nodes)}
|
||||
assert out["n1"] == {"node_id": "n1", "hostname": "mgr", "role": "manager",
|
||||
"availability": "active", "status": "ready", "leader": True}
|
||||
"availability": "active", "status": "ready", "leader": True,
|
||||
"reachability": "reachable"}
|
||||
assert out["n2"] == {"node_id": "n2", "hostname": "wrk", "role": "worker",
|
||||
"availability": "drain", "status": "down", "leader": False}
|
||||
"availability": "drain", "status": "down", "leader": False,
|
||||
"reachability": ""}
|
||||
|
||||
|
||||
def test_collect_swarm_skips_on_worker(monkeypatch):
|
||||
|
||||
Reference in New Issue
Block a user