feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m45s
CI / publish (push) Successful in 17s

On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
+11 -2
View File
@@ -575,17 +575,26 @@ def test_swarm_topology_persisted(app):
s.add(h)
await s.flush()
# No containers in this sample — swarm must still persist.
from steward.core.plugin_manager import loaded_plugin_submodule
loaded_plugin_submodule("docker", "swarm_health")._last_recorded = None
since = datetime.now(timezone.utc)
await persist(s, h, [], swarm)
hid = h.id
# The swarm-wide figures alert rules watch (#5788): web is 2 of 3.
figures = dict((await s.execute(text(
"SELECT metric_name, value FROM plugin_metrics WHERE source_module = 'docker' "
"AND resource_name = 'swarm' AND recorded_at >= :t"), {"t": since})).all())
svc = (await s.execute(text(
"SELECT mode, desired, running, image, placement_json, tasks_json "
"FROM docker_swarm_services WHERE host_id = :h"), {"h": hid})).first()
node = (await s.execute(text(
"SELECT role, availability, status, leader "
"FROM docker_swarm_nodes WHERE host_id = :h"), {"h": hid})).first()
return svc, node
return svc, node, figures
svc, node = asyncio.run(_go())
svc, node, figures = asyncio.run(_go())
assert figures == {"swarm_services_short": 1.0, "swarm_nodes_down": 0.0,
"swarm_managers_unreachable": 0.0}
assert svc[0] == "replicated" and svc[1] == 3 and svc[2] == 2 and svc[3] == "nginx"
assert "n1" in svc[4] # placement JSON carries the node id
(task,) = json.loads(svc[5]) # docker_010: the manager's task records survive