bvandeusen/FabledSteward
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
Details
CI / integration (push) Successful in 1m32s
Details
CI / unit (push) Successful in 1m45s
Details
CI / publish (push) Successful in 17s
Details

Browse Source 
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/host-agent-design.md
+14
View File
@@ -252,6 +252,20 @@ Maintenance** (`/plugins/host_agent/maintenance`).
last week, then each plan with its schedule, next slot and last five runs.
Each host's Agent panel shows its last maintenance job.
### The swarm's health, alertable (1.19.0, #5788)
Every manager's agent reports the whole swarm. From each report the docker plugin (`plugins/docker/swarm_health.py`) records three swarm-wide figures on resource `swarm`, source `docker`:
- `swarm_services_short`: services running fewer tasks than they want. A service scaled to 0 is never short.
- `swarm_nodes_down`: nodes not `ready`.
- `swarm_managers_unreachable`: managers the swarm reports as not `reachable`. Agent 1.19.0 sends `ManagerStatus.Reachability`; before that, a manager whose status isn't `ready` counts.
Drained nodes are left out of the last two. A node is drained on purpose (VDNT-Util01 always, any node during its maintenance), so its going away is expected, and its host's own "Agent offline" rule still covers it.
The figures are recorded at most once every 30 s, whichever manager sent them, so three managers make one series and losing one doesn't silence it.
A maintenance job on a swarm node opens a second window over `docker` / `swarm` beside its host's (`maintenance._swarm_window`). An alert the drain or reboot causes then waits for the job, and one still firing when the window ends is sent then.
### What happened to the host (1.14.0, #5756)
A post may carry `events`, said once each, with the next post that gets through: