feat(swarm): services short, nodes down and managers lost are alertable (#5788)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m45s
CI / publish (push) Successful in 17s

On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.

Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.

A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 22:23:52 -04:00
co-authored by Claude Opus 5.5
parent 26a3ec3339
commit fdc5c07183
10 changed files with 246 additions and 18 deletions
+15
View File
@@ -292,6 +292,20 @@ async def _persist_status(session, host, status: dict) -> None:
row.reported_at = status["at"]
async def _record_swarm_health(session, swarm: dict) -> None:
"""The swarm-wide figures alert rules watch (#5788), at most once every
RECORD_EVERY whichever manager reported."""
from datetime import timezone
from steward.core.alerts import MetricBatch
from . import swarm_health
if not swarm_health.due(datetime.now(timezone.utc)):
return
batch = MetricBatch()
for metric, value in swarm_health.figures(swarm).items():
batch.add(source_module="docker", resource_name=swarm_health.RESOURCE, metric_name=metric, value=value)
await batch.record(session)
async def _expire_swarm(session, host) -> None:
"""Drop the swarm rows a host reported while it was a manager."""
from .models import DockerSwarmNode, DockerSwarmService
@@ -338,6 +352,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
await mark("status")
if swarm is not None:
await _persist_swarm(session, host, swarm)
await _record_swarm_health(session, swarm)
await mark("swarm")
if disk is not None:
await _persist_disk(session, host, disk)