feat(swarm): services short, nodes down and managers lost are alertable (#5788)
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each manager's swarm report the docker plugin now records three figures on resource "swarm": swarm_services_short, swarm_nodes_down and swarm_managers_unreachable. Drained nodes are left out of the last two, since a drain is deliberate. The figures are recorded at most once every 30 s whichever manager reported, and are offered to alert rules. Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents fall back to the node's status. A maintenance job on a swarm node also keeps a window over docker/swarm, so its drain and reboot hold those alerts until the job ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -292,6 +292,20 @@ async def _persist_status(session, host, status: dict) -> None:
|
||||
row.reported_at = status["at"]
|
||||
|
||||
|
||||
async def _record_swarm_health(session, swarm: dict) -> None:
|
||||
"""The swarm-wide figures alert rules watch (#5788), at most once every
|
||||
RECORD_EVERY whichever manager reported."""
|
||||
from datetime import timezone
|
||||
from steward.core.alerts import MetricBatch
|
||||
from . import swarm_health
|
||||
if not swarm_health.due(datetime.now(timezone.utc)):
|
||||
return
|
||||
batch = MetricBatch()
|
||||
for metric, value in swarm_health.figures(swarm).items():
|
||||
batch.add(source_module="docker", resource_name=swarm_health.RESOURCE, metric_name=metric, value=value)
|
||||
await batch.record(session)
|
||||
|
||||
|
||||
async def _expire_swarm(session, host) -> None:
|
||||
"""Drop the swarm rows a host reported while it was a manager."""
|
||||
from .models import DockerSwarmNode, DockerSwarmService
|
||||
@@ -338,6 +352,7 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
|
||||
await mark("status")
|
||||
if swarm is not None:
|
||||
await _persist_swarm(session, host, swarm)
|
||||
await _record_swarm_health(session, swarm)
|
||||
await mark("swarm")
|
||||
if disk is not None:
|
||||
await _persist_disk(session, host, disk)
|
||||
|
||||
Reference in New Issue
Block a user