feat(docker): a swarm service's metrics stay under one key across its tasks (#6718)

#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". The operator: "the metrics
should be keys to the service name not the task name... as that's what's
consistent between updates."

CPU, memory, restart count and health now go under "swarm/<service>", so a
normal, its anomalies and an alert rule carry across a redeploy or a move
between nodes. Network rates stay "<host>/<service>", the task id gone, as they
answer who on a host is sending. Tasks of a multi-task service share the key,
so its level reads as an average across them (of 51 live services, only
authentik-worker and portainer_agent run more than one task).

History under task names moves in a background task, 5000 raw samples every
15 s, not a migration: the raw table holds about 2M such rows and the app
would wait on it at boot. Hourly rows combine where two tasks share an hour,
and rows still to review merge to one per metric.

A service's anomalies no longer reach a host's page, as it has no one host;
get_anomaly reads what happened around one from the service's own events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-30 19:45:16 -04:00
co-authored by Claude Opus 5.5
parent 18191aa858
commit db39fc2604
11 changed files with 427 additions and 33 deletions
+22 -9
View File
@@ -49,9 +49,12 @@ def setup(app: "Quart") -> None:
from steward.core.norms import floor_setting, learn_normal
learn_normal("docker", "cpu_pct", floor_setting(app, "container_cpu_pct"))
learn_normal("docker", "mem_used_pct", floor_setting(app, "container_mem_pct"))
# A host's agent carries its containers' figures: they go silent with it (#5721).
# A host's agent carries its containers' figures: they go silent with it
# (#5721). A swarm service's aren't tied to one host.
from steward.core.alerts import agent_fed
agent_fed("docker", lambda resource: resource.split("/", 1)[0])
from .service import SERVICE_RESOURCE
agent_fed("docker", lambda resource: None if resource.startswith(SERVICE_RESOURCE)
else resource.split("/", 1)[0])
# An alert on a swarm figure names the services or nodes it counts (#6400).
from steward.core.alerts import names_behind
from .swarm_health import NAMERS
@@ -63,9 +66,10 @@ def setup(app: "Quart") -> None:
def get_scheduled_tasks() -> list:
# Collection is now agent-driven (pushed to the host_agent ingest); the
# central socket scrape was removed. No periodic task to register.
return []
# Collection is agent-driven (pushed to the host_agent ingest). The one
# task here moves task-named history to its service.
from .rekey import make_rekey_task
return [make_rekey_task(_app)]
def get_blueprint():
@@ -88,16 +92,25 @@ async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dic
return await events(db, host_id, host_name, since, until)
async def mcp_resource_events(db, source: str, resource: str, since, until) -> list[dict]:
from .service import SERVICE_RESOURCE
if source != "docker" or not resource.startswith(SERVICE_RESOURCE):
return []
from .mcp_tools import service_events
return await service_events(db, resource[len(SERVICE_RESOURCE):], since, until)
def get_nav() -> list[dict]:
# Surfaces in the sidebar's Infrastructure group (fleet-wide container view).
return [{"label": "Docker", "href": "/plugins/docker/"}]
def get_alert_metrics() -> dict[str, list[str]]:
"""What ingest.py records per container (resource "<host>/<container>"):
restart_count is cumulative (alert on crash-looping); is_healthy is 1.0 /
0.0 from the container's HEALTHCHECK, only for containers that define one.
net_rx_bps / net_tx_bps are the bytes a second it receives and sends
"""What ingest.py records per container (resource "swarm/<service>" for a
swarm task, else "<host>/<container>"): restart_count is cumulative (alert
on crash-looping); is_healthy is 1.0 / 0.0 from the container's HEALTHCHECK,
only for containers that define one. net_rx_bps / net_tx_bps are the bytes
a second it receives and sends, under "<host>/<service or container>"
(#6115): "a container flooding the uplink" is net_tx_bps over a level.
Per host (resource "<host>"), what Docker keeps on its disk (#5304).
Swarm-wide (resource "swarm"), from any manager's report: services short