feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -750,13 +750,18 @@ async def prune_docker(host: str, what: str, check: bool = True) -> dict:
|
||||
"next": f'wait_for_run(run_id="{run.id}")'}
|
||||
|
||||
|
||||
# Deploys, restarts, deaths and OOM kills are on the timeline (timeline.py),
|
||||
# which get_anomaly reads; these are the container events it doesn't hold.
|
||||
_UNTIMELINED = ("start", "stop", "health_change")
|
||||
|
||||
|
||||
async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dict]:
|
||||
"""Container lifecycle events on a host in a window (get_anomaly): starts
|
||||
(with the image, so a deploy shows), deaths with their exit, OOM kills,
|
||||
restarts and health changes."""
|
||||
"""Container events on a host in a window (get_anomaly) that aren't on
|
||||
the timeline: starts, stops and health changes."""
|
||||
from steward.mcp.tools._events import event
|
||||
rows = (await db.execute(select(DockerEvent).where(
|
||||
DockerEvent.host_id == host_id, DockerEvent.at >= since, DockerEvent.at <= until)
|
||||
DockerEvent.host_id == host_id, DockerEvent.event.in_(_UNTIMELINED),
|
||||
DockerEvent.at >= since, DockerEvent.at <= until)
|
||||
.order_by(DockerEvent.at).limit(500))).scalars()
|
||||
return [event(e.at, "container", f"{e.container_name} {e.event}" + (f" ({e.detail})" if e.detail else ""),
|
||||
subject=f"{host_name}/{e.container_name}") for e in rows]
|
||||
|
||||
Reference in New Issue
Block a user