feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m52s
CI / publish (push) Successful in 1m12s

The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.

A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.

get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-24 17:21:44 -04:00
co-authored by Claude Opus 5.5
parent 8b42a6a865
commit 2b97c36fd4
23 changed files with 884 additions and 219 deletions
+9 -4
View File
@@ -750,13 +750,18 @@ async def prune_docker(host: str, what: str, check: bool = True) -> dict:
"next": f'wait_for_run(run_id="{run.id}")'}
# Deploys, restarts, deaths and OOM kills are on the timeline (timeline.py),
# which get_anomaly reads; these are the container events it doesn't hold.
_UNTIMELINED = ("start", "stop", "health_change")
async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dict]:
"""Container lifecycle events on a host in a window (get_anomaly): starts
(with the image, so a deploy shows), deaths with their exit, OOM kills,
restarts and health changes."""
"""Container events on a host in a window (get_anomaly) that aren't on
the timeline: starts, stops and health changes."""
from steward.mcp.tools._events import event
rows = (await db.execute(select(DockerEvent).where(
DockerEvent.host_id == host_id, DockerEvent.at >= since, DockerEvent.at <= until)
DockerEvent.host_id == host_id, DockerEvent.event.in_(_UNTIMELINED),
DockerEvent.at >= since, DockerEvent.at <= until)
.order_by(DockerEvent.at).limit(500))).scalars()
return [event(e.at, "container", f"{e.container_name} {e.event}" + (f" ({e.detail})" if e.detail else ""),
subject=f"{host_name}/{e.container_name}") for e in rows]