feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -56,6 +56,15 @@ def net_rates(before: dict, snapshots: list) -> list:
|
||||
return out
|
||||
|
||||
|
||||
def _crash_loop_deaths() -> int:
|
||||
from quart import current_app
|
||||
from .service import CRASH_LOOP_DEATHS, crash_loop_deaths
|
||||
try:
|
||||
return crash_loop_deaths(current_app.config)
|
||||
except RuntimeError: # outside the app (a unit test)
|
||||
return CRASH_LOOP_DEATHS
|
||||
|
||||
|
||||
def _derive_events(old_state: dict, new_containers: list) -> list:
|
||||
"""Diff a fresh snapshot against stored per-container state → lifecycle events.
|
||||
|
||||
@@ -460,12 +469,19 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
|
||||
# the sample that noticed it was taken.
|
||||
finished = {c.get("name"): _parse_started_at(c.get("finished_at"))
|
||||
for c in latest_containers}
|
||||
derived = []
|
||||
for name, event, detail in _derive_events(old_state, latest_containers):
|
||||
at = finished.get(name) if event in ("die", "stop") else None
|
||||
session.add(DockerEvent(
|
||||
host_id=host.id, container_name=name,
|
||||
event=event, detail=detail, at=at or latest_at,
|
||||
))
|
||||
derived.append((name, event, detail, at or latest_at))
|
||||
# Deploys and crash-loops go on the timeline, judged against the
|
||||
# containers as they were before this snapshot.
|
||||
from . import timeline
|
||||
await timeline.from_snapshot(session, host, old_rows, latest_containers, derived, latest_at,
|
||||
_crash_loop_deaths())
|
||||
await mark("events")
|
||||
|
||||
# Current state, and the restart and health alerts, from the newest snapshot only.
|
||||
|
||||
Reference in New Issue
Block a user