feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m52s
CI / publish (push) Successful in 1m12s

The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.

A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.

get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-24 17:21:44 -04:00
co-authored by Claude Opus 5.5
parent 8b42a6a865
commit 2b97c36fd4
23 changed files with 884 additions and 219 deletions
+10 -7
View File
@@ -157,13 +157,16 @@ async def get_anomaly(anomaly_id: str, occurrences: int = 5) -> dict:
has one: an agent series, a container, a monitor pinned to a host.
- `occurrences[]`: `started_at`, `ended_at`, `held_s`, `peak_level`,
and `usual` for that hour as Steward has learned it now (`usual_hold`,
`usual_peak`, `flag_above`), plus `around`: alert transitions, Ansible
runs, other anomalies on the host, what plugins saw (container
starts, deaths and OOM kills; maintenance jobs; reboots and agent
stops; for a Traefik router, its swarm service's containers on any
host; syslog warnings and worse from the host's or the SNMP device's
own log), and kind "steward": a stretch Steward itself wasn't recording
(a restart or deploy), around which any reading can be wrong. From 30
`usual_peak`, `flag_above`), plus `around`: what the timeline holds
about the host or resource, its kind the entry's type (alert.fired,
ansible.changed, maintenance.done, host.rebooted, docker.deployed,
docker.restarted, docker.crash_loop, steward.config …, hidden types
included); other anomalies on the host; what plugins saw that the
timeline doesn't hold (container starts, stops and health changes; for
a Traefik router, its swarm service's containers on any host; syslog
warnings and worse from the host's or the SNMP device's own log); and
kind "steward": a stretch Steward itself wasn't recording (a restart
or deploy), around which any reading can be wrong. From 30
minutes before to 30 minutes after. Each kind keeps the 40 events
nearest the occurrence's start; `around_left_out` counts the rest
(0 when nothing was cut). And