feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -157,13 +157,16 @@ async def get_anomaly(anomaly_id: str, occurrences: int = 5) -> dict:
|
||||
has one: an agent series, a container, a monitor pinned to a host.
|
||||
- `occurrences[]`: `started_at`, `ended_at`, `held_s`, `peak_level`,
|
||||
and `usual` for that hour as Steward has learned it now (`usual_hold`,
|
||||
`usual_peak`, `flag_above`), plus `around`: alert transitions, Ansible
|
||||
runs, other anomalies on the host, what plugins saw (container
|
||||
starts, deaths and OOM kills; maintenance jobs; reboots and agent
|
||||
stops; for a Traefik router, its swarm service's containers on any
|
||||
host; syslog warnings and worse from the host's or the SNMP device's
|
||||
own log), and kind "steward": a stretch Steward itself wasn't recording
|
||||
(a restart or deploy), around which any reading can be wrong. From 30
|
||||
`usual_peak`, `flag_above`), plus `around`: what the timeline holds
|
||||
about the host or resource, its kind the entry's type (alert.fired,
|
||||
ansible.changed, maintenance.done, host.rebooted, docker.deployed,
|
||||
docker.restarted, docker.crash_loop, steward.config …, hidden types
|
||||
included); other anomalies on the host; what plugins saw that the
|
||||
timeline doesn't hold (container starts, stops and health changes; for
|
||||
a Traefik router, its swarm service's containers on any host; syslog
|
||||
warnings and worse from the host's or the SNMP device's own log); and
|
||||
kind "steward": a stretch Steward itself wasn't recording (a restart
|
||||
or deploy), around which any reading can be wrong. From 30
|
||||
minutes before to 30 minutes after. Each kind keeps the 40 events
|
||||
nearest the occurrence's start; `around_left_out` counts the rest
|
||||
(0 when nothing was cut). And
|
||||
|
||||
Reference in New Issue
Block a user