feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m36s
CI / integration (push) Successful in 1m52s
CI / publish (push) Successful in 1m12s

The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.

A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.

get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-24 17:21:44 -04:00
co-authored by Claude Opus 5.5
parent 8b42a6a865
commit 2b97c36fd4
23 changed files with 884 additions and 219 deletions
+21
View File
@@ -23,6 +23,7 @@ KINDS: dict[str, str] = {
"alert": "Alert",
"anomaly": "Anomaly",
"ticket": "Ticket",
"observed": "Observed", # Steward saw it happen: a reboot, a crash-loop
"steward": "Steward", # Steward's own settings and access
}
@@ -53,6 +54,16 @@ def register_type(key: str, label: str, kind: str, area: str, shown: bool, about
TYPES[key] = EntryType(key, label, kind, area, shown, about)
# The full records an entry can point to: its ref_type and the model whose
# id it names, so the timeline can tell a record still kept from one aged out.
# A plugin adds its own with register_ref() from its setup().
REFS: dict[str, object] = {}
def register_ref(ref_type: str, model) -> None:
REFS[ref_type] = model
for _t in (
("alert.fired", "Alert fired", "alert", "monitoring", True,
"An alert rule's condition held long enough to fire."),
@@ -80,6 +91,16 @@ for _t in (
"Someone signed in, or failed to."),
("steward.backup", "Steward backups", "steward", "storage", True,
"Steward backed itself up, or was restored from a backup."),
# An anomaly's entry changes type as it is reviewed; its area is what the
# metric is about.
("anomaly.to_review", "Anomaly to review", "anomaly", "monitoring", False,
"A metric left its normal and nobody has reviewed it yet."),
("anomaly.real", "Anomaly judged real", "anomaly", "monitoring", True,
"An anomaly someone reviewed and marked a real problem."),
("anomaly.normal", "Anomaly accepted as normal", "anomaly", "monitoring", False,
"An anomaly someone reviewed and marked normal: its level is learned."),
("anomaly.lapsed", "Anomaly not reviewed", "anomaly", "monitoring", False,
"An anomaly nobody reviewed within the review window."),
):
register_type(*_t)