feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -23,6 +23,7 @@ KINDS: dict[str, str] = {
|
||||
"alert": "Alert",
|
||||
"anomaly": "Anomaly",
|
||||
"ticket": "Ticket",
|
||||
"observed": "Observed", # Steward saw it happen: a reboot, a crash-loop
|
||||
"steward": "Steward", # Steward's own settings and access
|
||||
}
|
||||
|
||||
@@ -53,6 +54,16 @@ def register_type(key: str, label: str, kind: str, area: str, shown: bool, about
|
||||
TYPES[key] = EntryType(key, label, kind, area, shown, about)
|
||||
|
||||
|
||||
# The full records an entry can point to: its ref_type and the model whose
|
||||
# id it names, so the timeline can tell a record still kept from one aged out.
|
||||
# A plugin adds its own with register_ref() from its setup().
|
||||
REFS: dict[str, object] = {}
|
||||
|
||||
|
||||
def register_ref(ref_type: str, model) -> None:
|
||||
REFS[ref_type] = model
|
||||
|
||||
|
||||
for _t in (
|
||||
("alert.fired", "Alert fired", "alert", "monitoring", True,
|
||||
"An alert rule's condition held long enough to fire."),
|
||||
@@ -80,6 +91,16 @@ for _t in (
|
||||
"Someone signed in, or failed to."),
|
||||
("steward.backup", "Steward backups", "steward", "storage", True,
|
||||
"Steward backed itself up, or was restored from a backup."),
|
||||
# An anomaly's entry changes type as it is reviewed; its area is what the
|
||||
# metric is about.
|
||||
("anomaly.to_review", "Anomaly to review", "anomaly", "monitoring", False,
|
||||
"A metric left its normal and nobody has reviewed it yet."),
|
||||
("anomaly.real", "Anomaly judged real", "anomaly", "monitoring", True,
|
||||
"An anomaly someone reviewed and marked a real problem."),
|
||||
("anomaly.normal", "Anomaly accepted as normal", "anomaly", "monitoring", False,
|
||||
"An anomaly someone reviewed and marked normal: its level is learned."),
|
||||
("anomaly.lapsed", "Anomaly not reviewed", "anomaly", "monitoring", False,
|
||||
"An anomaly nobody reviewed within the review window."),
|
||||
):
|
||||
register_type(*_t)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user