feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)

The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.

A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.

get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/core/timeline.md

@@ -1,8 +1,8 @@
 # Timeline
 
-The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, and changes to Steward's own settings. Open it from **Timeline** in the sidebar, from a host's **History** card, or through the `get_timeline` MCP tool.
+The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, maintenance runs and reboots, deploys and crash-loops, anomalies once reviewed, and changes to Steward's own settings. Open it from **Timeline** in the sidebar, from a host's **History** card, or through the `get_timeline` MCP tool.
 
-Each entry stands on its own. An alert event, an Ansible run or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
+Each entry stands on its own. An alert event, an Ansible run, a maintenance job or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
 
 **Source:** `steward/timeline/`
 **Table:** `timeline_entries` (`steward/models/timeline.py`)
@@ -18,14 +18,24 @@ Every source writes its entry in the same transaction as its own record, through
 | Alert events | `core/alerts._add_event` | `alert.fired`, `alert.resolved`, `alert.acknowledged`, `alert.silent`, `alert.resumed` |
 | Ansible runs, when they finish | `ansible/executor.start_run`, and the startup sweep of interrupted runs | `ansible.changed`, `ansible.unchanged`, `ansible.failed`, `ansible.check` |
 | Audit rows | `core/audit.log_audit` | `steward.config`, `steward.access`, `steward.signin`, `steward.backup` |
+| Metric anomalies: one entry each, whose type follows its review | `core/anomalies` (opened, ended, marked, lapsed) | `anomaly.to_review`, `anomaly.real`, `anomaly.normal`, `anomaly.lapsed` |
+| Maintenance jobs, when they end (and again if the host comes back late) | host agent: `maintenance`, `cadence` | `maintenance.done`, `maintenance.failed` |
+| A host away: a reboot, a crash, lost contact, the agent restarting; kept up to date until it ends | host agent: `downtime.record_events` | `host.rebooted`, `host.lost_contact`, `host.agent_restarted` |
+| Container snapshots: a new image, a recreated container, a restart or death | Docker: `ingest.persist_host_docker` → `timeline.from_snapshot` | `docker.deployed`, `docker.crash_loop`, `docker.restarted` |
+
+A source whose record changes after it is written keeps one entry for it (`record.record_or_update`): a downtime gets its length when it ends, an anomaly changes type when it is reviewed.
+
+Docker's units are swarm services where a container has one, so a service's tasks, replaced on every deploy and crash, read as one subject. A swarm task moving to another host is not a deploy. A crash loop uses the same rule as the Docker pages: Settings → Thresholds' deaths within an hour. The death that reaches it opens one `docker.crash_loop` entry, and each later one extends it until the container has been quiet for an hour. The deaths before it stay as hidden `docker.restarted` entries.
 
 Some audit actions are left to another writer, because the event itself is recorded there: starting a run, acknowledging an alert, and starting maintenance. A person's own preferences (time zone, email settings) are not recorded. `catalog.audit_type` holds that mapping.
 
-On its first start, Steward fills the timeline from the last 90 days of alert events, finished runs and audit rows (`service.backfill`). It does this once, and records when in `timeline.backfilled_at`. An entry for a record that is already covered is skipped.
+On its first start, Steward fills the timeline from the last 90 days of records that predate it (`service.backfill`): core's alert events, finished runs and audit rows, and each loaded plugin's own through its `timeline_backfill(session, since)` (the host agent's maintenance jobs and downtime). It fills from each source once and records which in `timeline.filled`, so a plugin loaded later fills on the next daily turn. An entry for a record that is already covered is skipped. Docker has no fill: its events can't tell a deploy from a restart after the fact.
+
+`get_anomaly`'s "what else was going on" reads the timeline for the host or resource, hidden types included, beside each occurrence.
 
 ## Areas and kinds
 
-An entry has a **kind**, which says how it came about: Change, Automation, Alert, Anomaly, Ticket or Steward. Kinds are fixed in `catalog.KINDS`.
+An entry has a **kind**, which says how it came about: Change, Automation, Alert, Anomaly, Ticket, Observed (Steward saw it happen with no rule or person behind it: a reboot, a crash-loop) or Steward. Kinds are fixed in `catalog.KINDS`.
 
 An entry also has an **area**, which says what part of the infrastructure it is about. The areas start as Service, Host OS, Hardware, Storage, Network, Security and Monitoring.
 
@@ -41,8 +51,8 @@ Each entry is written with an area of its own. An alert's area comes from what i
 
 These are applied when the timeline is read, never written onto an entry. A change re-sorts the whole history. A hidden type is still recorded, so showing it again brings its entries back.
 
-Hidden by default: dry runs, playbook runs that changed nothing, and sign-ins.
+Hidden by default: dry runs, playbook runs that changed nothing, sign-ins, single container restarts, the agent restarting, and anomalies not yet reviewed, accepted as normal or lapsed.
 
 ## Adding a type
 
-A plugin registers its types in `setup()` with `catalog.register_type(key, label, kind, area, shown, about)`, and writes entries with `record.record(session, type=…, at=…, summary=…, hosts=…, ref=(kind, id))`. Settings → Timeline lists the new type at once.
+A plugin registers its types in `setup()` with `catalog.register_type(key, label, kind, area, shown, about)`, and the records its entries point to with `catalog.register_ref(ref_type, model)`, so an entry whose record has aged out reads "Details no longer kept". It writes entries with `record.record(session, type=…, at=…, summary=…, hosts=…, ref=(ref_type, id))`, or `record.record_or_update` for a record that changes. It may add `timeline_backfill(session, since)` to its `__init__.py` for records that predate the timeline. Settings → Timeline lists the new type at once. `plugins/host_agent/timeline.py` and `plugins/docker/timeline.py` are worked examples.

plugins/docker/__init__.py

@@ -52,6 +52,9 @@ def setup(app: "Quart") -> None:
     # A host's agent carries its containers' figures: they go silent with it (#5721).
     from steward.core.alerts import agent_fed
     agent_fed("docker", lambda resource: resource.split("/", 1)[0])
+    # Deploys and crash-loops go on the timeline.
+    from .timeline import register as register_timeline
+    register_timeline()
 
 
 def get_scheduled_tasks() -> list:

plugins/docker/ingest.py

@@ -56,6 +56,15 @@ def net_rates(before: dict, snapshots: list) -> list:
     return out
 
 
+def _crash_loop_deaths() -> int:
+    from quart import current_app
+    from .service import CRASH_LOOP_DEATHS, crash_loop_deaths
+    try:
+        return crash_loop_deaths(current_app.config)
+    except RuntimeError:                    # outside the app (a unit test)
+        return CRASH_LOOP_DEATHS
+
+
 def _derive_events(old_state: dict, new_containers: list) -> list:
     """Diff a fresh snapshot against stored per-container state → lifecycle events.
 
@@ -460,12 +469,19 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
         # the sample that noticed it was taken.
         finished = {c.get("name"): _parse_started_at(c.get("finished_at"))
                     for c in latest_containers}
+        derived = []
         for name, event, detail in _derive_events(old_state, latest_containers):
             at = finished.get(name) if event in ("die", "stop") else None
             session.add(DockerEvent(
                 host_id=host.id, container_name=name,
                 event=event, detail=detail, at=at or latest_at,
             ))
+            derived.append((name, event, detail, at or latest_at))
+        # Deploys and crash-loops go on the timeline, judged against the
+        # containers as they were before this snapshot.
+        from . import timeline
+        await timeline.from_snapshot(session, host, old_rows, latest_containers, derived, latest_at,
+                                     _crash_loop_deaths())
     await mark("events")
 
     # Current state, and the restart and health alerts, from the newest snapshot only.

plugins/docker/mcp_tools.py

@@ -750,13 +750,18 @@ async def prune_docker(host: str, what: str, check: bool = True) -> dict:
             "next": f'wait_for_run(run_id="{run.id}")'}
 
 
+# Deploys, restarts, deaths and OOM kills are on the timeline (timeline.py),
+# which get_anomaly reads; these are the container events it doesn't hold.
+_UNTIMELINED = ("start", "stop", "health_change")
+
+
 async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dict]:
-    """Container lifecycle events on a host in a window (get_anomaly): starts
+    """Container events on a host in a window (get_anomaly) that aren't on
-    (with the image, so a deploy shows), deaths with their exit, OOM kills,
+    the timeline: starts, stops and health changes."""
-    restarts and health changes."""
     from steward.mcp.tools._events import event
     rows = (await db.execute(select(DockerEvent).where(
-        DockerEvent.host_id == host_id, DockerEvent.at >= since, DockerEvent.at <= until)
+        DockerEvent.host_id == host_id, DockerEvent.event.in_(_UNTIMELINED),
+        DockerEvent.at >= since, DockerEvent.at <= until)
         .order_by(DockerEvent.at).limit(500))).scalars()
     return [event(e.at, "container", f"{e.container_name} {e.event}" + (f" ({e.detail})" if e.detail else ""),
                   subject=f"{host_name}/{e.container_name}") for e in rows]

plugins/docker/timeline.py

@@ -0,0 +1,160 @@
+# plugins/docker/timeline.py
+"""What Docker puts on the timeline, judged from each snapshot as it is
+ingested: a deploy (a container or swarm service running a new image, or a
+container recreated), and a crash-loop as one entry for the whole episode.
+A single restart or death is recorded but hidden. As on the Docker pages, a
+crash loop is Settings → Thresholds' number of deaths within CRASH_WINDOW:
+the one that reaches it opens a crash-loop entry, which each later one
+extends until the container has been quiet for CRASH_WINDOW.
+
+A container belongs to its swarm service when it has one, so a service's
+tasks, replaced on every deploy and every crash, read as one subject.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import timedelta
+
+from sqlalchemy import func, select
+
+from steward.models.timeline import TimelineEntry
+from steward.timeline import catalog
+from steward.timeline.record import record
+
+logger = logging.getLogger(__name__)
+
+TYPES = (
+    ("docker.deployed", "Deployed", "change", "service", True,
+     "A container or swarm service started on a new image, or a container was recreated."),
+    ("docker.crash_loop", "Crash-looping", "observed", "service", True,
+     "A container kept restarting or dying: one entry for the whole episode."),
+    ("docker.restarted", "Container restarted or died", "observed", "service", False,
+     "A single restart, death or out-of-memory kill."),
+)
+
+CRASH_WINDOW = timedelta(hours=1)
+_TROUBLE = ("restart", "die", "oom")
+
+
+def register() -> None:
+    for t in TYPES:
+        catalog.register_type(*t)
+
+
+def unit(c: dict) -> str:
+    """What a container is part of: its swarm service, else itself."""
+    return c.get("service_name") or c.get("name")
+
+
+def short_image(image: str | None) -> str:
+    """An image as a person reads it: the name and tag, and a few characters
+    of the digest when there is one."""
+    if not image:
+        return "an unknown image"
+    name, _, digest = image.partition("@")
+    name = name.rsplit("/", 1)[-1]
+    if digest:
+        return f"{name} ({digest.split(':', 1)[-1][:12]})"
+    return name
+
+
+def deploys(old_rows: list, latest: list[dict]) -> list[dict]:
+    """Each unit whose containers now run an image they didn't, or a
+    container recreated under the same name: {unit, name, image, was,
+    recreated, swarm}. A swarm task landing on a host that ran none of its
+    service before is the scheduler moving it, not a deploy."""
+    images: dict[str, set] = {}
+    ids = {}
+    for r in old_rows:
+        images.setdefault(r.service_name or r.name, set()).add(r.image)
+        ids[r.name] = (r.container_id, r.image)
+    out, seen = [], set()
+    for c in latest:
+        if (c.get("status") or "").lower() != "running" or not c.get("name"):
+            continue
+        key, image = unit(c), c.get("image") or ""
+        if key in seen:
+            continue
+        before = images.get(key)
+        old_id, old_image = ids.get(c["name"], (None, None))
+        if before and image not in before:
+            was = old_image if old_image and old_image != image else sorted(before)[0]
+            out.append({"unit": key, "name": c["name"], "image": image, "was": was,
+                        "recreated": False, "swarm": bool(c.get("service_name"))})
+        elif old_id and c.get("container_id") and old_id != c["container_id"] and not c.get("service_name"):
+            out.append({"unit": key, "name": c["name"], "image": image, "was": old_image,
+                        "recreated": True, "swarm": False})
+        elif before is None and not c.get("service_name"):
+            out.append({"unit": key, "name": c["name"], "image": image, "was": None,
+                        "recreated": False, "swarm": False})
+        else:
+            continue
+        seen.add(key)
+    return out
+
+
+def _deploy_summary(d: dict, host_name: str) -> str:
+    what = f"{d['unit']} on {host_name}"
+    if d["recreated"]:
+        return f"{what}: recreated on {short_image(d['image'])}"
+    if d["was"] is None:
+        return f"{what}: started on {short_image(d['image'])}"
+    new, old = short_image(d["image"]), short_image(d["was"])
+    if d["image"].split("@")[0] == (d["was"] or "").split("@")[0]:
+        return f"{what}: new build of {new}"
+    return f"{what}: {old} → {new}"
+
+
+async def _trouble(session, host, key: str, name: str, at, said: str, deaths: int) -> None:
+    """One restart, death or OOM kill of `key`: extends its crash-loop, opens
+    one when it is the `deaths`-th within CRASH_WINDOW, else a hidden entry."""
+    subject = f"{host.name}/{key}"
+    loop = (await session.execute(select(TimelineEntry).where(
+        TimelineEntry.type == "docker.crash_loop", TimelineEntry.subject == subject,
+        TimelineEntry.until >= at - CRASH_WINDOW).order_by(TimelineEntry.at.desc()).limit(1))
+    ).scalar_one_or_none()
+    if loop is not None:
+        count = (loop.detail or {}).get("count", 0) + 1
+        loop.until = max(loop.until, at)
+        loop.detail = {**(loop.detail or {}), "count": count, "last": said}
+        loop.summary = f"{key} on {host.name} crash-looping: {count} restarts or deaths"
+        return
+    recent = (await session.execute(select(func.count(), func.min(TimelineEntry.at)).where(
+        TimelineEntry.type == "docker.restarted", TimelineEntry.subject == subject,
+        TimelineEntry.at >= at - CRASH_WINDOW))).one()
+    url = f"/plugins/docker/container/{host.id}/{name}"
+    if recent[0] + 1 >= deaths:
+        count = recent[0] + 1
+        record(session, type="docker.crash_loop", at=recent[1] or at, until=at,
+               summary=f"{key} on {host.name} crash-looping: {count} restarts or deaths",
+               hosts=[host.name], subject=subject, actor="Docker",
+               detail={"count": count, "last": said, "url": url})
+        return
+    record(session, type="docker.restarted", at=at, summary=f"{key} on {host.name}: {said}",
+           hosts=[host.name], subject=subject, actor="Docker", detail={"url": url})
+
+
+_SAID = {"restart": "restarted", "die": "died", "oom": "killed, out of memory"}
+
+
+async def from_snapshot(session, host, old_rows: list, latest: list[dict], events: list, at,
+                        deaths: int) -> None:
+    """The entries one snapshot makes: its deploys, and its restarts and
+    deaths (`events`: (name, event, detail, when)), in a savepoint so a
+    failure here never loses the snapshot. `deaths` makes a crash loop."""
+    try:
+        async with session.begin_nested():
+            for d in deploys(old_rows, latest):
+                url = "/plugins/docker/swarm" if d["swarm"] else f"/plugins/docker/container/{host.id}/{d['name']}"
+                record(session, type="docker.deployed", at=at, summary=_deploy_summary(d, host.name),
+                       hosts=[host.name], subject=f"{host.name}/{d['unit']}", actor="Docker",
+                       detail={"image": d["image"], "was": d["was"], "url": url})
+            units = {c.get("name"): unit(c) for c in latest if c.get("name")}
+            units.update({r.name: r.service_name or r.name for r in old_rows if r.name not in units})
+            for name, event, detail, when in events:
+                if event in _TROUBLE:
+                    said = _SAID[event] + (f" ({detail})" if detail else "")
+                    await _trouble(session, host, units.get(name, name), name, when or at, said, deaths)
+                    await session.flush()
+    except Exception:
+        logger.exception("Docker on %s: timeline entries could not be written", host.name)

plugins/host_agent/__init__.py

@@ -34,6 +34,9 @@ def setup(app: "Quart") -> None:
     from steward.core.cleanup import prune_by_age
     from .models import HostDowntime
     prune_by_age(HostDowntime.started_at)
+    # Maintenance runs and the stretches a host was away go on the timeline.
+    from .timeline import register as register_timeline
+    register_timeline()
 
 
 def get_audit_entities() -> dict:
@@ -62,9 +65,9 @@ async def mcp_overview(db) -> list[dict]:
     return await findings(db)
 
 
-async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dict]:
+async def timeline_backfill(session, since) -> list:
-    from .mcp_tools import mcp_events as events
+    from .timeline import backfill
-    return await events(db, host_id, host_name, since, until)
+    return await backfill(session, since)
 
 
 def get_alert_metrics() -> dict[str, list[str]]:

plugins/host_agent/cadence.py

@@ -22,7 +22,7 @@ from __future__ import annotations
 import json
 from datetime import datetime, time, timedelta, timezone
 
-from . import maintenance
+from . import maintenance, timeline
 
 WEEKDAYS = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
 # How long a host may take to be back in service after its job finished.
@@ -241,6 +241,8 @@ async def advance(session, run, plan, now: datetime) -> str:
         if move == "fail":
             if job is not None and job.state == "done":
                 maintenance._finish(job, "failed", now, why)     # the host never came back into service
+                await timeline.write(session, f"maintenance job {job.id}",
+                                     timeline.job_entry(session, job, host.name if host else None))
             name = host.name if host else host_id
             _end(run, "failed", now, f"{name}: {why}")
         elif move == "stop":
@@ -325,5 +327,7 @@ async def failed_hosts(session, now: datetime) -> dict[str, float]:
                 HostAgentRegistration.host_id == host.id))).scalar_one_or_none()
             if await _back(session, host, job, reg, now) is None:
                 job.came_back_at = now
+                await timeline.write(session, f"maintenance job {job.id}",
+                                     timeline.job_entry(session, job, host.name))
         out[name] = 1.0 if maintenance.still_failed(job) else 0.0
     return out

plugins/host_agent/downtime.py

@@ -51,36 +51,45 @@ async def record_events(session, host_id: str, events, last_seen_at: datetime |
     """Apply one post's events to the host's downtime, oldest first; returns
     how many rows were added or changed. `last_seen_at` is when the agent was
     last heard before this post: a boot after it that nobody announced is a
-    restart without a shutdown."""
+    restart without a shutdown. Each row added or changed goes on the
+    timeline, kept up to date as it ends."""
     if not isinstance(events, list):
         return 0
     timed = sorted(((t, e) for e in events if isinstance(e, dict)
                     for t in [_when(e)] if t is not None), key=lambda te: te[0])
-    changed = 0
+    changed = []
     for at, event in timed:
         kind = event.get("kind")
+        row = None
         if kind == "stopping":
-            changed += await _stopping(session, host_id, at, event.get("host_shutdown"))
+            row = await _stopping(session, host_id, at, event.get("host_shutdown"))
         elif kind == "started":
-            changed += await _started(session, host_id, at, _at(event.get("boot_at")), last_seen_at)
+            row = await _started(session, host_id, at, _at(event.get("boot_at")), last_seen_at)
         elif kind == "lost_contact":
-            changed += await _lost_contact(session, host_id, at, _at(event.get("to")))
+            row = await _lost_contact(session, host_id, at, _at(event.get("to")))
-    return changed
+        if row is not None and row not in changed:
+            changed.append(row)
+    if changed:
+        from . import timeline
+        await session.flush()
+        for row in changed:
+            await timeline.write(session, f"host downtime {row.id}", timeline.downtime_entry(session, row))
+    return len(changed)
 
 
-async def _stopping(session, host_id: str, at: datetime, host_shutdown) -> int:
+async def _stopping(session, host_id: str, at: datetime, host_shutdown) -> HostDowntime | None:
     seen = (await session.execute(select(HostDowntime.id).where(
         HostDowntime.host_id == host_id, HostDowntime.kind.in_(ANNOUNCED),
         HostDowntime.started_at == at))).first()
     if seen:
-        return 0
+        return None
-    session.add(HostDowntime(host_id=host_id, kind="reboot" if host_shutdown else "agent_stop",
+    row = HostDowntime(host_id=host_id, kind="reboot" if host_shutdown else "agent_stop", started_at=at)
-                             started_at=at))
+    session.add(row)
-    return 1
+    return row
 
 
 async def _started(session, host_id: str, at: datetime, boot_at: datetime | None,
-                   last_seen_at: datetime | None) -> int:
+                   last_seen_at: datetime | None) -> HostDowntime | None:
     stop = (await session.execute(select(HostDowntime).where(
         HostDowntime.host_id == host_id, HostDowntime.kind.in_(ANNOUNCED),
         HostDowntime.ended_at.is_(None), HostDowntime.started_at <= at)
@@ -91,31 +100,34 @@ async def _started(session, host_id: str, at: datetime, boot_at: datetime | None
         # boot after it, or it stayed up while only the agent restarted.
         if boot_at is not None:
             stop.kind = "reboot" if boot_at > stop.started_at else "agent_stop"
-        return 1
+        return stop
     if boot_at is None or last_seen_at is None or boot_at <= last_seen_at:
-        return 0
+        return None
     seen = (await session.execute(select(HostDowntime.id).where(
         HostDowntime.host_id == host_id, HostDowntime.kind == "unclean",
         HostDowntime.ended_at == at))).first()
     if seen:
-        return 0
+        return None
-    session.add(HostDowntime(host_id=host_id, kind="unclean", started_at=last_seen_at, ended_at=at))
+    row = HostDowntime(host_id=host_id, kind="unclean", started_at=last_seen_at, ended_at=at)
-    return 1
+    session.add(row)
+    return row
 
 
-async def _lost_contact(session, host_id: str, since: datetime, until: datetime | None) -> int:
+async def _lost_contact(session, host_id: str, since: datetime,
+                        until: datetime | None) -> HostDowntime | None:
     if until is None or until <= since:
-        return 0
+        return None
     row = (await session.execute(select(HostDowntime).where(
         HostDowntime.host_id == host_id, HostDowntime.kind == "lost_contact",
         HostDowntime.started_at == since))).scalar_one_or_none()
     if row is None:
-        session.add(HostDowntime(host_id=host_id, kind="lost_contact", started_at=since, ended_at=until))
+        row = HostDowntime(host_id=host_id, kind="lost_contact", started_at=since, ended_at=until)
-        return 1
+        session.add(row)
+        return row
     if row.ended_at is None or until > row.ended_at:
         row.ended_at = until
-        return 1
+        return row
-    return 0
+    return None
 
 
 async def announced_stops(session, now: datetime, grace_seconds: int) -> dict[str, datetime]:

plugins/host_agent/maintenance.py

@@ -354,6 +354,17 @@ async def _host_name(session, host_id: str) -> str:
     return host.name if host is not None else host_id
 
 
+async def _moved(session, job, now: datetime) -> None:
+    """After a job moved: its window follows it, and once it ends it goes on
+    the timeline."""
+    host_name = await _host_name(session, job.host_id)
+    await _window(session, job, host_name, now)
+    if job.state in FINISHED:
+        from . import timeline
+        await timeline.write(session, f"maintenance job {job.id}",
+                             timeline.job_entry(session, job, host_name))
+
+
 async def apply_events(session, host_id: str, events, now: datetime) -> int:
     """Apply one post's `maintenance` results and `started` boots to the jobs
     they belong to; returns how many jobs changed. A result is matched by its
@@ -382,7 +393,7 @@ async def apply_events(session, host_id: str, events, now: datetime) -> int:
                 if apply_boot(job, boot_at, now):
                     changed.add(job)
     for job in changed:
-        await _window(session, job, await _host_name(session, job.host_id), now)
+        await _moved(session, job, now)
     return len(changed)
 
 
@@ -440,7 +451,7 @@ async def tick(session, now: datetime) -> int:
             gave_up += 1
             moved = True
         if moved:
-            await _window(session, job, await _host_name(session, job.host_id), now)
+            await _moved(session, job, now)
     return gave_up
 
 

plugins/host_agent/mcp_tools.py

@@ -15,9 +15,8 @@ from steward.mcp.tools.overview import finding
 from steward.models.hosts import Host
 from steward.models.users import UserRole
 
-from . import cadence, downtime, maintenance, senders
+from . import cadence, maintenance, senders
-from .models import (HostAgentRegistration, HostDowntime, HostMaintenanceJob, HostMaintenancePlan,
+from .models import HostAgentRegistration, HostMaintenanceJob, HostMaintenancePlan, HostMaintenanceRun
-                     HostMaintenanceRun)
 
 # A registration this old that has never checked in is a failed install, not one in progress.
 PENDING_GRACE = timedelta(minutes=30)
@@ -354,31 +353,6 @@ async def mcp_overview(db) -> list[dict]:
     return out
 
 
-async def mcp_events(db, host_id: str, host_name: str, since, until) -> list[dict]:
-    """What the host agent knows happened on a host in a window (get_anomaly):
-    maintenance jobs, and downtime (reboots, agent stops, lost contact)."""
-    from steward.mcp.tools._events import event
-    out = []
-    jobs = (await db.execute(select(HostMaintenanceJob).where(
-        HostMaintenanceJob.host_id == host_id, HostMaintenanceJob.created_at <= until,
-        (HostMaintenanceJob.finished_at.is_(None)) | (HostMaintenanceJob.finished_at >= since))
-        .order_by(HostMaintenanceJob.created_at).limit(200))).scalars()
-    for job in jobs:
-        out.append(event(job.created_at, "maintenance", f"Maintenance: {maintenance.summary(job)}",
-                         until=job.finished_at, ids={"job_id": job.id}))
-    now = now_utc()
-    rows = (await db.execute(select(HostDowntime).where(
-        HostDowntime.host_id == host_id, HostDowntime.started_at <= until,
-        (HostDowntime.ended_at.is_(None)) | (HostDowntime.ended_at >= since))
-        .order_by(HostDowntime.started_at).limit(200))).scalars()
-    for row in rows:
-        seen = downtime.describe(row, now)
-        out.append(event(row.started_at, "downtime",
-                         f"{seen['label']} ({int(seen['seconds'])} s{', still going' if seen['ongoing'] else ''})",
-                         until=row.ended_at))
-    return out
-
-
 def register_tools(register) -> None:
     register(list_agents, role=UserRole.viewer, read_only=True)
     register(list_maintenance_plans, role=UserRole.viewer, read_only=True)

plugins/host_agent/timeline.py

@@ -0,0 +1,105 @@
+# plugins/host_agent/timeline.py
+"""What the host agent puts on the timeline: each maintenance job once it
+ends, and each stretch a host was away (a reboot, a crash, lost contact, the
+agent restarting). One entry per job and per downtime row, kept up to date
+as the row changes: a downtime that ends gets its length, a failed job
+whose host came back late reads as back.
+"""
+from __future__ import annotations
+
+import logging
+
+from sqlalchemy import select
+
+from steward.timeline import catalog
+from steward.timeline.record import record_or_update, username
+
+from . import downtime, maintenance
+
+logger = logging.getLogger(__name__)
+
+TYPES = (
+    ("maintenance.done", "Maintenance run", "automation", "host_os", True,
+     "A host's agent updated it, and rebooted it when the update needed one."),
+    ("maintenance.failed", "Maintenance failed", "automation", "host_os", True,
+     "A maintenance job failed or timed out."),
+    ("host.rebooted", "Host rebooted", "observed", "host_os", True,
+     "A host went down and came back, announced or not (a crash, a power cut)."),
+    ("host.lost_contact", "Agent lost contact", "observed", "network", True,
+     "A host's agent ran but couldn't reach Steward."),
+    ("host.agent_restarted", "Agent stopped or restarted", "observed", "host_os", False,
+     "The agent stopped or restarted while its host stayed up."),
+)
+
+
+def register() -> None:
+    for t in TYPES:
+        catalog.register_type(*t)
+    from .models import HostDowntime, HostMaintenanceJob
+    catalog.register_ref("maintenance_job", HostMaintenanceJob)
+    catalog.register_ref("host_downtime", HostDowntime)
+
+
+async def _host_name(session, host_id: str) -> str | None:
+    from steward.models.hosts import Host
+    host = await session.get(Host, host_id)
+    return host.name if host is not None else None
+
+
+async def job_entry(session, job, host_name: str | None = None):
+    """The entry for a finished maintenance job, written or brought up to date."""
+    if job.state not in maintenance.FINISHED:
+        return None
+    host_name = host_name or await _host_name(session, job.host_id) or job.host_id
+    return await record_or_update(
+        session, ("maintenance_job", job.id),
+        type="maintenance.failed" if maintenance.still_failed(job) else "maintenance.done",
+        at=job.created_at, until=job.finished_at,
+        summary=f"Maintenance on {host_name}: {maintenance.summary(job)}",
+        hosts=[host_name], subject=host_name,
+        detail={"state": job.state, "packages": job.packages, "rebooted": job.back_at is not None,
+                "note": job.detail, "url": f"/hosts/{job.host_id}"},
+        actor=await username(session, job.created_by) or "Steward")
+
+
+_DOWNTIME_TYPES = {"reboot": "host.rebooted", "unclean": "host.rebooted",
+                   "lost_contact": "host.lost_contact", "agent_stop": "host.agent_restarted"}
+
+
+async def downtime_entry(session, row, host_name: str | None = None):
+    """The entry for one stretch a host was away, written or brought up to date."""
+    from steward.core.formatting import human_duration
+    host_name = host_name or await _host_name(session, row.host_id) or row.host_id
+    seen = downtime.describe(row, row.ended_at or row.started_at)
+    said = seen["label"][:1].lower() + seen["label"][1:]
+    said += f" for {human_duration(seen['seconds'])}" if row.ended_at is not None else ", not back yet"
+    return await record_or_update(
+        session, ("host_downtime", row.id),
+        type=_DOWNTIME_TYPES.get(row.kind, "host.rebooted"),
+        at=row.started_at, until=row.ended_at, summary=f"{host_name}: {said}",
+        hosts=[host_name], subject=host_name,
+        detail={"kind": row.kind, "url": f"/hosts/{row.host_id}"}, actor="Steward")
+
+
+async def write(session, what: str, coro) -> None:
+    """Run one entry writer in a savepoint: the source's own row matters more
+    than its entry, so a failure is logged and nothing else is undone."""
+    try:
+        async with session.begin_nested():
+            await coro
+    except Exception:
+        logger.exception("Timeline: the entry for %s could not be written", what)
+
+
+async def backfill(session, since) -> list:
+    """Entries for the jobs and downtime that predate the timeline."""
+    from .models import HostDowntime, HostMaintenanceJob
+    out = []
+    for job in (await session.execute(select(HostMaintenanceJob).where(
+            HostMaintenanceJob.created_at >= since,
+            HostMaintenanceJob.state.in_(maintenance.FINISHED)))).scalars().all():
+        out.append(await job_entry(session, job))
+    for row in (await session.execute(select(HostDowntime).where(
+            HostDowntime.started_at >= since))).scalars().all():
+        out.append(await downtime_entry(session, row))
+    return out

steward/core/anomalies.py

@@ -24,6 +24,7 @@ from sqlalchemy import ARRAY, String, bindparam, case, delete, func, or_, select
 
 from steward.core.norms import LEARNED, Norm, norms_at
 from steward.models.norms import MetricAnomaly, MetricAnomalySpan
+from steward.timeline.record import anomaly_changed
 
 WINDOW = timedelta(minutes=15)   # the level judged is this window's average
 DEPART = 1.25                    # times the slot's usual hold that counts as a departure
@@ -133,6 +134,7 @@ async def detect(session, now: datetime | None = None) -> dict:
             .where(MetricAnomalySpan.ended_at.is_(None)))).all()
     }
     opened = closed = 0
+    changed: list[MetricAnomaly] = []       # opened, joined or ended: their timeline entries follow
     for key, level in levels.items():
         norm = norms.get(key)
         learned = LEARNED.get((key[0], key[2]))
@@ -144,15 +146,19 @@ async def detect(session, now: datetime | None = None) -> dict:
                 _raise_peak(anomaly, span, level, norm, now)
             else:
                 _close(anomaly, span, now)
+                changed.append(anomaly)
                 closed += 1
         elif out:
-            await _occur(session, key, level, norm, now)
+            changed.append(await _occur(session, key, level, norm, now))
             opened += 1
     # An open occurrence whose key stopped reporting, lost its normal or fell
     # below its gate can't be judged any more: it ends now.
     for span, anomaly in open_now.values():
         _close(anomaly, span, now)
+        changed.append(anomaly)
         closed += 1
+    for anomaly in changed:
+        await anomaly_changed(session, anomaly)
     return {"opened": opened, "closed": closed}
 
 
@@ -169,9 +175,9 @@ def _close(anomaly: MetricAnomaly, span: MetricAnomalySpan, now: datetime) -> No
     anomaly.ended_at = now
 
 
-async def _occur(session, key, level: float, norm: Norm, now: datetime) -> None:
+async def _occur(session, key, level: float, norm: Norm, now: datetime) -> MetricAnomaly:
     """A departure on a key with no open occurrence: join the key's anomaly
-    still to review, or open a new one."""
+    still to review, or open a new one. Returns the anomaly."""
     source, resource, metric = key
     start = now - WINDOW
     anomaly = (await session.execute(
@@ -188,7 +194,7 @@ async def _occur(session, key, level: float, norm: Norm, now: datetime) -> None:
         session.add(anomaly)
         await session.flush()
         session.add(MetricAnomalySpan(anomaly_id=anomaly.id, started_at=start, peak_level=level))
-        return
+        return anomaly
     last = (await session.execute(
         select(MetricAnomalySpan).where(MetricAnomalySpan.anomaly_id == anomaly.id)
         .order_by(MetricAnomalySpan.started_at.desc()).limit(1))).scalar_one_or_none()
@@ -202,17 +208,25 @@ async def _occur(session, key, level: float, norm: Norm, now: datetime) -> None:
         anomaly.last_started_at = start
     anomaly.ended_at = None
     _raise_peak(anomaly, span, level, norm, now)
+    return anomaly
 
 
 async def lapse(session, now: datetime, review_days: int) -> int:
     """Ended anomalies still to review after `review_days` lapse. One still
     going never does. reviewed_at records when it lapsed. Returns how many."""
-    result = await session.execute(
+    ids = list((await session.execute(
         update(MetricAnomaly)
         .where(MetricAnomaly.review == TO_REVIEW, MetricAnomaly.ended_at.is_not(None),
                MetricAnomaly.ended_at < now - timedelta(days=review_days))
-        .values(review=LAPSED, reviewed_at=now))
+        .values(review=LAPSED, reviewed_at=now)
-    return result.rowcount or 0
+        .returning(MetricAnomaly.id)
+        .execution_options(synchronize_session=False))).scalars())
+    if ids:
+        for anomaly in (await session.execute(
+                select(MetricAnomaly).where(MetricAnomaly.id.in_(ids))
+                .execution_options(populate_existing=True))).scalars():
+            await anomaly_changed(session, anomaly)
+    return len(ids)
 
 
 async def prune(session, before: datetime) -> int:
@@ -238,6 +252,7 @@ async def mark(session, anomaly_id: str, review: str, by: str,
     anomaly.reviewed_by = by
     anomaly.reviewed_at = now or datetime.now(timezone.utc)
     anomaly.note = (note or "").strip() or None
+    await anomaly_changed(session, anomaly)
     return anomaly
 
 

steward/core/settings.py

@@ -61,8 +61,9 @@ DEFAULTS: dict[str, Any] = {
                        ["monitoring", "Monitoring"]],
     "timeline.types": {},
     "timeline.area_moved": {},
-    # When the timeline was first filled from the records that predate it.
+    # {source: when}: each source (core, a plugin) the timeline was filled
-    "timeline.backfilled_at": "",
+    # from once, from the records that predate it (timeline.service.backfill).
+    "timeline.filled": {},
     "monitors.poll_interval_seconds": 60,
     # Root log level: DEBUG, INFO, WARNING or ERROR (steward/core/logging_setup.py).
     # WARNING also hides the per-request access lines.

steward/mcp/tools/_events.py

@@ -1,11 +1,12 @@
 """What else was going on: events on a host, or one resource, in a window
 (#5837), so an anomaly comes with its circumstances.
 
-Core gathers alert transitions, Ansible runs, other metric anomalies and
+Core reads the timeline (alert episodes, Ansible runs, and what plugins
-Steward's own gaps in recording. A plugin adds its own through
+wrote there: maintenance, downtime, deploys, restarts), other metric
-`mcp_events(db, host_id, host_name, since, until)` in its __init__.py (the
+anomalies, and Steward's own gaps in recording. A plugin adds what the
-host agent its maintenance jobs and downtime, Docker its container events),
+timeline doesn't hold through `mcp_events(db, host_id, host_name, since,
-and for a resource with no host through
+until)` in its __init__.py (Docker its stops and health changes, syslog its
+warnings), and for a resource with no host through
 `mcp_resource_events(db, source, resource, since, until)` (Traefik: the
 containers of the swarm service behind a router, #5862); each returns a list
 of `event(...)`.
@@ -16,15 +17,11 @@ import logging
 from dataclasses import dataclass
 from datetime import datetime, timedelta
 
-from sqlalchemy import and_, or_, select
+from sqlalchemy import and_, func, or_, select
 
-from steward.alerts.service import watches_host
 from steward.core.alerts import host_of_resource
 from steward.core.formatting import human_duration
 from steward.mcp._context import mcp_app
-from steward.models.alerts import AlertEvent, AlertRule
-from steward.models.ansible import AnsibleRun
-from steward.models.ansible_inventory import AnsibleTarget
 from steward.models.hosts import Host
 from steward.models.metrics import PluginMetric
 from steward.models.monitors import Monitor
@@ -80,47 +77,23 @@ def _overlaps(start, end):
     return clause
 
 
-async def alert_events(db, place: Place, since: datetime, until: datetime) -> list[dict]:
+async def timeline_events(db, place: Place, since: datetime, until: datetime) -> list[dict]:
-    """Alert transitions on the resource itself, or anything its host
+    """What the timeline holds about the host, or the resource itself, while
-    carries: its agent series, its containers, its monitors."""
+    the window was open: alert episodes, playbook runs, maintenance, reboots
-    monitor_keys = set()
+    and lost contact, deploys, restarts and crash-loops, settings changes.
-    if place.host_id:
+    Hidden types are included: a dry run or a single restart can be what
-        monitor_keys = {(t, n) for t, n in (await db.execute(
+    explains a reading. Anomalies come from other_anomalies, by occurrence."""
-            select(Monitor.type, Monitor.name).where(Monitor.host_id == place.host_id))).all()}
+    from steward.models.timeline import TimelineEntry
+    about = [TimelineEntry.subject == place.resource]
+    if place.host_name:
+        about.append(TimelineEntry.hosts.contains([place.host_name]))
     rows = (await db.execute(
-        select(AlertEvent, AlertRule).join(AlertRule, AlertRule.id == AlertEvent.rule_id)
+        select(TimelineEntry)
-        .where(AlertEvent.transitioned_at >= since, AlertEvent.transitioned_at <= until)
+        .where(or_(*about), _overlaps(TimelineEntry.at, func.coalesce(TimelineEntry.until, TimelineEntry.at))
-        .order_by(AlertEvent.transitioned_at).limit(500))).all()
+               (since, until), or_(TimelineEntry.ref_type.is_(None), TimelineEntry.ref_type != "metric_anomaly"))
-    out = []
+        .order_by(TimelineEntry.at).limit(500))).scalars()
-    for ev, rule in rows:
+    return [event(e.at, e.type, e.summary, until=e.until, subject=e.subject,
-        mine = (rule.source_module, rule.resource_name) == (place.source, place.resource)
+                  ids={e.ref_type: e.ref_id} if e.ref_type else None) for e in rows]
-        if not mine and not (place.host_name and watches_host(rule, place.host_name, monitor_keys)):
-            continue
-        note = f" ({ev.note})" if ev.note else ""
-        out.append(event(ev.transitioned_at, "alert", f"{rule.name}: {ev.from_state} → {ev.to_state}{note}",
-                         subject=rule.resource_name, ids={"rule_id": rule.id}))
-    return out
-
-
-async def ansible_runs(db, place: Place, since: datetime, until: datetime) -> list[dict]:
-    """Ansible runs that touched the host while the window was open (by the
-    hosts in their recap: a run still going hasn't said yet)."""
-    if not place.host_id:
-        return []
-    names = {place.host_name} | set((await db.execute(
-        select(AnsibleTarget.name).where(AnsibleTarget.host_id == place.host_id))).scalars())
-    runs = (await db.execute(
-        select(AnsibleRun).where(_overlaps(AnsibleRun.started_at, AnsibleRun.finished_at)(since, until))
-        .order_by(AnsibleRun.started_at).limit(200))).scalars()
-    out = []
-    for r in runs:
-        if not names & set(((r.results or {}).get("hosts") or {}).keys()):
-            continue
-        check = " in check mode" if (r.params or {}).get("check") else ""
-        out.append(event(r.started_at, "ansible_run",
-                         f"Ansible {r.playbook_path}{check}: {r.status.value}",
-                         until=r.finished_at, ids={"run_id": r.id}))
-    return out
 
 
 def _same_host(source: str, resource: str, place: Place) -> bool:
@@ -248,8 +221,8 @@ async def around(db, place: Place, anomaly_id: str, since: datetime, until: date
     elsewhere; and how many events were left out, each source keeping its
     PER_SOURCE nearest `focus`."""
     anomalies, elsewhere = await other_anomalies(db, place, anomaly_id, since, until)
-    groups = [await alert_events(db, place, since, until), await ansible_runs(db, place, since, until),
+    groups = [await timeline_events(db, place, since, until), anomalies,
-              anomalies, *await plugin_events(place, since, until),
+              *await plugin_events(place, since, until),
               *await resource_events(place, since, until), await steward_gaps(db, since, until)]
     found, left_out = [], 0
     for group in groups:

steward/mcp/tools/anomalies.py

@@ -157,13 +157,16 @@ async def get_anomaly(anomaly_id: str, occurrences: int = 5) -> dict:
       has one: an agent series, a container, a monitor pinned to a host.
     - `occurrences[]`: `started_at`, `ended_at`, `held_s`, `peak_level`,
       and `usual` for that hour as Steward has learned it now (`usual_hold`,
-      `usual_peak`, `flag_above`), plus `around`: alert transitions, Ansible
+      `usual_peak`, `flag_above`), plus `around`: what the timeline holds
-      runs, other anomalies on the host, what plugins saw (container
+      about the host or resource, its kind the entry's type (alert.fired,
-      starts, deaths and OOM kills; maintenance jobs; reboots and agent
+      ansible.changed, maintenance.done, host.rebooted, docker.deployed,
-      stops; for a Traefik router, its swarm service's containers on any
+      docker.restarted, docker.crash_loop, steward.config …, hidden types
-      host; syslog warnings and worse from the host's or the SNMP device's
+      included); other anomalies on the host; what plugins saw that the
-      own log), and kind "steward": a stretch Steward itself wasn't recording
+      timeline doesn't hold (container starts, stops and health changes; for
-      (a restart or deploy), around which any reading can be wrong. From 30
+      a Traefik router, its swarm service's containers on any host; syslog
+      warnings and worse from the host's or the SNMP device's own log); and
+      kind "steward": a stretch Steward itself wasn't recording (a restart
+      or deploy), around which any reading can be wrong. From 30
       minutes before to 30 minutes after. Each kind keeps the 40 events
       nearest the occurrence's start; `around_left_out` counts the rest
       (0 when nothing was cut). And

steward/mcp/tools/timeline.py

@@ -12,9 +12,11 @@ async def get_timeline(host: str = "", area: str = "", kind: str = "", type: str
                        range: str = "7d", include_hidden: bool = False, limit: int = 50) -> dict:
     """What happened to the infrastructure, newest first, from every source in
     one list: alerts firing, resolving and going silent; playbook runs and
-    what they changed; changes to Steward's own settings, users and backups.
+    what they changed; maintenance runs, reboots and lost contact; deploys
-    Each entry outlives the record it came from, so this reaches further back
+    and crash-loops; anomalies once reviewed as real; changes to Steward's
-    than get_alert_history, list_ansible_runs or list_audit_events.
+    own settings, users and backups. Each entry outlives the record it came
+    from, so this reaches further back than get_alert_history,
+    list_ansible_runs or list_audit_events.
 
     Reach for it first for "what changed on this host lately?" or "what else
     was going on when this broke?". Follow an entry's `url` or ids into the
@@ -24,11 +26,13 @@ async def get_timeline(host: str = "", area: str = "", kind: str = "", type: str
     area: what part of the infrastructure an entry is about, as Settings →
       Timeline names its areas (`areas` in the reply lists them); empty for all.
     kind: how it came about: "change", "automation", "alert", "anomaly",
-      "ticket" or "steward"; empty for all.
+      "ticket", "observed" (seen happening: a reboot, a crash-loop) or
+      "steward"; empty for all.
     type: one entry type (`types` in the reply); it is returned even if hidden.
     range: "1h", "6h", "24h", "7d" (default) or "30d".
     include_hidden: also the types Settings hides from the timeline (dry runs,
-      sign-ins, playbook runs that changed nothing).
+      sign-ins, playbook runs that changed nothing, single container
+      restarts, anomalies not reviewed as real).
     limit: default 50, max 500.
     """
     if kind and kind not in catalog.KINDS:

steward/timeline/catalog.py

@@ -23,6 +23,7 @@ KINDS: dict[str, str] = {
     "alert": "Alert",
     "anomaly": "Anomaly",
     "ticket": "Ticket",
+    "observed": "Observed",      # Steward saw it happen: a reboot, a crash-loop
     "steward": "Steward",        # Steward's own settings and access
 }
 
@@ -53,6 +54,16 @@ def register_type(key: str, label: str, kind: str, area: str, shown: bool, about
     TYPES[key] = EntryType(key, label, kind, area, shown, about)
 
 
+# The full records an entry can point to: its ref_type and the model whose
+# id it names, so the timeline can tell a record still kept from one aged out.
+# A plugin adds its own with register_ref() from its setup().
+REFS: dict[str, object] = {}
+
+
+def register_ref(ref_type: str, model) -> None:
+    REFS[ref_type] = model
+
+
 for _t in (
     ("alert.fired", "Alert fired", "alert", "monitoring", True,
      "An alert rule's condition held long enough to fire."),
@@ -80,6 +91,16 @@ for _t in (
      "Someone signed in, or failed to."),
     ("steward.backup", "Steward backups", "steward", "storage", True,
      "Steward backed itself up, or was restored from a backup."),
+    # An anomaly's entry changes type as it is reviewed; its area is what the
+    # metric is about.
+    ("anomaly.to_review", "Anomaly to review", "anomaly", "monitoring", False,
+     "A metric left its normal and nobody has reviewed it yet."),
+    ("anomaly.real", "Anomaly judged real", "anomaly", "monitoring", True,
+     "An anomaly someone reviewed and marked a real problem."),
+    ("anomaly.normal", "Anomaly accepted as normal", "anomaly", "monitoring", False,
+     "An anomaly someone reviewed and marked normal: its level is learned."),
+    ("anomaly.lapsed", "Anomaly not reviewed", "anomaly", "monitoring", False,
+     "An anomaly nobody reviewed within the review window."),
 ):
     register_type(*_t)
 

steward/timeline/record.py

@@ -19,23 +19,44 @@ from steward.models.timeline import TimelineEntry
 logger = logging.getLogger(__name__)
 
 
-def record(session, *, type: str, at: datetime, summary: str, area: str | None = None,
+def _entry(*, type: str, at: datetime, summary: str, area: str | None = None,
            hosts=(), subject: str | None = None, detail: dict | None = None,
            actor: str = "system", until: datetime | None = None,
            ref: tuple[str, str] | None = None) -> TimelineEntry:
-    """Add one entry to `session`. `area` defaults to its type's; `ref` is
-    (kind, id) of the full record it points to, which is at most one entry."""
     known = catalog.TYPES.get(type)
-    entry = TimelineEntry(
+    return TimelineEntry(
         at=at, until=until, type=type,
         area=area or (known.area if known else catalog.OTHER[0]),
         hosts=sorted({h for h in hosts if h}), subject=subject[:255] if subject else None, summary=summary,
         detail=detail or None, actor=(actor or "system")[:64],
         ref_type=ref[0] if ref else None, ref_id=str(ref[1]) if ref else None)
+
+
+def record(session, **fields) -> TimelineEntry:
+    """Add one entry to `session`: type, at and summary, and optionally
+    area (defaults to its type's), hosts, subject, detail, actor, until and
+    ref, the (kind, id) of the full record it points to, which is at most
+    one entry."""
+    entry = _entry(**fields)
     session.add(entry)
     return entry
 
 
+async def record_or_update(session, ref: tuple[str, str], **fields) -> TimelineEntry:
+    """record() the entry for the record `ref` points to, or, when it has
+    one already, bring that one up to date: a source whose record changes
+    after it is written (a downtime that ends, an anomaly reviewed, a job
+    whose host came back late) keeps one entry for it, not one per change."""
+    entry = (await session.execute(select(TimelineEntry).where(
+        TimelineEntry.ref_type == ref[0], TimelineEntry.ref_id == str(ref[1])))).scalar_one_or_none()
+    if entry is None:
+        return record(session, ref=ref, **fields)
+    fresh = _entry(ref=ref, **fields)
+    for column in ("at", "until", "type", "area", "hosts", "subject", "summary", "detail", "actor"):
+        setattr(entry, column, getattr(fresh, column))
+    return entry
+
+
 async def username(session, user_id: str | None) -> str | None:
     if not user_id:
         return None
@@ -125,6 +146,43 @@ async def from_alert(session, rule, event, actor: str | None = None,
         ref=("alert_event", event.id))
 
 
+# ── Anomalies ───────────────────────────────────────────────────────────────
+
+_REVIEWED = {"to_review": "", "real": ": a real problem", "normal": ": accepted as normal",
+             "lapsed": ": not reviewed"}
+
+
+async def from_anomaly(session, anomaly) -> TimelineEntry:
+    """The entry for one anomaly, written when it opens and brought up to
+    date as it recurs and is reviewed: its type follows its review."""
+    from steward.core.formatting import metric_label, metric_value
+    metric, res = anomaly.metric_name, anomaly.resource_name
+    times = f", {anomaly.occurrences} times" if anomaly.occurrences > 1 else ""
+    return await record_or_update(
+        session, ("metric_anomaly", anomaly.id),
+        type=f"anomaly.{anomaly.review}", at=anomaly.started_at, until=anomaly.ended_at,
+        summary=(f"{metric_label(metric)} on {res} out of its normal (peak "
+                 f"{metric_value(metric, anomaly.peak_level)}, usual "
+                 f"{metric_value(metric, anomaly.usual_hold)}{times}){_REVIEWED.get(anomaly.review, '')}"),
+        area=catalog.area_of_metric(anomaly.source_module, metric),
+        hosts=await hosts_of_rule(session, anomaly.source_module, res), subject=res,
+        detail={"source": anomaly.source_module, "metric": metric, "review": anomaly.review,
+                "note": anomaly.note, "url": "/alerts/anomalies"},
+        actor=anomaly.reviewed_by if anomaly.review in ("real", "normal") and anomaly.reviewed_by
+        else "Steward")
+
+
+async def anomaly_changed(session, anomaly) -> None:
+    """Write an anomaly's entry in a savepoint. Never raises: the anomaly's
+    own record matters more than its entry."""
+    await session.flush()                   # the anomaly's id, and its row before the savepoint
+    try:
+        async with session.begin_nested():
+            await from_anomaly(session, anomaly)
+    except Exception:
+        logger.exception("Anomaly %s: its timeline entry could not be written", anomaly.id)
+
+
 # ── Ansible runs ────────────────────────────────────────────────────────────
 
 def _run_type(run) -> str:

steward/timeline/service.py

@@ -41,12 +41,15 @@ def query(cfg: Config, *, host: str = "", area: str = "", kind: str = "", type_:
     return stmt
 
 
-# The full records an entry can point to, and whether one is still kept.
 def _ref_models() -> dict:
+    """The full records an entry can point to: core's, and the ones loaded
+    plugins registered (catalog.register_ref)."""
     from steward.models.alerts import AlertEvent
     from steward.models.ansible import AnsibleRun
     from steward.models.audit import AuditEvent
-    return {"alert_event": AlertEvent, "ansible_run": AnsibleRun, "audit_event": AuditEvent}
+    from steward.models.norms import MetricAnomaly
+    return {"alert_event": AlertEvent, "ansible_run": AnsibleRun, "audit_event": AuditEvent,
+            "metric_anomaly": MetricAnomaly, **catalog.REFS}
 
 
 async def kept_refs(db, entries) -> set[tuple[str, str]]:
@@ -59,7 +62,7 @@ async def kept_refs(db, entries) -> set[tuple[str, str]]:
     models = _ref_models()
     for ref_type, ids in wanted.items():
         model = models.get(ref_type)
-        if model is None:           # a kind only a plugin knows: assume it is kept
+        if model is None:           # a plugin no longer loaded: assume it is kept
             kept |= {(ref_type, i) for i in ids}
             continue
         found = (await db.execute(select(model.id).where(model.id.in_(ids)))).scalars()
@@ -126,58 +129,87 @@ async def carry_host_rename(session, old: str, new: str) -> int:
     return result.rowcount or 0
 
 
-async def backfill(app, now: datetime | None = None) -> int:
+async def _core_entries(session, since: datetime) -> list:
-    """Fill the timeline once from the alert events, Ansible runs and audit
+    """Entries for the alert events, finished Ansible runs and audit rows
-    rows that predate it, BACKFILL_DAYS back, through the same builders the
+    since `since`, built by the live writers' own builders. One record that
-    live writers use. An entry for a record already covered is skipped, so
+    can't be read is skipped, not the whole fill."""
-    it is safe to run again. Records the run in timeline.backfilled_at."""
-    from steward.core.settings import get_setting, set_setting
     from steward.models.alerts import AlertEvent, AlertRule
     from steward.models.ansible import AnsibleRun
     from steward.models.audit import AuditEvent
     from .record import from_alert, from_audit, from_run
+    built: list = []
+
+    async def build(what: str, coro) -> None:
+        try:
+            built.append(await coro)
+        except Exception:
+            logger.exception("Timeline fill: %s skipped", what)
+
+    for event, rule in (await session.execute(
+            select(AlertEvent, AlertRule).join(AlertRule, AlertEvent.rule_id == AlertRule.id)
+            .where(AlertEvent.transitioned_at >= since))).all():
+        await build(f"alert event {event.id}", from_alert(session, rule, event))
+    for run in (await session.execute(
+            select(AnsibleRun).where(AnsibleRun.started_at >= since,
+                                     AnsibleRun.finished_at.is_not(None)))).scalars().all():
+        await build(f"Ansible run {run.id}", from_run(session, run))
+    for event in (await session.execute(
+            select(AuditEvent).where(AuditEvent.timestamp >= since))).scalars().all():
+        await build(f"audit row {event.id}", from_audit(session, event, _audit_detail(event)))
+    return built
+
+
+def fill_sources() -> dict:
+    """{source: its `(session, since) -> [entries]`}: core's, and each loaded
+    plugin's `timeline_backfill`."""
+    from steward.core.plugin_manager import get_loaded_plugin_modules
+    sources = {"steward": _core_entries}
+    for name, module in sorted(get_loaded_plugin_modules().items()):
+        hook = getattr(module, "timeline_backfill", None)
+        if callable(hook):
+            sources[name] = hook
+    return sources
+
+
+async def backfill(app, now: datetime | None = None, sources: dict | None = None) -> int:
+    """Fill the timeline from the records that predate it, BACKFILL_DAYS
+    back, once per source: core's, then each loaded plugin's. Which sources
+    are done, and when, is kept in timeline.filled, so a plugin loaded later
+    fills on the next turn. An entry for a record already covered is
+    skipped, so a source filled again adds nothing twice. Returns how many
+    entries were added."""
+    from steward.core.settings import get_setting, set_setting
 
     now = now or datetime.now(timezone.utc)
+    since = now - timedelta(days=BACKFILL_DAYS)
+    sources = fill_sources() if sources is None else sources
+    added = 0
     async with app.db_sessionmaker() as session:
         async with session.begin():
-            if await get_setting(session, "timeline.backfilled_at"):
+            filled = dict(await get_setting(session, "timeline.filled") or {})
-                return 0
+        # The builders add each entry to the session; no flush may write
-        since = now - timedelta(days=BACKFILL_DAYS)
+        # them before the insert below, which skips any already covered.
-        built: list[TimelineEntry] = []
+        session.sync_session.autoflush = False
-        async with session.begin():
+        for name, source in sources.items():
-            # The builders add each entry to the session; no flush may write
+            if name in filled:
-            # them before the insert below, which skips any already covered.
+                continue
-            session.sync_session.autoflush = False
+            try:
-
+                async with session.begin():
-            async def build(what: str, coro) -> None:
+                    rows = [e for e in await source(session, since) if e is not None]
-                # One record that can't be read is skipped, not the whole fill.
+                    for entry in rows:
-                try:
+                        session.expunge(entry)
-                    built.append(await coro)
+                    for chunk in range(0, len(rows), 500):
-                except Exception:
+                        result = await session.execute(
-                    logger.exception("Timeline fill: %s skipped", what)
+                            insert(TimelineEntry).values([_row(e) for e in rows[chunk:chunk + 500]])
-
+                            .on_conflict_do_nothing(index_elements=["ref_type", "ref_id"],
-            for event, rule in (await session.execute(
+                                                    index_where=text("ref_id IS NOT NULL")))
-                    select(AlertEvent, AlertRule).join(AlertRule, AlertEvent.rule_id == AlertRule.id)
+                        added += result.rowcount or 0
-                    .where(AlertEvent.transitioned_at >= since))).all():
+                    filled[name] = now.isoformat()
-                await build(f"alert event {event.id}", from_alert(session, rule, event))
+                    await set_setting(session, "timeline.filled", filled)
-            for run in (await session.execute(
+                logger.info("Timeline: filled from %s, the last %d days", name, BACKFILL_DAYS)
-                    select(AnsibleRun).where(AnsibleRun.started_at >= since,
+            except Exception:
-                                             AnsibleRun.finished_at.is_not(None)))).scalars().all():
+                logger.exception("Timeline: the fill from %s failed; it is tried again tomorrow", name)
-                await build(f"Ansible run {run.id}", from_run(session, run))
+    return added
-            for event in (await session.execute(
-                    select(AuditEvent).where(AuditEvent.timestamp >= since))).scalars().all():
-                await build(f"audit row {event.id}", from_audit(session, event, _audit_detail(event)))
-            rows = [e for e in built if e is not None]
-            for entry in rows:
-                session.expunge(entry)
-            for chunk in range(0, len(rows), 500):
-                await session.execute(
-                    insert(TimelineEntry).values([_row(e) for e in rows[chunk:chunk + 500]])
-                    .on_conflict_do_nothing(index_elements=["ref_type", "ref_id"],
-                                            index_where=text("ref_id IS NOT NULL")))
-            await set_setting(session, "timeline.backfilled_at", now.isoformat())
-    logger.info("Timeline: filled %d entries from the last %d days of records", len(rows), BACKFILL_DAYS)
-    return len(rows)
 
 
 def _audit_detail(event) -> dict:

tests/integration/test_mcp_anomaly_review.py

@@ -44,6 +44,7 @@ def test_an_anomaly_with_its_circumstances_judged_through_mcp(app):
     from steward.models.metrics import PluginMetric
     from steward.models.norms import MetricAnomaly, MetricAnomalySpan, MetricNorm
     from steward.models.users import User, UserRole
+    from steward.timeline.record import from_alert, from_run, record
 
     tag = uuid.uuid4().hex[:8]
     host_name, other_host = f"an-{tag}", f"an-other-{tag}"
@@ -98,21 +99,28 @@ def test_an_anomaly_with_its_circumstances_judged_through_mcp(app):
                                  created_by=user.id)
                 s.add(rule)
                 await s.flush()
-                s.add(AlertEvent(rule_id=rule.id, from_state="ok", to_state="firing", metric_value=91.0,
+                fired = AlertEvent(rule_id=rule.id, from_state="ok", to_state="firing", metric_value=91.0,
-                                 transitioned_at=new_start + timedelta(minutes=20)))
+                                   transitioned_at=new_start + timedelta(minutes=20))
-                s.add(AnsibleRun(playbook_path="maintenance/backup.yml", source_name="steward-builtin",
+                run = AnsibleRun(playbook_path="maintenance/backup.yml", source_name="steward-builtin",
                                  status=AnsibleRunStatus.success, started_at=new_start - timedelta(minutes=10),
                                  finished_at=new_start + timedelta(minutes=5), params={"check": False},
-                                 results={"hosts": {host_name: {"ok": 3}}, "failures": []}))
+                                 results={"hosts": {host_name: {"ok": 3}}, "failures": []})
+                s.add_all([fired, run])
+                await s.flush()
+                # The timeline, as the alert, the run and the plugins write it.
+                await from_alert(s, rule, fired)
+                await from_run(s, run)
+                record(s, type="docker.restarted", at=new_start + timedelta(minutes=15),
+                       summary=f"web on {host_name}: died (exit 137)", hosts=[host_name],
+                       subject=f"{host_name}/web")
+                record(s, type="host.rebooted", at=new_end - timedelta(minutes=5),
+                       until=new_end + timedelta(minutes=1), summary=f"{host_name}: rebooted for 6m",
+                       hosts=[host_name], subject=host_name)
+                # A container event the timeline doesn't hold, from Docker's own source.
                 await s.execute(text(
                     "INSERT INTO docker_events (id, host_id, container_name, event, detail, at) "
-                    "VALUES (:id, :h, 'web', 'die', 'exit 137', :at)"),
+                    "VALUES (:id, :h, 'web', 'health_change', 'healthy→unhealthy', :at)"),
-                    {"id": str(uuid.uuid4()), "h": host.id, "at": new_start + timedelta(minutes=15)})
+                    {"id": str(uuid.uuid4()), "h": host.id, "at": new_start + timedelta(minutes=16)})
-                await s.execute(text(
-                    "INSERT INTO host_downtime (id, host_id, kind, started_at, ended_at, created_at) "
-                    "VALUES (:id, :h, 'reboot', :s, :e, now())"),
-                    {"id": str(uuid.uuid4()), "h": host.id, "s": new_end - timedelta(minutes=5),
-                     "e": new_end + timedelta(minutes=1)})
 
     async def _go():
         await _seed()
@@ -127,12 +135,14 @@ def test_an_anomaly_with_its_circumstances_judged_through_mcp(app):
         assert newest["held_s"] == 3600 and newest["peak_level"] == 93.0
         assert newest["usual"]["usual_hold"] == 20.0 and newest["usual"]["flag_above"] >= 25.0
         kinds = {e["kind"] for e in newest["around"]}
-        assert {"alert", "ansible_run", "container", "downtime", "anomaly"} <= kinds
+        assert {"alert.fired", "ansible.unchanged", "docker.restarted", "host.rebooted", "container",
+                "anomaly"} <= kinds
         whats = " | ".join(e["what"] for e in newest["around"])
-        assert "web die (exit 137)" in whats and "maintenance/backup.yml" in whats
+        assert "died (exit 137)" in whats and "maintenance/backup.yml" in whats
+        assert "web health_change (healthy→unhealthy)" in whats
         assert any(e["kind"] == "anomaly" and e["ids"]["anomaly_id"] == ids["mem"] for e in newest["around"])
         assert newest["anomalies_elsewhere"] >= 1
-        assert not [e for e in oldest["around"] if e["kind"] in ("container", "ansible_run")]
+        assert not [e for e in oldest["around"] if e["kind"] in ("container", "ansible.unchanged")]
         points = got["series"]["points"]
         assert max(p[1] for p in points) == 90.0 and min(p[1] for p in points) == 18.0
         assert any(p[2] == 20.0 for p in points)

tests/integration/test_timeline.py

@@ -172,14 +172,15 @@ def test_the_first_fill_writes_what_predates_the_timeline_once(app):
     from steward.core.settings import get_setting, set_setting
     from steward.models.alerts import AlertEvent, AlertOperator, AlertRule
     from steward.models.users import User, UserRole
-    from steward.timeline.service import backfill
+    from steward.timeline.record import record
+    from steward.timeline.service import backfill, fill_sources
 
     async def _go():
         tag = uuid.uuid4().hex[:8]
         name = f"tl-{tag}"
         async with app.db_sessionmaker() as s:
             async with s.begin():
-                before = await get_setting(s, "timeline.backfilled_at")
+                before = await get_setting(s, "timeline.filled")
                 user = User(username=f"tl-{tag}", email=f"tl-{tag}@test", password_hash="x",
                             role=UserRole.admin)
                 s.add(user)
@@ -192,24 +193,31 @@ def test_the_first_fill_writes_what_predates_the_timeline_once(app):
                 # Written straight to the table, as before the timeline existed.
                 s.add(AlertEvent(rule_id=rule.id, from_state="pending", to_state="firing", metric_value=95.0,
                                  transitioned_at=datetime.now(timezone.utc) - timedelta(days=3)))
-                await set_setting(s, "timeline.backfilled_at", "")
+                await set_setting(s, "timeline.filled", {})
+
+        # A plugin's source: its entries are added beside core's, once.
+        async def plugin_source(session, since):
+            return [record(session, type="host.rebooted", at=datetime.now(timezone.utc) - timedelta(days=1),
+                           summary=f"{name}: rebooted for 2m", hosts=[name], ref=("test_fill", name))]
+
+        sources = {"steward": fill_sources()["steward"], "test_plugin": plugin_source}
         try:
-            assert await backfill(app) >= 1
+            assert await backfill(app, sources=sources) >= 2
-            [entry] = await _entries(app, name)
+            got = {e.type: e for e in await _entries(app, name)}
-            assert entry.type == "alert.fired" and entry.area == "storage"
+            assert set(got) == {"alert.fired", "host.rebooted"}
-            assert entry.subject == f"{name}:/"
+            assert got["alert.fired"].area == "storage" and got["alert.fired"].subject == f"{name}:/"
-            assert await backfill(app) == 0                         # once
+            async with app.db_sessionmaker() as s:
+                assert set(await get_setting(s, "timeline.filled")) == {"steward", "test_plugin"}
+            assert await backfill(app, sources=sources) == 0                 # once per source
             async with app.db_sessionmaker() as s:
                 async with s.begin():
-                    await set_setting(s, "timeline.backfilled_at", "")
+                    await set_setting(s, "timeline.filled", {"test_plugin": "done"})
-            await backfill(app)                                      # again: what is covered is skipped
+            assert await backfill(app, sources=sources) == 0                 # again: what is covered is skipped
-            assert len(await _entries(app, name)) == 1
+            assert len(await _entries(app, name)) == 2
         finally:
             async with app.db_sessionmaker() as s:
                 async with s.begin():
-                    await set_setting(s, "timeline.backfilled_at", before or "done")
+                    await set_setting(s, "timeline.filled", before or {"steward": "done"})
-
-    asyncio.run(_go())
 
 
 def test_retention_prunes_only_once_it_is_set(app):

tests/integration/test_timeline_sources.py

@@ -0,0 +1,186 @@
+"""Integration: the plugin sources and anomalies write the timeline as things
+happen. A host away is one entry kept up to date until it is back; a
+container on a new image is a deploy; deaths past the crash-loop count are
+one crash-loop entry; an anomaly's entry follows its review. The database is
+shared with other tests, so each test looks for its own uniquely named host.
+Requires STEWARD_DATABASE_URL.
+"""
+from __future__ import annotations
+
+import asyncio
+import os
+import uuid
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+pytestmark = pytest.mark.integration
+
+
+@pytest.fixture
+def app():
+    if not os.environ.get("STEWARD_DATABASE_URL"):
+        pytest.skip("needs Postgres")
+    from steward.app import create_app
+    return create_app(testing=False)
+
+
+async def _entries(app, host: str):
+    from sqlalchemy import select
+    from steward.models.timeline import TimelineEntry
+    async with app.db_sessionmaker() as s:
+        return (await s.execute(select(TimelineEntry).where(TimelineEntry.hosts.contains([host]))
+                                .order_by(TimelineEntry.at, TimelineEntry.id))).scalars().all()
+
+
+async def _host(app, name: str):
+    from steward.models.hosts import Host
+    async with app.db_sessionmaker() as s:
+        async with s.begin():
+            host = Host(name=name, address="10.7.7.7")
+            s.add(host)
+        return host
+
+
+def test_a_reboot_is_one_entry_that_gets_its_length_when_the_host_is_back(app):
+    from steward.core.plugin_manager import loaded_plugin_submodule
+    downtime = loaded_plugin_submodule("host_agent", "downtime")
+
+    async def _go():
+        name = f"tl-dt-{uuid.uuid4().hex[:8]}"
+        host = await _host(app, name)
+        down = datetime.now(timezone.utc).replace(microsecond=0) - timedelta(minutes=10)
+
+        async def post(events):
+            async with app.db_sessionmaker() as s:
+                async with s.begin():
+                    await downtime.record_events(s, host.id, events, down - timedelta(minutes=1))
+
+        await post([{"kind": "stopping", "at": down.isoformat(), "host_shutdown": True}])
+        [entry] = await _entries(app, name)
+        assert entry.type == "host.rebooted" and entry.until is None
+        assert entry.summary == f"{name}: rebooted, not back yet"
+        back = down + timedelta(minutes=3)
+        await post([{"kind": "started", "at": back.isoformat(),
+                     "boot_at": (down + timedelta(minutes=2)).isoformat()}])
+        [entry] = await _entries(app, name)                      # the same entry, now with its end
+        assert entry.until == back and entry.summary.startswith(f"{name}: rebooted for ")
+        assert entry.ref_type == "host_downtime" and entry.area == "host_os"
+        # Lost contact sorts under Network.
+        await post([{"kind": "lost_contact", "from": (back + timedelta(minutes=1)).isoformat(),
+                     "to": (back + timedelta(minutes=4)).isoformat()}])
+        lost = [e for e in await _entries(app, name) if e.type == "host.lost_contact"]
+        assert len(lost) == 1 and lost[0].area == "network"
+
+    asyncio.run(_go())
+
+
+def test_a_finished_maintenance_job_goes_on_the_timeline(app):
+    from sqlalchemy import text
+    from steward.core.plugin_manager import loaded_plugin_submodule
+    maintenance = loaded_plugin_submodule("host_agent", "maintenance")
+
+    async def _go():
+        name = f"tl-mj-{uuid.uuid4().hex[:8]}"
+        host = await _host(app, name)
+        job_id = str(uuid.uuid4())
+        now = datetime.now(timezone.utc)
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                await s.execute(text(
+                    "INSERT INTO host_maintenance_jobs (id, host_id, state, state_at, reboot_policy, "
+                    "step_seen, step_id, packages, reboot_needed, created_at) "
+                    "VALUES (:id, :h, 'updating', :at, 'if_required', true, :step, NULL, NULL, :at)"),
+                    {"id": job_id, "h": host.id, "at": now - timedelta(minutes=5),
+                     "step": f"{job_id}-update"})
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                assert await maintenance.apply_events(s, host.id, [
+                    {"kind": "maintenance", "id": f"{job_id}-update", "action": "update",
+                     "state": "done", "ok": True, "packages": 4, "reboot_required": False}], now) == 1
+        [entry] = await _entries(app, name)
+        assert entry.type == "maintenance.done"
+        assert (entry.ref_type, entry.ref_id) == ("maintenance_job", job_id)
+        assert entry.summary == f"Maintenance on {name}: Done: 4 packages updated, no reboot needed"
+        assert entry.until == now and entry.area == "host_os"
+
+    asyncio.run(_go())
+
+
+def test_docker_deploys_and_a_crash_loop_as_one_episode(app):
+    from steward.core.capabilities import get_capability
+
+    persist = get_capability("docker.persist_host_samples").fn
+
+    def c(name, image, cid, restarts=0, **kw):
+        return {"name": name, "container_id": cid, "image": image, "status": "running",
+                "cpu_pct": 1.0, "mem_used_pct": 1.0, "mem_used_bytes": 1, "mem_limit_bytes": 2,
+                "ports": [], "started_at": None, "health": None, "restart_count": restarts,
+                "exit_code": 0, "oom_killed": False, **kw}
+
+    async def _go():
+        name = f"tl-dk-{uuid.uuid4().hex[:8]}"
+        host = await _host(app, name)
+        t = datetime.now(timezone.utc) - timedelta(minutes=30)
+
+        async def snap(minutes, containers):
+            async with app.db_sessionmaker() as s:
+                async with s.begin():
+                    await persist(s, host, [(t + timedelta(minutes=minutes), containers)])
+
+        await snap(0, [c("web", "nginx:1.27", "a1"), c("job", "worker:3", "j1")])   # the baseline: nothing
+        assert await _entries(app, name) == []
+        await snap(1, [c("web", "nginx:1.28", "a2"), c("job", "worker:3", "j1")])
+        [deploy] = await _entries(app, name)
+        assert deploy.type == "docker.deployed" and deploy.subject == f"{name}/web"
+        assert deploy.summary == f"web on {name}: nginx:1.27 → nginx:1.28"
+        for i in (1, 2):
+            await snap(1 + i, [c("web", "nginx:1.28", "a2"), c("job", "worker:3", "j1", restarts=i)])
+        singles = [e for e in await _entries(app, name) if e.type == "docker.restarted"]
+        assert len(singles) == 2                                     # recorded, hidden
+        for i in (3, 4):
+            await snap(1 + i, [c("web", "nginx:1.28", "a2"), c("job", "worker:3", "j1", restarts=i)])
+        [loop] = [e for e in await _entries(app, name) if e.type == "docker.crash_loop"]
+        assert loop.detail["count"] == 4 and loop.until == t + timedelta(minutes=5)
+        assert loop.at == singles[0].at
+        assert loop.summary == f"job on {name} crash-looping: 4 restarts or deaths"
+        assert len([e for e in await _entries(app, name) if e.type == "docker.restarted"]) == 2
+
+    asyncio.run(_go())
+
+
+def test_an_anomaly_s_entry_follows_its_review(app):
+    from steward.core.anomalies import REAL, lapse, mark
+    from steward.models.norms import MetricAnomaly
+    from steward.timeline.record import anomaly_changed
+
+    async def _go():
+        name = f"tl-an-{uuid.uuid4().hex[:8]}"
+        await _host(app, name)
+        start = datetime.now(timezone.utc) - timedelta(days=20)
+
+        def anomaly(metric):
+            return MetricAnomaly(source_module="host_agent", resource_name=name, metric_name=metric,
+                                 started_at=start, last_started_at=start, ended_at=start + timedelta(hours=1),
+                                 occurrences=1, peak_level=93.0, peak_at=start, usual_hold=20.0,
+                                 usual_peak=35.0, review="to_review")
+
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                cpu, mem = anomaly("cpu_pct"), anomaly("mem_used_pct")
+                s.add_all([cpu, mem])
+                await anomaly_changed(s, cpu)
+                await anomaly_changed(s, mem)
+        assert {e.type for e in await _entries(app, name)} == {"anomaly.to_review"}
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                await mark(s, cpu.id, REAL, "tester", "the backup hung")
+                assert await lapse(s, datetime.now(timezone.utc), 14) >= 1
+        got = {e.ref_id: e for e in await _entries(app, name)}
+        assert len(got) == 2
+        assert got[cpu.id].type == "anomaly.real" and got[cpu.id].actor == "tester"
+        assert got[cpu.id].summary.endswith(": a real problem")
+        assert got[cpu.id].detail["note"] == "the backup hung"
+        assert got[mem.id].type == "anomaly.lapsed"
+
+    asyncio.run(_go())

tests/plugins/docker/test_timeline.py

@@ -0,0 +1,51 @@
+"""What a snapshot puts on the timeline as a deploy: a unit (a container, or
+its swarm service) running an image it didn't, a container recreated under
+its name, a new container; not a swarm task the scheduler moved here, and
+not a container that only restarted."""
+from types import SimpleNamespace as NS
+
+from plugins.docker.timeline import _deploy_summary, deploys, short_image
+
+
+def _row(name, image, cid, service=None):
+    return NS(name=name, image=image, container_id=cid, service_name=service)
+
+
+def _c(name, image, cid, service=None, status="running"):
+    return {"name": name, "image": image, "container_id": cid, "service_name": service, "status": status}
+
+
+def test_a_new_image_is_a_deploy_and_names_the_old_one():
+    [d] = deploys([_row("web", "nginx:1.27", "a1")], [_c("web", "nginx:1.28", "a2")])
+    assert (d["unit"], d["was"], d["recreated"]) == ("web", "nginx:1.27", False)
+    assert _deploy_summary(d, "h1") == "web on h1: nginx:1.27 → nginx:1.28"
+
+
+def test_a_swarm_service_is_one_unit_and_a_new_build_of_the_same_tag_says_so():
+    old = [_row("app.1.aaa", "reg/steward:dev@sha256:1111aaaa2222bbbb", "x", "steward_app")]
+    new = [_c("app.1.bbb", "reg/steward:dev@sha256:3333cccc4444dddd", "y", "steward_app"),
+           _c("app.2.ccc", "reg/steward:dev@sha256:3333cccc4444dddd", "z", "steward_app")]
+    [d] = deploys(old, new)                                   # one per service, not per task
+    assert d["unit"] == "steward_app" and d["swarm"]
+    assert _deploy_summary(d, "h1") == "steward_app on h1: new build of steward:dev (3333cccc4444)"
+
+
+def test_a_recreated_container_and_a_new_one_are_deploys():
+    [d] = deploys([_row("db", "postgres:16", "c1")], [_c("db", "postgres:16", "c2")])
+    assert d["recreated"] and _deploy_summary(d, "h1") == "db on h1: recreated on postgres:16"
+    [d] = deploys([_row("db", "postgres:16", "c1")], [_c("db", "postgres:16", "c1"), _c("cache", "redis:7", "r1")])
+    assert _deploy_summary(d, "h1") == "cache on h1: started on redis:7"
+
+
+def test_a_moved_swarm_task_a_restart_and_a_stopped_container_are_not():
+    assert deploys([], [_c("app.1.aaa", "img:1", "x", "svc")]) == []           # landed here, new to the host
+    assert deploys([_row("web", "nginx:1", "a1")], [_c("web", "nginx:1", "a1")]) == []
+    assert deploys([_row("web", "nginx:1", "a1")], [_c("web", "nginx:2", "a2", status="exited")]) == []
+    # A swarm task replaced on the same image (a crash, a reschedule) is not a deploy either.
+    assert deploys([_row("app.1.aaa", "img:1", "x", "svc")], [_c("app.1.bbb", "img:1", "y", "svc")]) == []
+
+
+def test_short_image():
+    assert short_image("git.example.com/team/steward:dev@sha256:e12594c948a294693c") == "steward:dev (e12594c948a2)"
+    assert short_image("nginx:1.28") == "nginx:1.28"
+    assert short_image("") == "an unknown image"