feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)

The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.

A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.

get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/core/timeline.md

@@ -1,8 +1,8 @@
 # Timeline
 
-The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, and changes to Steward's own settings. Open it from **Timeline** in the sidebar, from a host's **History** card, or through the `get_timeline` MCP tool.
+The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, maintenance runs and reboots, deploys and crash-loops, anomalies once reviewed, and changes to Steward's own settings. Open it from **Timeline** in the sidebar, from a host's **History** card, or through the `get_timeline` MCP tool.
 
-Each entry stands on its own. An alert event, an Ansible run or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
+Each entry stands on its own. An alert event, an Ansible run, a maintenance job or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
 
 **Source:** `steward/timeline/`
 **Table:** `timeline_entries` (`steward/models/timeline.py`)
@@ -18,14 +18,24 @@ Every source writes its entry in the same transaction as its own record, through
 | Alert events | `core/alerts._add_event` | `alert.fired`, `alert.resolved`, `alert.acknowledged`, `alert.silent`, `alert.resumed` |
 | Ansible runs, when they finish | `ansible/executor.start_run`, and the startup sweep of interrupted runs | `ansible.changed`, `ansible.unchanged`, `ansible.failed`, `ansible.check` |
 | Audit rows | `core/audit.log_audit` | `steward.config`, `steward.access`, `steward.signin`, `steward.backup` |
+| Metric anomalies: one entry each, whose type follows its review | `core/anomalies` (opened, ended, marked, lapsed) | `anomaly.to_review`, `anomaly.real`, `anomaly.normal`, `anomaly.lapsed` |
+| Maintenance jobs, when they end (and again if the host comes back late) | host agent: `maintenance`, `cadence` | `maintenance.done`, `maintenance.failed` |
+| A host away: a reboot, a crash, lost contact, the agent restarting; kept up to date until it ends | host agent: `downtime.record_events` | `host.rebooted`, `host.lost_contact`, `host.agent_restarted` |
+| Container snapshots: a new image, a recreated container, a restart or death | Docker: `ingest.persist_host_docker` → `timeline.from_snapshot` | `docker.deployed`, `docker.crash_loop`, `docker.restarted` |
+
+A source whose record changes after it is written keeps one entry for it (`record.record_or_update`): a downtime gets its length when it ends, an anomaly changes type when it is reviewed.
+
+Docker's units are swarm services where a container has one, so a service's tasks, replaced on every deploy and crash, read as one subject. A swarm task moving to another host is not a deploy. A crash loop uses the same rule as the Docker pages: Settings → Thresholds' deaths within an hour. The death that reaches it opens one `docker.crash_loop` entry, and each later one extends it until the container has been quiet for an hour. The deaths before it stay as hidden `docker.restarted` entries.
 
 Some audit actions are left to another writer, because the event itself is recorded there: starting a run, acknowledging an alert, and starting maintenance. A person's own preferences (time zone, email settings) are not recorded. `catalog.audit_type` holds that mapping.
 
-On its first start, Steward fills the timeline from the last 90 days of alert events, finished runs and audit rows (`service.backfill`). It does this once, and records when in `timeline.backfilled_at`. An entry for a record that is already covered is skipped.
+On its first start, Steward fills the timeline from the last 90 days of records that predate it (`service.backfill`): core's alert events, finished runs and audit rows, and each loaded plugin's own through its `timeline_backfill(session, since)` (the host agent's maintenance jobs and downtime). It fills from each source once and records which in `timeline.filled`, so a plugin loaded later fills on the next daily turn. An entry for a record that is already covered is skipped. Docker has no fill: its events can't tell a deploy from a restart after the fact.
+
+`get_anomaly`'s "what else was going on" reads the timeline for the host or resource, hidden types included, beside each occurrence.
 
 ## Areas and kinds
 
-An entry has a **kind**, which says how it came about: Change, Automation, Alert, Anomaly, Ticket or Steward. Kinds are fixed in `catalog.KINDS`.
+An entry has a **kind**, which says how it came about: Change, Automation, Alert, Anomaly, Ticket, Observed (Steward saw it happen with no rule or person behind it: a reboot, a crash-loop) or Steward. Kinds are fixed in `catalog.KINDS`.
 
 An entry also has an **area**, which says what part of the infrastructure it is about. The areas start as Service, Host OS, Hardware, Storage, Network, Security and Monitoring.
 
@@ -41,8 +51,8 @@ Each entry is written with an area of its own. An alert's area comes from what i
 
 These are applied when the timeline is read, never written onto an entry. A change re-sorts the whole history. A hidden type is still recorded, so showing it again brings its entries back.
 
-Hidden by default: dry runs, playbook runs that changed nothing, and sign-ins.
+Hidden by default: dry runs, playbook runs that changed nothing, sign-ins, single container restarts, the agent restarting, and anomalies not yet reviewed, accepted as normal or lapsed.
 
 ## Adding a type
 
-A plugin registers its types in `setup()` with `catalog.register_type(key, label, kind, area, shown, about)`, and writes entries with `record.record(session, type=…, at=…, summary=…, hosts=…, ref=(kind, id))`. Settings → Timeline lists the new type at once.
+A plugin registers its types in `setup()` with `catalog.register_type(key, label, kind, area, shown, about)`, and the records its entries point to with `catalog.register_ref(ref_type, model)`, so an entry whose record has aged out reads "Details no longer kept". It writes entries with `record.record(session, type=…, at=…, summary=…, hosts=…, ref=(ref_type, id))`, or `record.record_or_update` for a record that changes. It may add `timeline_backfill(session, since)` to its `__init__.py` for records that predate the timeline. Settings → Timeline lists the new type at once. `plugins/host_agent/timeline.py` and `plugins/docker/timeline.py` are worked examples.