bvandeusenandClaude Opus 5.5
329aa15e3d
A change Steward can't see happen (a config edited, a disk swapped) is recorded by whoever made it: what was done, its area, when (in the reader's own zone), the hosts, what it was done to and a detail. Its author or an admin edits or deletes it; recording, editing and deleting are audited and those audit rows stay off the timeline. log_change records one as "<user> via MCP", editable by the token's user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
66 lines
6.8 KiB
Markdown
66 lines
6.8 KiB
Markdown
# Timeline
|
|
|
|
The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, maintenance runs and reboots, deploys and crash-loops, anomalies once reviewed, changes recorded by hand, and changes to Steward's own settings. Open it from **Timeline** in the sidebar, from a host's **History** card, or through the `get_timeline` MCP tool.
|
|
|
|
Each entry stands on its own. An alert event, an Ansible run, a maintenance job or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
|
|
|
|
**Source:** `steward/timeline/`
|
|
**Table:** `timeline_entries` (`steward/models/timeline.py`)
|
|
|
|
---
|
|
|
|
## What is recorded
|
|
|
|
Every source writes its entry in the same transaction as its own record, through `steward/timeline/record.py`:
|
|
|
|
| Source | Written by | Entry types |
|
|
|---|---|---|
|
|
| Alert events | `core/alerts._add_event` | `alert.fired`, `alert.resolved`, `alert.acknowledged`, `alert.silent`, `alert.resumed` |
|
|
| Ansible runs, when they finish | `ansible/executor.start_run`, and the startup sweep of interrupted runs | `ansible.changed`, `ansible.unchanged`, `ansible.failed`, `ansible.check` |
|
|
| Audit rows | `core/audit.log_audit` | `steward.config`, `steward.access`, `steward.signin`, `steward.backup` |
|
|
| Metric anomalies: one entry each, whose type follows its review | `core/anomalies` (opened, ended, marked, lapsed) | `anomaly.to_review`, `anomaly.real`, `anomaly.normal`, `anomaly.lapsed` |
|
|
| Maintenance jobs, when they end (and again if the host comes back late) | host agent: `maintenance`, `cadence` | `maintenance.done`, `maintenance.failed` |
|
|
| A host away: a reboot, a crash, lost contact, the agent restarting; kept up to date until it ends | host agent: `downtime.record_events` | `host.rebooted`, `host.lost_contact`, `host.agent_restarted` |
|
|
| Container snapshots: a new image, a recreated container, a restart or death | Docker: `ingest.persist_host_docker` → `timeline.from_snapshot` | `docker.deployed`, `docker.crash_loop`, `docker.restarted` |
|
|
| A change recorded by hand, from the Timeline page, a host's History card or MCP `log_change` | `timeline/changes.py` | `change.recorded` |
|
|
|
|
A source whose record changes after it is written keeps one entry for it (`record.record_or_update`): a downtime gets its length when it ends, an anomaly changes type when it is reviewed.
|
|
|
|
Docker's units are swarm services where a container has one, so a service's tasks, replaced on every deploy and crash, read as one subject. A swarm task moving to another host is not a deploy. A crash loop uses the same rule as the Docker pages: Settings → Thresholds' deaths within an hour. The death that reaches it opens one `docker.crash_loop` entry, and each later one extends it until the container has been quiet for an hour. The deaths before it stay as hidden `docker.restarted` entries.
|
|
|
|
Some audit actions are left to another writer, because the event itself is recorded there: starting a run, acknowledging an alert, starting maintenance, and recording a change by hand. A person's own preferences (time zone, email settings) are not recorded. `catalog.audit_type` holds that mapping.
|
|
|
|
On its first start, Steward fills the timeline from the last 90 days of records that predate it (`service.backfill`): core's alert events, finished runs and audit rows, and each loaded plugin's own through its `timeline_backfill(session, since)` (the host agent's maintenance jobs and downtime). It fills from each source once and records which in `timeline.filled`, so a plugin loaded later fills on the next daily turn. An entry for a record that is already covered is skipped. Docker has no fill: its events can't tell a deploy from a restart after the fact.
|
|
|
|
`get_anomaly`'s "what else was going on" reads the timeline for the host or resource, hidden types included, beside each occurrence.
|
|
|
|
## Recording a change by hand
|
|
|
|
Steward can't see every change happen: a config edited on a host, a disk swapped, a service moved. Whoever made it records it, with **Record a change** on the Timeline page or a host's History card, or the `log_change` MCP tool. It says what was done, the area it is about, when, the hosts it was done on (none for a change not on any one host), optionally what it was done to (a container, a file) and a longer detail.
|
|
|
|
An operator or admin can record one. Its author can edit or delete it from its **Edit** link, and so can an admin. One recorded through MCP is shown as by "<user> via MCP" and is editable by that token's user. Recording, editing and deleting are audited (`timeline.change_recorded`, `…_edited`, `…_deleted`), and those audit rows stay off the timeline, which holds the change itself.
|
|
|
|
## Areas and kinds
|
|
|
|
An entry has a **kind**, which says how it came about: Change, Automation, Alert, Anomaly, Ticket, Observed (Steward saw it happen with no rule or person behind it: a reboot, a crash-loop) or Steward. Kinds are fixed in `catalog.KINDS`.
|
|
|
|
An entry also has an **area**, which says what part of the infrastructure it is about. The areas start as Service, Host OS, Hardware, Storage, Network, Security and Monitoring.
|
|
|
|
Each entry is written with an area of its own. An alert's area comes from what its rule watches (`catalog.area_of_metric`): drive health is Hardware, a mount is Storage, an HTTP monitor is Service. A change recorded by hand has the area its author picked. Other entries use their type's area.
|
|
|
|
## Settings → Timeline
|
|
|
|
| Setting | Key | Default |
|
|
|---|---|---|
|
|
| Areas, in order: rename, add, or remove one and move its entries to another | `timeline.areas`, `timeline.area_moved` | the seven above |
|
|
| Each type's area ("As recorded" keeps each entry's own) and whether it is shown | `timeline.types` | from the catalog |
|
|
| Keep for, in days (0 keeps everything) | `timeline.retention_days` | 0 |
|
|
|
|
These are applied when the timeline is read, never written onto an entry. A change re-sorts the whole history. A hidden type is still recorded, so showing it again brings its entries back.
|
|
|
|
Hidden by default: dry runs, playbook runs that changed nothing, sign-ins, single container restarts, the agent restarting, and anomalies not yet reviewed, accepted as normal or lapsed.
|
|
|
|
## Adding a type
|
|
|
|
A plugin registers its types in `setup()` with `catalog.register_type(key, label, kind, area, shown, about)`, and the records its entries point to with `catalog.register_ref(ref_type, model)`, so an entry whose record has aged out reads "Details no longer kept". It writes entries with `record.record(session, type=…, at=…, summary=…, hosts=…, ref=(ref_type, id))`, or `record.record_or_update` for a record that changes. It may add `timeline_backfill(session, since)` to its `__init__.py` for records that predate the timeline. Settings → Timeline lists the new type at once. `plugins/host_agent/timeline.py` and `plugins/docker/timeline.py` are worked examples.
|