bvandeusenandClaude Opus 5.5
2b97c36fd4
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.8 KiB
Timeline
The timeline is one list of everything that happened to the infrastructure, newest first: alerts firing and clearing, playbook runs and what they changed, maintenance runs and reboots, deploys and crash-loops, anomalies once reviewed, and changes to Steward's own settings. Open it from Timeline in the sidebar, from a host's History card, or through the get_timeline MCP tool.
Each entry stands on its own. An alert event, an Ansible run, a maintenance job or an audit row is pruned after its own retention, but the entry for it stays. Its link then reads "Details no longer kept".
Source: steward/timeline/
Table: timeline_entries (steward/models/timeline.py)
What is recorded
Every source writes its entry in the same transaction as its own record, through steward/timeline/record.py:
| Source | Written by | Entry types |
|---|---|---|
| Alert events | core/alerts._add_event |
alert.fired, alert.resolved, alert.acknowledged, alert.silent, alert.resumed |
| Ansible runs, when they finish | ansible/executor.start_run, and the startup sweep of interrupted runs |
ansible.changed, ansible.unchanged, ansible.failed, ansible.check |
| Audit rows | core/audit.log_audit |
steward.config, steward.access, steward.signin, steward.backup |
| Metric anomalies: one entry each, whose type follows its review | core/anomalies (opened, ended, marked, lapsed) |
anomaly.to_review, anomaly.real, anomaly.normal, anomaly.lapsed |
| Maintenance jobs, when they end (and again if the host comes back late) | host agent: maintenance, cadence |
maintenance.done, maintenance.failed |
| A host away: a reboot, a crash, lost contact, the agent restarting; kept up to date until it ends | host agent: downtime.record_events |
host.rebooted, host.lost_contact, host.agent_restarted |
| Container snapshots: a new image, a recreated container, a restart or death | Docker: ingest.persist_host_docker → timeline.from_snapshot |
docker.deployed, docker.crash_loop, docker.restarted |
A source whose record changes after it is written keeps one entry for it (record.record_or_update): a downtime gets its length when it ends, an anomaly changes type when it is reviewed.
Docker's units are swarm services where a container has one, so a service's tasks, replaced on every deploy and crash, read as one subject. A swarm task moving to another host is not a deploy. A crash loop uses the same rule as the Docker pages: Settings → Thresholds' deaths within an hour. The death that reaches it opens one docker.crash_loop entry, and each later one extends it until the container has been quiet for an hour. The deaths before it stay as hidden docker.restarted entries.
Some audit actions are left to another writer, because the event itself is recorded there: starting a run, acknowledging an alert, and starting maintenance. A person's own preferences (time zone, email settings) are not recorded. catalog.audit_type holds that mapping.
On its first start, Steward fills the timeline from the last 90 days of records that predate it (service.backfill): core's alert events, finished runs and audit rows, and each loaded plugin's own through its timeline_backfill(session, since) (the host agent's maintenance jobs and downtime). It fills from each source once and records which in timeline.filled, so a plugin loaded later fills on the next daily turn. An entry for a record that is already covered is skipped. Docker has no fill: its events can't tell a deploy from a restart after the fact.
get_anomaly's "what else was going on" reads the timeline for the host or resource, hidden types included, beside each occurrence.
Areas and kinds
An entry has a kind, which says how it came about: Change, Automation, Alert, Anomaly, Ticket, Observed (Steward saw it happen with no rule or person behind it: a reboot, a crash-loop) or Steward. Kinds are fixed in catalog.KINDS.
An entry also has an area, which says what part of the infrastructure it is about. The areas start as Service, Host OS, Hardware, Storage, Network, Security and Monitoring.
Each entry is written with an area of its own. An alert's area comes from what its rule watches (catalog.area_of_metric): drive health is Hardware, a mount is Storage, an HTTP monitor is Service. Other entries use their type's area.
Settings → Timeline
| Setting | Key | Default |
|---|---|---|
| Areas, in order: rename, add, or remove one and move its entries to another | timeline.areas, timeline.area_moved |
the seven above |
| Each type's area ("As recorded" keeps each entry's own) and whether it is shown | timeline.types |
from the catalog |
| Keep for, in days (0 keeps everything) | timeline.retention_days |
0 |
These are applied when the timeline is read, never written onto an entry. A change re-sorts the whole history. A hidden type is still recorded, so showing it again brings its entries back.
Hidden by default: dry runs, playbook runs that changed nothing, sign-ins, single container restarts, the agent restarting, and anomalies not yet reviewed, accepted as normal or lapsed.
Adding a type
A plugin registers its types in setup() with catalog.register_type(key, label, kind, area, shown, about), and the records its entries point to with catalog.register_ref(ref_type, model), so an entry whose record has aged out reads "Details no longer kept". It writes entries with record.record(session, type=…, at=…, summary=…, hosts=…, ref=(ref_type, id)), or record.record_or_update for a record that changes. It may add timeline_backfill(session, since) to its __init__.py for records that predate the timeline. Settings → Timeline lists the new type at once. plugins/host_agent/timeline.py and plugins/docker/timeline.py are worked examples.