feat(timeline): maintenance, reboots, deploys, crash-loops and reviewed anomalies on the timeline (#6349)
The host agent writes each maintenance job once it ends, and each stretch a host was away (a reboot, a crash, lost contact, the agent restarting), kept up to date until it is back. Docker judges each snapshot: a unit on a new image, or a container recreated, is a deploy; deaths past the Thresholds crash-loop count are one crash-loop entry that later deaths extend; single restarts are recorded but hidden. An anomaly has one entry whose type follows its review: only those judged real are shown by default. A source whose record changes keeps one entry for it (record_or_update). Plugins register the records their entries point to, so an aged-out job or downtime reads "Details no longer kept". The first fill runs once per source (timeline.filled), the host agent's jobs and downtime included. get_anomaly's "what else was going on" reads the timeline for the host or resource instead of querying alerts, runs, maintenance, downtime and container deaths itself; Docker's own source keeps only starts, stops and health changes. A new kind, Observed, is for what Steward saw happen with no rule or person behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -354,6 +354,17 @@ async def _host_name(session, host_id: str) -> str:
|
||||
return host.name if host is not None else host_id
|
||||
|
||||
|
||||
async def _moved(session, job, now: datetime) -> None:
|
||||
"""After a job moved: its window follows it, and once it ends it goes on
|
||||
the timeline."""
|
||||
host_name = await _host_name(session, job.host_id)
|
||||
await _window(session, job, host_name, now)
|
||||
if job.state in FINISHED:
|
||||
from . import timeline
|
||||
await timeline.write(session, f"maintenance job {job.id}",
|
||||
timeline.job_entry(session, job, host_name))
|
||||
|
||||
|
||||
async def apply_events(session, host_id: str, events, now: datetime) -> int:
|
||||
"""Apply one post's `maintenance` results and `started` boots to the jobs
|
||||
they belong to; returns how many jobs changed. A result is matched by its
|
||||
@@ -382,7 +393,7 @@ async def apply_events(session, host_id: str, events, now: datetime) -> int:
|
||||
if apply_boot(job, boot_at, now):
|
||||
changed.add(job)
|
||||
for job in changed:
|
||||
await _window(session, job, await _host_name(session, job.host_id), now)
|
||||
await _moved(session, job, now)
|
||||
return len(changed)
|
||||
|
||||
|
||||
@@ -440,7 +451,7 @@ async def tick(session, now: datetime) -> int:
|
||||
gave_up += 1
|
||||
moved = True
|
||||
if moved:
|
||||
await _window(session, job, await _host_name(session, job.host_id), now)
|
||||
await _moved(session, job, now)
|
||||
return gave_up
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user