fix(docker): a container that crashes between two samples leaves a die event and its last log lines (#5049)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 46s
CI / unit (push) Successful in 1m10s
CI / publish (push) Successful in 8s

Agent 1.11.0 sends finished_at (State.FinishedAt) and reads a stopped
container's logs once per stop: since its cursor if it was being read, or
its last 200 lines when first seen already crashed (non-zero exit, stopped
in the last 10 min). Cursors and finals are forgotten once a container is no
longer listed.

Ingest gives a container first seen already exited non-zero a die event, and
dates die/stop events by finished_at when the agent sends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 20:58:46 -04:00
co-authored by Claude Opus 5.5
parent 4978d28a51
commit 326a9cd254
4 changed files with 151 additions and 26 deletions
@@ -43,6 +43,22 @@ def test_die_on_nonzero_exit():
assert events == [("job", "die", "exit 137")]
@pytest.mark.parametrize("status,exit_code,events", [
("exited", 1, [("codex_db.1.x", "die", "exit 1")]), # crashed between samples
("dead", 137, [("codex_db.1.x", "die", "exit 137")]),
("exited", 0, []), # a one-shot that finished
("exited", None, []), # inspect failed: no claim
("created", 1, []), # never ran
])
def test_new_container_first_seen_down(status, exit_code, events):
# #5049: a task that starts and exits between two samples is first seen
# already down; a non-zero exit is the death the samples missed.
old = {"other": {"status": "running", "health": None, "oom_killed": False, "exit_code": None}}
got = _derive_events(old, [_c("other", "running"),
_c("codex_db.1.x", status, exit_code=exit_code)])
assert got == events
def test_oom_on_flip_to_true():
old = {"hog": {"status": "running", "health": None, "oom_killed": False, "exit_code": None}}
# OOM-killed → also no longer running with a non-zero exit, so die + oom both fire.