feat(docker): persist pushed container logs into docker_logs [M79 step 3]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 41s
CI / integration (push) Successful in 2m25s
CI / publish (push) Successful in 8s

Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.

- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
  records (drops the _steward truncation marker, malformed + lineless records,
  normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
This commit is contained in:
2026-07-19 18:46:31 -04:00
parent 45565b2c01
commit a8de3570fe
4 changed files with 126 additions and 4 deletions
+47 -2
View File
@@ -92,6 +92,46 @@ def _derive_events(old_state: dict, new_containers: list) -> list:
return events
def _log_rows(log_batches, host_id: str):
"""Pure: flatten the agent's log batches → docker_logs row kwargs (no DB).
`log_batches` is a list of (recorded_at, records) where each record is
{"container", "stream", "ts", "line"}. A line's own Docker timestamp wins;
recorded_at is the fallback when the agent couldn't parse one. Drops the
agent's advisory truncation marker (container "_steward" — it signals a
deferral, not a real container), malformed records, and lineless records;
normalises an unknown stream to stdout. Unit-testable in isolation.
"""
for recorded_at, records in log_batches:
if not isinstance(records, list):
continue
for rec in records:
if not isinstance(rec, dict):
continue
name = rec.get("container")
line = rec.get("line")
if not name or name == "_steward" or line is None:
continue
ts = _parse_started_at(rec.get("ts")) or recorded_at
stream = rec.get("stream")
if stream not in ("stdout", "stderr"):
stream = "stdout"
yield {"host_id": host_id, "container_name": str(name)[:255],
"ts": ts, "stream": stream, "line": str(line)}
async def _persist_logs(session, host, log_batches) -> None:
"""Append pushed container log lines (one row per line) for this host.
Time-series / append-only — the per-container size+age ring (retention)
bounds growth, so a chatty container just keeps a shorter window.
"""
from .models import DockerLog
for row in _log_rows(log_batches, host.id):
session.add(DockerLog(**row))
async def _persist_swarm(session, host, swarm: dict) -> None:
"""Upsert this manager's swarm topology; drop rows no longer reported.
@@ -201,7 +241,8 @@ async def _persist_disk(session, host, disk: dict) -> None:
await session.execute(stale)
async def persist_host_docker(session, host, snapshots, swarm=None, disk=None) -> None:
async def persist_host_docker(session, host, snapshots, swarm=None, disk=None,
logs=None) -> None:
"""Upsert containers + time-series + lifecycle events + swarm for one host.
`snapshots` is a list of (recorded_at: datetime, containers: list[dict]) —
@@ -211,7 +252,9 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None) -
alert pipeline, and lifecycle-event derivation. `swarm` is the newest
sample's swarm object (or None off managers) — persisted when present.
`disk` is the newest sample's /system/df summary (or None on Docker-less
hosts) — persisted when present.
hosts) — persisted when present. `logs` is a list of (recorded_at, records)
log batches (m79) — appended one row per line when present, independent of
whether this sample also carried container metrics.
"""
from steward.core.alerts import record_metric
from .models import DockerContainer, DockerEvent, DockerMetric
@@ -220,6 +263,8 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None) -
await _persist_swarm(session, host, swarm)
if disk is not None:
await _persist_disk(session, host, disk)
if logs:
await _persist_logs(session, host, logs)
if not snapshots:
return