feat(docker): per-container size+age log ring rotation [M79 step 4]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 41s
CI / integration (push) Successful in 2m26s
CI / publish (push) Successful in 59s

Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.

- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
  max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
This commit is contained in:
2026-07-19 18:51:12 -04:00
parent a8de3570fe
commit 07a841d91e
4 changed files with 107 additions and 2 deletions
+33 -1
View File
@@ -38,6 +38,8 @@ async def run_docker_retention(
events_days: int,
metrics_raw_days: int,
metrics_rollup_days: int,
logs_retention_days: int = 3,
logs_max_bytes_per_container: int = 5_000_000,
now: datetime | None = None,
) -> dict:
"""Roll up + prune Docker time-series. Returns a counts dict for logging.
@@ -47,18 +49,22 @@ async def run_docker_retention(
re-run is idempotent, then delete those raw rows.
2. Prune rolled-up rows older than the rollup window.
3. Prune docker_events older than the events window.
4. Prune docker_logs with a per-container size+age ring (m79): drop lines
older than the age window, then keep only the newest ~cap bytes per
(host, container).
"""
from datetime import timezone
from sqlalchemy import delete, func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from .models import DockerEvent, DockerMetric, DockerMetricHourly
from .models import DockerEvent, DockerLog, DockerMetric, DockerMetricHourly
if now is None:
now = datetime.now(timezone.utc)
rolled = rolled_rows = events_pruned = rollup_pruned = 0
logs_age_pruned = logs_size_pruned = 0
# ── 1. Roll up raw metrics older than the raw window into hourly buckets ──
raw_cutoff = _rollup_cutoff(now, metrics_raw_days)
@@ -120,9 +126,35 @@ async def run_docker_retention(
)
events_pruned = res.rowcount or 0
# ── 4. Container-log ring: age cutoff, then per-container byte cap (m79) ──
logs_cutoff = now - timedelta(days=logs_retention_days)
res = await session.execute(
delete(DockerLog).where(DockerLog.ts < logs_cutoff)
)
logs_age_pruned = res.rowcount or 0
# Size ring: per (host, container), sum line bytes newest-first; delete a row
# once its strictly-newer siblings already fill the cap. Using the EXCLUSIVE
# prefix (running total minus this row) means the newest row always survives,
# so a single line larger than the cap is never wiped out.
running = func.sum(func.length(DockerLog.line)).over(
partition_by=[DockerLog.host_id, DockerLog.container_name],
order_by=[DockerLog.ts.desc(), DockerLog.id.desc()],
)
prefix_excl = (running - func.length(DockerLog.line)).label("prefix_excl")
ranked = select(DockerLog.id, prefix_excl).subquery()
over_cap = select(ranked.c.id).where(
ranked.c.prefix_excl >= logs_max_bytes_per_container)
res = await session.execute(
delete(DockerLog).where(DockerLog.id.in_(over_cap))
)
logs_size_pruned = res.rowcount or 0
return {
"buckets_rolled": rolled,
"raw_rows_rolled": rolled_rows,
"rollup_pruned": rollup_pruned,
"events_pruned": events_pruned,
"logs_age_pruned": logs_age_pruned,
"logs_size_pruned": logs_size_pruned,
}