docs(host-agent): document container-log collection [M79 step 6]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 42s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 5s

Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config
keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the
agent internals, and a wire-format design point on docker_logs riding in the same
push (with server-side toggle/exclude enforcement + the per-container ring).

The end-to-end chain (store → rotate → query + ingest enforcement) is already
covered by the integration tests added across steps 2–5, so no capstone test is
added. AGENT_VERSION bump (1.7.0) landed in step 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
This commit is contained in:
2026-07-19 19:07:53 -04:00
parent 414c8efa98
commit 8c2d994e4c
+16 -1
View File
@@ -74,8 +74,15 @@ token = a1b2c3d4...
interval_seconds = 30 interval_seconds = 30
hostname = myhost # optional; defaults to uname -n hostname = myhost # optional; defaults to uname -n
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.) mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
docker_logs_enabled = true # optional; on by default — per-host container-log kill-switch
docker_log_exclude = watchtower, noisy-svc # optional; container names whose logs this host skips collecting
``` ```
Container-log collection is on by default (no config needed). The two keys above
are the per-host opt-outs; the operator-facing global toggle + exclude live in
Settings and are enforced server-side (the push model has no channel to command
an agent), so they don't require touching a host's conf.
### Agent internals (function list, not classes) ### Agent internals (function list, not classes)
- `read_config(path)` — parses the conf file into a dict. - `read_config(path)` — parses the conf file into a dict.
@@ -85,10 +92,17 @@ mounts = /, /mnt/data # optional; defaults to all real mounts (excludin
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`. - `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int). - `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
- `collect_metadata()``os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached. - `collect_metadata()``os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
container, fetches new log lines over the Docker socket with an incremental
`since` cursor kept per container in `state` (a container's first interval
seeds from a short tail). Demuxes the multiplexed stream, parses each line's
RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
a truncation; the deferred lines come next interval). Folded into the sample as
`docker_logs`; omitted when empty.
- `build_payload()` — assembles a snapshot from all collectors into one dict. - `build_payload()` — assembles a snapshot from all collectors into one dict.
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure. - `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full. - `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. - `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping. **Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
@@ -255,6 +269,7 @@ Content-Type: application/json
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts. - **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent. - **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it. - **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.
### Server expansion into `PluginMetric` rows ### Server expansion into `PluginMetric` rows