From 8c2d994e4ccc7b97b60dc48aeeb1ba27285f71c3 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sun, 19 Jul 2026 19:07:53 -0400 Subject: [PATCH] docs(host-agent): document container-log collection [M79 step 6] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the agent internals, and a wire-format design point on docker_logs riding in the same push (with server-side toggle/exclude enforcement + the per-container ring). The end-to-end chain (store → rotate → query + ingest enforcement) is already covered by the integration tests added across steps 2–5, so no capstone test is added. AGENT_VERSION bump (1.7.0) landed in step 1. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C --- docs/plugins/host-agent-design.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/plugins/host-agent-design.md b/docs/plugins/host-agent-design.md index ef6611a..e81e5bd 100644 --- a/docs/plugins/host-agent-design.md +++ b/docs/plugins/host-agent-design.md @@ -74,8 +74,15 @@ token = a1b2c3d4... interval_seconds = 30 hostname = myhost # optional; defaults to uname -n mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.) +docker_logs_enabled = true # optional; on by default — per-host container-log kill-switch +docker_log_exclude = watchtower, noisy-svc # optional; container names whose logs this host skips collecting ``` +Container-log collection is on by default (no config needed). The two keys above +are the per-host opt-outs; the operator-facing global toggle + exclude live in +Settings and are enforced server-side (the push model has no channel to command +an agent), so they don't require touching a host's conf. + ### Agent internals (function list, not classes) - `read_config(path)` — parses the conf file into a dict. @@ -85,10 +92,17 @@ mounts = /, /mnt/data # optional; defaults to all real mounts (excludin - `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`. - `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int). - `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached. +- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running + container, fetches new log lines over the Docker socket with an incremental + `since` cursor kept per container in `state` (a container's first interval + seeds from a short tail). Demuxes the multiplexed stream, parses each line's + RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records + a truncation; the deferred lines come next interval). Folded into the sample as + `docker_logs`; omitted when empty. - `build_payload()` — assembles a snapshot from all collectors into one dict. - `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure. - `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full. -- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. +- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped). **Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping. @@ -255,6 +269,7 @@ Content-Type: application/json - **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts. - **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent. - **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it. +- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new. ### Server expansion into `PluginMetric` rows