docs(host-agent): document container-log collection [M79 step 6]
Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the agent internals, and a wire-format design point on docker_logs riding in the same push (with server-side toggle/exclude enforcement + the per-container ring). The end-to-end chain (store → rotate → query + ingest enforcement) is already covered by the integration tests added across steps 2–5, so no capstone test is added. AGENT_VERSION bump (1.7.0) landed in step 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
This commit is contained in:
@@ -74,8 +74,15 @@ token = a1b2c3d4...
|
|||||||
interval_seconds = 30
|
interval_seconds = 30
|
||||||
hostname = myhost # optional; defaults to uname -n
|
hostname = myhost # optional; defaults to uname -n
|
||||||
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
|
mounts = /, /mnt/data # optional; defaults to all real mounts (excluding tmpfs/devtmpfs/etc.)
|
||||||
|
docker_logs_enabled = true # optional; on by default — per-host container-log kill-switch
|
||||||
|
docker_log_exclude = watchtower, noisy-svc # optional; container names whose logs this host skips collecting
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Container-log collection is on by default (no config needed). The two keys above
|
||||||
|
are the per-host opt-outs; the operator-facing global toggle + exclude live in
|
||||||
|
Settings and are enforced server-side (the push model has no channel to command
|
||||||
|
an agent), so they don't require touching a host's conf.
|
||||||
|
|
||||||
### Agent internals (function list, not classes)
|
### Agent internals (function list, not classes)
|
||||||
|
|
||||||
- `read_config(path)` — parses the conf file into a dict.
|
- `read_config(path)` — parses the conf file into a dict.
|
||||||
@@ -85,10 +92,17 @@ mounts = /, /mnt/data # optional; defaults to all real mounts (excludin
|
|||||||
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
|
- `collect_load()` — reads `/proc/loadavg`, returns `[1m, 5m, 15m]`.
|
||||||
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
|
- `collect_uptime()` — reads `/proc/uptime`, returns seconds since boot (int).
|
||||||
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
|
- `collect_metadata()` — `os.uname()` for kernel + arch, `/etc/os-release` for distro. Called once at startup and cached.
|
||||||
|
- `collect_docker_logs(socket, containers, state, exclude)` — (m79) per running
|
||||||
|
container, fetches new log lines over the Docker socket with an incremental
|
||||||
|
`since` cursor kept per container in `state` (a container's first interval
|
||||||
|
seeds from a short tail). Demuxes the multiplexed stream, parses each line's
|
||||||
|
RFC3339 timestamp, caps the whole batch at a byte limit (a marker line records
|
||||||
|
a truncation; the deferred lines come next interval). Folded into the sample as
|
||||||
|
`docker_logs`; omitted when empty.
|
||||||
- `build_payload()` — assembles a snapshot from all collectors into one dict.
|
- `build_payload()` — assembles a snapshot from all collectors into one dict.
|
||||||
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
|
- `post_payload(url, token, payloads)` — POSTs a list of samples, returns success/failure.
|
||||||
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
|
- `RingBuffer(maxlen=20)` — tiny FIFO wrapper, drops oldest when full.
|
||||||
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer.
|
- `main_loop()` — the 30s loop: collect → try POST → on failure push to buffer + backoff → on success flush buffer. Container logs are stripped from a sample before it's buffered (metrics survive an outage; stale logs are dropped).
|
||||||
|
|
||||||
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
|
**Target: ~300 lines total including docstrings.** More than that is a smell that the agent is over-scoping.
|
||||||
|
|
||||||
@@ -255,6 +269,7 @@ Content-Type: application/json
|
|||||||
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
|
- **`metadata` is sent on every POST**, not just on change. Server-side diff detects actual changes and only writes on change. Cost per POST is one dict — negligible. Benefit: server can cleanly detect agent restarts.
|
||||||
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
|
- **Raw bytes, not percentages, for memory and storage.** Percentages are derived server-side. Changing the "what counts as used" math doesn't require re-releasing the agent.
|
||||||
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
|
- **CPU is the one exception** — reported as a percentage because it's inherently a derivative (delta over time), not a snapshot. The agent must sample twice to compute it.
|
||||||
|
- **Container logs (m79) ride in the same push** as `docker_logs`: a list of `{container, stream, ts, line}` records — incremental since the previous interval. This pushes the *same direction* as metrics, so batched log history needs no inbound channel (only sub-second live-follow would). The server ingests them into `docker_logs`, enforces the global toggle + exclude list on ingest, and bounds storage with a per-container size+age ring. `docker_logs` is omitted when there's nothing new.
|
||||||
|
|
||||||
### Server expansion into `PluginMetric` rows
|
### Server expansion into `PluginMetric` rows
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user