Commit Graph
4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 2f208a84fb fix(host_agent): the server writes all five samples an agent post carries (#5181)
MAX_SAMPLES_PER_INGEST was 2 while agent 1.10 drains a backlog five samples a
post, so the server dropped the oldest three of every such post. Posts now
answer in 0.4-1.3 s on dev, including the boot cleanup window, so five fit
the agent's 10 s. A test holds the server cap at or above the agent's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 21:05:22 -04:00
bvandeusenandClaude Opus 5.5 385a579afe fix(host_agent): an ingest writes the newest five samples, so a full backlog can land (#5149)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 46s
CI / unit (push) Successful in 1m8s
CI / publish (push) Successful in 1m13s
Measured on dev (f802c9f): every dropped ingest carried samples=21,
metric_rows≈4,200 and never finished the INSERT flush inside the agent's
10 s ("completed: parse=0.1s"). The rollback left the agent's ring buffer
full, so each later post was the same size and failed the same way: no
host's metrics have landed since.

An ingest now keeps the newest MAX_SAMPLES_PER_INGEST (5) samples by ts,
logs how many it left out, and returns samples_dropped. The older samples
are the ones the ring buffer would have overwritten anyway. Why the
database writes so few rows a second is not yet known; the slow-ingest
line now measures it on a post that finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:46:25 -04:00
bvandeusenandClaude Opus 5.5 f802c9f153 feat(host_agent): a dropped ingest says which step of the metric write it stalled in (#5149)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 48s
CI / unit (push) Successful in 1m11s
CI / publish (push) Successful in 1m8s
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:34:54 -04:00
bvandeusenandClaude Opus 5.5 fc9730a2e5 feat(host_agent): say where a slow ingest spends its time, and how far a dropped one got (#5149)
CI / publish (push) Successful in 6s
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 45s
CI / unit (push) Successful in 1m9s
Every host-agent ingest in the service logs ends at ~10 s with no status:
the agent gives up at its urlopen timeout (agent.py) before Steward answers.
Nothing says which part of ingest is slow, so this measures it:

- each phase (parse, metrics with alert evaluation, docker persist,
  commit) is timed, and a request past 2 s logs one line with the phases,
  sample count, metric rows, containers and log lines;
- a request the server drops when the agent hangs up (CancelledError) logs
  how long it ran and which phases completed, then re-raises.

The fix follows once the logs say which phase it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:28:30 -04:00