fix(host_agent): an ingest writes the newest five samples, so a full backlog can land (#5149)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 46s
CI / unit (push) Successful in 1m8s
CI / publish (push) Successful in 1m13s

Measured on dev (f802c9f): every dropped ingest carried samples=21,
metric_rows≈4,200 and never finished the INSERT flush inside the agent's
10 s ("completed: parse=0.1s"). The rollback left the agent's ring buffer
full, so each later post was the same size and failed the same way: no
host's metrics have landed since.

An ingest now keeps the newest MAX_SAMPLES_PER_INGEST (5) samples by ts,
logs how many it left out, and returns samples_dropped. The older samples
are the ones the ring buffer would have overwritten anyway. Why the
database writes so few rows a second is not yet known; the slow-ingest
line now measures it on a post that finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 18:46:25 -04:00
co-authored by Claude Opus 5.5
parent d8bacc7149
commit 385a579afe
2 changed files with 41 additions and 1 deletions
@@ -56,3 +56,20 @@ async def test_a_dropped_ingest_says_how_far_it_got(app, monkeypatch, caplog):
with caplog.at_level(logging.WARNING):
phases.cancelled("abcd1234")
assert "dropped after 10.1s for token=abcd1234… (completed: parse=0.2s) samples=21 metric_rows=840" in caplog.text
def _s(minute):
return {"ts": f"2026-10-06T12:{minute:02d}:00+00:00", "cpu_pct": 1.0}
def test_a_full_backlog_is_cut_to_its_newest_samples():
# Sent oldest first (the agent's buffer, then the new sample) or not.
backlog = [_s(m) for m in range(21)]
kept, dropped = routes.newest_samples(list(reversed(backlog)), 5)
assert [s["ts"] for s in kept] == [_s(m)["ts"] for m in range(16, 21)]
assert dropped == 16
def test_a_small_post_is_left_alone():
posted = [_s(3), _s(1)]
assert routes.newest_samples(posted, 5) == (posted, 0)