fix(telemetry): a surface that stopped recording is not one that never ran (#3720)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 34s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 24s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 34s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 24s
`out["sources"]` was built only from the windowed aggregate, so a source with rows in `retrieval_logs` but none inside the window got no bucket at all. Absent is exactly how a source that never existed renders, so a surface that WAS recording and went silent became unreadable — #2663 one level up, the failure that looks like the correct answer. Two queries at different scopes, and only one shaped the output. `_complete_from` reads all-time and knows every source the table has ever held; the windowed loop dropped whatever it did not return. Every such source now gets a zero bucket. Zero is a real measurement here rather than a manufactured one: the all-time query proves the source was recording, and it made no calls across a window it fully covers. No `covers_window` special case is needed either — a source whose first row fell after `since` would have that row IN the window and already hold a bucket, so anything reaching this branch began before it. The counts are 0 and everything else is null. A sampled distribution is not the same claim as a call count, and rendering p50 as 0.0 for a source nobody sampled would assert a measurement — #3311's mistake, in the readout built to prevent it. Found while fixing #3712's fixture, which failed with KeyError for this exact reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -213,6 +213,15 @@ def _bucket(rows: list) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# The aggregate row Postgres would have returned for a source with no rows in
|
||||
# the window: nothing counted, nothing scored. Positional, matching the SELECT
|
||||
# `_bucket` unpacks — calls, zero, cleared, p10, p50, p90, min, max, avg_n,
|
||||
# dur, measured, supp_calls, supp_zero. The three counts are 0 because zero
|
||||
# calls is a real observation; everything else is None because a distribution
|
||||
# nobody sampled has no value, and rendering it as 0.0 would state one.
|
||||
_NO_ROWS_IN_WINDOW = [0, 0, 0, None, None, None, None, None, None, None, 0, 0, 0]
|
||||
|
||||
|
||||
def _round(v, places: int = 4):
|
||||
return None if v is None else round(float(v), places)
|
||||
|
||||
@@ -375,6 +384,25 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
bucket.update(_coverage(log_complete.get(source), since))
|
||||
out["sources"][source] = bucket
|
||||
|
||||
# A source with rows in the table but NONE in this window would
|
||||
# otherwise be absent from the readout — and absent is exactly how
|
||||
# a source that never existed renders, so a surface that WAS
|
||||
# recording and went silent is unreadable (#3720). That is #2663
|
||||
# one level up: the failure that looks like the correct answer.
|
||||
#
|
||||
# Zero here is a real measurement, not a manufactured one. The
|
||||
# all-time query proves the source was recording, and it made no
|
||||
# calls across a window it fully covers — which is why no
|
||||
# `covers_window` special case is needed: a source whose first row
|
||||
# fell after `since` would have that row IN the window and already
|
||||
# hold a bucket, so anything reaching here began before it.
|
||||
for src, first_row in log_complete.items():
|
||||
if src == "*" or first_row is None or src in out["sources"]:
|
||||
continue
|
||||
quiet = _bucket(list(_NO_ROWS_IN_WINDOW))
|
||||
quiet.update(_coverage(first_row, since))
|
||||
out["sources"][src] = quiet
|
||||
|
||||
# The corpus side, at its own grain. `ambient` mirrors
|
||||
# note_usage.usage_for_notes: an ambient surfacing was not a scored
|
||||
# CHOICE, so folding it into pull-through would understate it.
|
||||
|
||||
Reference in New Issue
Block a user