fix(telemetry): silence is only an answer where the ledger was watching (#4268)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 59s
CI & Build / Python tests (push) Successful in 1m44s
CI & Build / Build & push image (push) Successful in 31s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 59s
CI & Build / Python tests (push) Successful in 1m44s
CI & Build / Build & push image (push) Successful in 31s
`band_hugs_floor` compares an arm's weakest tenth against its floor, and #4225 made it SUSPEND rather than soften when the floor moved inside the window — because across a change the scores and the bar come from two different populations. It reads `floor_moves_since`, which answers "did this arm's floor move", and it read an empty answer as "no, it held steady". Those are the same answer only where the ledger was watching. Before an arm's first floor row there is nothing to move, nothing to report, and no way to tell a steady floor from an unrecorded one. The suspension was reading absence of evidence as evidence of absence, and the symptom is the one #4225 documented: a band "-0.0208 above its floor" — an impossible negative distance, printed with the suspension silent. Every install passes through this. The ledger's first row for an arm is written when that install first boots the release that records baselines, so any window longer than the install is old reaches back past it. The lowered-floor direction hides: the gap comes out comfortably positive and reads as a clean bill of health. `floor_history_gaps(since)` answers the question its companion cannot: which arms' floor history does not REACH the start of the window. Arms come from `surface_names()`, not from the ledger — an arm the ledger has never heard of is exactly the one at risk, so it cannot be the ledger that decides which arms get asked about. Baselines count here, which is the one place the two deliberately disagree. A baseline records a default without changing it, so it is not a move and `floor_moves_since` filters it out. It IS the ledger beginning to observe the arm, and from that moment silence genuinely means the floor held — filtering it out here would suspend the band check forever on every install that has never tuned. `floor_history_unknown` sits between the known move and the band, so an arm with a date to give gives it. Two sentences, because the remedies differ: a date says ask again with a smaller `days`; no history at all says there is nothing to wait for, and names what starts the record. Why now: #4261 measures the work-log change by reading these warnings, and every window for the next month opens before this ledger's first row. Measuring against an instrument that prints a number it cannot support is the #4225 trap one level up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -43,7 +43,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy import func, or_, select
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.retrieval_tuning import RetrievalTuningEvent
|
||||
@@ -420,6 +420,56 @@ async def floor_moves_since(since) -> dict[str, str]:
|
||||
return out
|
||||
|
||||
|
||||
async def floor_history_gaps(since) -> dict[str, str | None]:
|
||||
"""Surfaces whose floor history does NOT cover a window starting at `since`.
|
||||
|
||||
The companion `floor_moves_since` needs, and the hole #4225's fix left.
|
||||
That function answers "did this arm's floor move inside the window", and an
|
||||
empty answer was being read as "no, it held steady". Those are the same
|
||||
answer only while the LEDGER covers the window. Before the ledger existed
|
||||
there are no rows for any arm, so every arm reported "no move" for a period
|
||||
nobody had recorded — absence of evidence arriving as evidence of absence.
|
||||
|
||||
It is not hypothetical, and it is not a one-off: every install passes
|
||||
through it. The ledger's first row for an arm is written when that install
|
||||
first boots the release that records baselines, so for a window longer
|
||||
than the install is old — or than the arm is old, for an arm added
|
||||
later — there is no history to consult, and the band check was reading
|
||||
that silence as a clean bill of health.
|
||||
|
||||
Returns ONLY the arms with a gap, mapped to the ISO timestamp from which
|
||||
their floor IS knowable — or None when the arm has no floor history at all.
|
||||
An arm absent from the result is fully covered and can be judged. Shaped
|
||||
that way so a caller loops over what it is already judging and asks one
|
||||
question per arm, rather than reasoning about the ledger itself.
|
||||
|
||||
Every row counts here, baselines included, which is the one place
|
||||
`floor_moves_since` and this deliberately disagree. A baseline is not a
|
||||
change — so it is not a move — but it IS the ledger observing the arm, and
|
||||
from that moment on silence genuinely means the floor held.
|
||||
"""
|
||||
out: dict[str, str | None] = {}
|
||||
async with async_session() as session:
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(
|
||||
RetrievalTuningEvent.surface,
|
||||
func.min(RetrievalTuningEvent.created_at),
|
||||
)
|
||||
.where(RetrievalTuningEvent.dial == "floor")
|
||||
.group_by(RetrievalTuningEvent.surface)
|
||||
)
|
||||
).all()
|
||||
earliest = {surface: first for surface, first in rows}
|
||||
for name in surface_names():
|
||||
first = earliest.get(name)
|
||||
if first is None:
|
||||
out[name] = None
|
||||
elif first > since:
|
||||
out[name] = first.isoformat()
|
||||
return out
|
||||
|
||||
|
||||
async def tuning_history(
|
||||
user_id: int, *, surface: str | None = None, limit: int = 20
|
||||
) -> list[dict]:
|
||||
|
||||
Reference in New Issue
Block a user