feat(telemetry): record WHAT the bar turned away, not only how close it came (#3807)
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / integration (push) Successful in 40s
CI & Build / TypeScript typecheck (push) Successful in 43s
CI & Build / Python tests (push) Successful in 1m14s
CI & Build / Build & push image (push) Successful in 2m59s
CI & Build / Python lint (push) Successful in 8s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / integration (push) Successful in 40s
CI & Build / TypeScript typecheck (push) Successful in 43s
CI & Build / Python tests (push) Successful in 1m14s
CI & Build / Build & push image (push) Successful in 2m59s
#3670 added `best_available_score` so a threshold could be judged from its rejections. It records how CLOSE the bar came to firing and not WHAT it refused, and that is the half a decision actually needs. Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 — about 117 declines a day within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to roughly 139: six-fold, on a surface that runs before every Bash call. The percentile says the mass is there. Nothing said whether it was worth showing. NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected rule line already carries title and trigger, so a session can comply without ever calling get_rule, and rule pull-through understates usefulness by construction. Reading the rejected records can — and `result_ids` holds only what was RETURNED, so on a zero-result call the near-missed record had no name at all. So the id, from the SAME ranked candidate as the score. Both searches unpack `best` once and read both fields off it, because splitting that into two expressions is exactly how a later edit pairs a score with its neighbour's id — and a score attached to the wrong record is worse than no id, since it invites judging the wrong one and concluding the bar is fine. write_path withholds the id on the same condition it withholds the score (#3739): a surviving id beside a null score names a record without saying what it scored, the pair disagreeing in the other direction. THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled, and a reader tuning a bar needs to go and read the records. Opt-in via `near_miss_samples` (0-20, default 0) so the ordinary readout keeps its size, and deliberately NOT a window function: this module's one production outage was a grouped query Postgres rejected, swallowed by the broad except, every counter reading zero while the mocked tests passed (#2663). One flat ordered query, overfetched, bucketed in Python — the shape that lesson prescribes. Migration 0097, nullable and unbackfilled. Not a foreign key: the table spans record types and `source` says which, exactly as result_ids works. The integration guard pins the listing as PER SOURCE. A global LIMIT would let a noisy source eat the whole quota and leave the surface being tuned showing nothing — which reads as "nothing was close", the misreading this milestone has spent itself correcting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -484,6 +484,7 @@ async def _reserve_slot_for_reuse(
|
||||
threshold=cfg["threshold"], limit=1, project_id=project_id,
|
||||
is_task=None, results=reuse,
|
||||
best_available=_rep.get("best_available_score"),
|
||||
best_available_id=_rep.get("best_available_id"),
|
||||
searched=bool(_rep.get("searched", True)),
|
||||
duration_ms=(time.perf_counter() - _t0) * 1000.0,
|
||||
)
|
||||
@@ -553,6 +554,7 @@ async def build_autoinject_hint(
|
||||
threshold=cfg["threshold"], limit=cfg["top_k"],
|
||||
project_id=(project_id or None), is_task=None, results=hits,
|
||||
best_available=_rep_ai.get("best_available_score"),
|
||||
best_available_id=_rep_ai.get("best_available_id"),
|
||||
searched=bool(_rep_ai.get("searched", True)),
|
||||
duration_ms=(time.perf_counter() - t0) * 1000.0,
|
||||
)
|
||||
@@ -1048,6 +1050,12 @@ async def build_write_path_hint(
|
||||
best_available=(
|
||||
None if withheld_here else _rep_wp.get("best_available_score")
|
||||
),
|
||||
# Withheld on the SAME condition as the score. A surviving id
|
||||
# beside a null score would name a record without saying what it
|
||||
# scored, which is the pair disagreeing in the other direction.
|
||||
best_available_id=(
|
||||
None if withheld_here else _rep_wp.get("best_available_id")
|
||||
),
|
||||
searched=bool(_rep_wp.get("searched", True)),
|
||||
duration_ms=(time.perf_counter() - t0) * 1000.0,
|
||||
)
|
||||
@@ -1345,6 +1353,7 @@ async def build_write_path_hint(
|
||||
project_id=project_id,
|
||||
is_task=None, results=fresh, duration_ms=rule_ms,
|
||||
best_available=_rep_wpr.get("best_available_score"),
|
||||
best_available_id=_rep_wpr.get("best_available_id"),
|
||||
searched=bool(_rep_wpr.get("searched", True)),
|
||||
# What the ranker found and this session had already been told.
|
||||
# Without it a zero row cannot say whether the bar was too high or
|
||||
@@ -1453,6 +1462,7 @@ async def build_tool_rule_hint(
|
||||
project_id=project_id,
|
||||
is_task=None, results=fresh, duration_ms=duration_ms,
|
||||
best_available=_rep_ptr.get("best_available_score"),
|
||||
best_available_id=_rep_ptr.get("best_available_id"),
|
||||
searched=bool(_rep_ptr.get("searched", True)),
|
||||
# See the sibling arm. It matters more here: this arm fires on every
|
||||
# Bash call, so a long session excludes its way to an all-zero row
|
||||
|
||||
Reference in New Issue
Block a user