docs(retrieval): stop asking the operator to diagnose retrieval (#4102)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 33s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 33s
The last item in this step's done-when: no wording anywhere asks an operator to tune for correctness. Four Settings hints told them to do exactly that — "raise it if rules keep arriving unread", "lower it if a git push arrives with nothing", "lower this if genuine duplicates go unnoticed". Every one of those asks the operator to diagnose a ranker from symptoms, which is the job the model now does from the records: the telemetry says what each bar refused, and reading those records is what separates a real miss from a bar doing its job. The hints keep the explanation of WHAT each number is — that is worth reading — and drop the homework. `plugin_context.py` said the defaults "are meant to be tuned from retrieval_logs once data accrues", which was true and had no owner. It now names who does it and with what. `retrieval_telemetry`'s docstring gained the warning that belongs beside it: this readout has been measured pointing the wrong way, so the ids it returns are the point, not its percentiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -1605,9 +1605,9 @@ async function deleteUser(userId: number) {
|
|||||||
Stricter than the prompt threshold above on purpose. Any two pieces of
|
Stricter than the prompt threshold above on purpose. Any two pieces of
|
||||||
code look somewhat alike — shared keywords, indentation, structure — so
|
code look somewhat alike — shared keywords, indentation, structure — so
|
||||||
resemblance scores start higher for code than for prose, and a bar tuned
|
resemblance scores start higher for code than for prose, and a bar tuned
|
||||||
for prompts flags unrelated code as prior art. Lower this if genuine
|
for prompts flags unrelated code as prior art. Claude keeps this
|
||||||
duplicates go unnoticed; raise it if you're being offered snippets that
|
one current from what the arm actually surfaced and refused; set it
|
||||||
have nothing to do with what's being written. Snippets recorded at the
|
yourself if you disagree with where it has landed. Snippets recorded at the
|
||||||
exact file are always shown regardless — those are prior art by
|
exact file are always shown regardless — those are prior art by
|
||||||
location, not by resemblance.
|
location, not by resemblance.
|
||||||
</p>
|
</p>
|
||||||
@@ -1644,8 +1644,9 @@ async function deleteUser(userId: number) {
|
|||||||
preloaded any more, so this is the only way a rule reaches a write.
|
preloaded any more, so this is the only way a rule reaches a write.
|
||||||
Stricter than the threshold above, because there are far fewer rules
|
Stricter than the threshold above, because there are far fewer rules
|
||||||
than snippets: with a small set something always ranks first, so the
|
than snippets: with a small set something always ranks first, so the
|
||||||
bar has to carry more of the judgement. Raise it if rules keep
|
bar has to carry more of the judgement. If rules arrive unread, or one
|
||||||
arriving unread; lower it if a rule you needed never showed up.
|
you needed never showed up, that is Claude's to notice and correct
|
||||||
|
from the telemetry — and the budget below is usually the better lever.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="field">
|
<div class="field">
|
||||||
@@ -1679,8 +1680,7 @@ async function deleteUser(userId: number) {
|
|||||||
query is the command text rather than code. Lower than the one above
|
query is the command text rather than code. Lower than the one above
|
||||||
on purpose: a shell command is short, so it scores lower for the same
|
on purpose: a shell command is short, so it scores lower for the same
|
||||||
relevance — at a shared bar this arm spoke on 2% of calls against the
|
relevance — at a shared bar this arm spoke on 2% of calls against the
|
||||||
write path's 37%. Raise it if commands attract rules that do not
|
write path's 37%.
|
||||||
apply; lower it if a <code>git push</code> arrives with nothing.
|
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="field">
|
<div class="field">
|
||||||
@@ -1748,9 +1748,9 @@ async function deleteUser(userId: number) {
|
|||||||
written</em>, looked up when a task closes. Unlike every other bar
|
written</em>, looked up when a task closes. Unlike every other bar
|
||||||
here, the question this arm asks never changes — so its score is
|
here, the question this arm asks never changes — so its score is
|
||||||
fixed by your preferences alone, and it will either always find one
|
fixed by your preferences alone, and it will either always find one
|
||||||
or never find one. If you have written a preference for report shape
|
or never find one, and no run of calls will reveal a dead one on its
|
||||||
and it is not arriving, lower this; there is no run of calls that
|
own. That is why this arm is worth looking up in the panel below when
|
||||||
will reveal the problem on its own.
|
a report preference never seems to arrive.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="field">
|
<div class="field">
|
||||||
|
|||||||
@@ -209,11 +209,13 @@ async def retrieval_telemetry(
|
|||||||
) -> dict:
|
) -> dict:
|
||||||
"""What the retrieval telemetry says about YOUR surfaces, over a window.
|
"""What the retrieval telemetry says about YOUR surfaces, over a window.
|
||||||
|
|
||||||
The read half of the loop the ranker's thresholds are meant to be tuned
|
The read half of the tuning loop, whose write half is `tune_retrieval`
|
||||||
from (#2975). Reach for it before changing a similarity threshold, a top-k,
|
(#2975, #4102). Reach for it before moving any floor or budget, and read
|
||||||
or deciding whether a reranker is worth building — the alternative is
|
the records it names rather than its percentiles alone: this readout has
|
||||||
hand-probing the live instance, which is how the last such decision had to
|
been measured pointing the WRONG WAY — 69 consecutive declines where every
|
||||||
be made.
|
percentile said "lower the bar" and the refused record was a false positive
|
||||||
|
— so `near_miss_samples=5` and opening the ids it returns is the step that
|
||||||
|
separates a real miss from a bar doing its job.
|
||||||
|
|
||||||
Three readouts, from the three tables built for them:
|
Three readouts, from the three tables built for them:
|
||||||
|
|
||||||
|
|||||||
@@ -56,9 +56,14 @@ _GOAL_CHARS = 200
|
|||||||
# Per-user settings (keys live in the generic settings table). The threshold is
|
# Per-user settings (keys live in the generic settings table). The threshold is
|
||||||
# deliberately STRICTER than the pull-search default (embeddings
|
# deliberately STRICTER than the pull-search default (embeddings
|
||||||
# DEFAULT_SIMILARITY_THRESHOLD = 0.45): an unsolicited per-turn inject must clear
|
# DEFAULT_SIMILARITY_THRESHOLD = 0.45): an unsolicited per-turn inject must clear
|
||||||
# a higher bar than a search the agent chose to run. Defaults start conservative
|
# a higher bar than a search the agent chose to run.
|
||||||
# and are meant to be tuned from retrieval_logs (source='auto_inject') once data
|
#
|
||||||
# accrues — they're exposed in the Settings UI, no restart needed.
|
# The defaults below are STARTING POINTS, and correcting them is the model's
|
||||||
|
# job, not the operator's (#4102): `retrieval_surfaces` says what is in force,
|
||||||
|
# `retrieval_telemetry(near_miss_samples=N)` says what it refused, and
|
||||||
|
# `tune_retrieval` moves it with the reason attached. The operator can set any
|
||||||
|
# of them in Settings and their change is recorded the same way — but nobody
|
||||||
|
# has to read a log to get correct behaviour out of this.
|
||||||
AUTOINJECT_ENABLED_KEY = "kb_autoinject_enabled"
|
AUTOINJECT_ENABLED_KEY = "kb_autoinject_enabled"
|
||||||
# The key and the value both live in the registry now (#4102); these names
|
# The key and the value both live in the registry now (#4102); these names
|
||||||
# survive because the comments above them are where each number's measurement
|
# survive because the comments above them are where each number's measurement
|
||||||
|
|||||||
Reference in New Issue
Block a user