fix(write-path): give the semantic arm its own threshold + a payload floor
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 6s
CI & Build / integration (push) Successful in 20s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / Python tests (push) Successful in 41s
CI & Build / Build & push image (push) Successful in 49s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 6s
CI & Build / integration (push) Successful in 20s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / Python tests (push) Successful in 41s
CI & Build / Build & push image (push) Successful in 49s
Closes #2223. The write-path prior-art trigger's semantic arm shared auto-inject's 0.55 threshold, which was tuned on prose. Code embeddings sit on a much higher similarity floor — any two Python-shaped payloads share keywords, indentation and structure — so 0.55 landed INSIDE the noise band. Measured against the live instance: near-duplicate of a recorded helper 0.73-0.74 true positive unrelated colour math / Vue SFC / CSS 0.55-0.63 false positive `x = 1` 0.58 false positive 6 of 8 probe payloads produced a nudge; 4 were noise. The margin gate couldn't help — _AUTOINJECT_BAND is relative to the top hit, so with a single hit it never engages. Two gates are now the write-path arm's own: - kb_writepath_threshold, default 0.68 — above every measured false positive, still 0.05 below both true positives. Auto-inject keeps 0.55; it was tuned on prose and is not implicated. The comment this replaces explicitly reserved the split for when telemetry showed the surfaces wanted different values, so this is the change it described, not a reversal of it. - WRITEPATH_MIN_CODE_CHARS = 48 non-whitespace chars, below which the semantic arm doesn't run at all. Whitespace is excluded so a deeply indented one-liner can't pass on padding. 48 sits under the smallest plausible reusable helper (~60) and well over a degenerate edit, so it errs toward keeping recall — precision is the threshold's job. This is the cheap half of the operator's #89 idea; the full length<->threshold curve stays open there, since they asked to brainstorm it rather than have a scale invented for them. top_k stays shared — "how many titles at once" means the same thing on both surfaces. The existing tests were passing `code="x"` / `code="def f(): ..."` into the semantic arm, i.e. exactly the payloads the floor now drops, so the gate tests were never exercising a realistic payload. They now use a REAL_CODE fixture, plus new coverage for the floor (trivial payload, padding, place-arm unaffected, real helper passes) and a guard on the constant itself. Settings UI carries the new knob with the reasoning in its hint, and the write-path checkbox no longer claims it shares the threshold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
@@ -56,16 +56,47 @@ AUTOINJECT_DEFAULT_ENABLED = True
|
||||
AUTOINJECT_DEFAULT_THRESHOLD = 0.55
|
||||
AUTOINJECT_DEFAULT_TOP_K = 3
|
||||
|
||||
# The write-path trigger (#2082) gets its own on/off switch but SHARES the
|
||||
# threshold and top-k above. One knob for "how loud may Scribe be" is easier to
|
||||
# reason about than two that drift; the switch is separate because wanting prior
|
||||
# art on writes while keeping prompts quiet (or the reverse) is a real
|
||||
# preference, and it's the only part of the gate an operator can judge without
|
||||
# data. The two surfaces log under different `source` values, so if the
|
||||
# telemetry ever shows they want different thresholds, splitting them is a
|
||||
# data-backed change rather than a guess made up front.
|
||||
# The write-path trigger (#2082) gets its own on/off switch, its own threshold,
|
||||
# and shares only top-k. It originally shared the threshold too, on the argument
|
||||
# that one "how loud may Scribe be" knob beats two that drift — and reserved the
|
||||
# split for when telemetry showed the two surfaces wanted different values.
|
||||
#
|
||||
# #2223 is that evidence. Measured against the live instance, the semantic arm's
|
||||
# scores for CODE sit far above what the same threshold means for PROSE:
|
||||
# near-duplicate of a recorded helper 0.73-0.74 (true positive)
|
||||
# unrelated colour math / Vue SFC / CSS 0.55-0.63 (false positive)
|
||||
# `x = 1` 0.58 (false positive)
|
||||
# Any two Python-shaped payloads share keywords, indentation and structure, so
|
||||
# the floor for "some code" is ~0.55-0.63 — auto-inject's 0.55 lands INSIDE that
|
||||
# noise band, and 6 of 8 probe payloads produced a nudge (4 of them noise). The
|
||||
# margin gate can't rescue it either: _AUTOINJECT_BAND is relative to the top
|
||||
# hit, so with a single hit it never engages.
|
||||
#
|
||||
# 0.68 clears every measured false positive with margin and still sits 0.05
|
||||
# below both true positives. Auto-inject keeps 0.55 — it was tuned on prose and
|
||||
# is not implicated. Tune from retrieval_logs (source='write_path') + note_usage
|
||||
# pull-through (#2085) once a real corpus accrues; a cross-encoder rerank
|
||||
# (#1038) would subsume this bump.
|
||||
WRITEPATH_ENABLED_KEY = "kb_writepath_enabled"
|
||||
WRITEPATH_THRESHOLD_KEY = "kb_writepath_threshold"
|
||||
WRITEPATH_DEFAULT_ENABLED = True
|
||||
WRITEPATH_DEFAULT_THRESHOLD = 0.68
|
||||
|
||||
# Minimum SUBSTANCE (non-whitespace chars) a payload must carry before the
|
||||
# semantic arm will run at all — the cheap half of the operator's #89 idea
|
||||
# ("a sliding scale between number of characters and semantic threshold").
|
||||
#
|
||||
# Deliberately NOT a settings knob and deliberately conservative. Its job is
|
||||
# only to drop payloads too small to carry meaning, where an embedding is noise
|
||||
# rather than signal: `x = 1`, a renamed variable, a changed string literal —
|
||||
# which is what most single-line Edits look like, and the majority of Edits are
|
||||
# single-line. 48 sits below the smallest plausible reusable helper (a one-line
|
||||
# `def` with a body runs ~60), so it errs toward keeping recall and leaves
|
||||
# precision to the threshold above, which is where the measured separation is.
|
||||
# The full length↔threshold CURVE is still open in #89 — the operator flagged it
|
||||
# as wanting a brainstorm, so this stays a flat floor rather than an invented
|
||||
# scale. It also saves a pointless embedding round-trip on trivial edits.
|
||||
WRITEPATH_MIN_CODE_CHARS = 48
|
||||
|
||||
# Margin gate: drop any hit more than this far below the top hit's score, so a
|
||||
# single strong match doesn't drag in a wall of barely-passing neighbours.
|
||||
@@ -307,15 +338,31 @@ def _prior_art_line(item: dict, marker: str, owner: str | None) -> str:
|
||||
|
||||
|
||||
async def get_writepath_config(user_id: int) -> dict:
|
||||
"""Write-path trigger settings: its own `enabled`, auto-inject's gates."""
|
||||
"""Write-path trigger settings: its own `enabled` and `threshold`, auto-inject's top_k.
|
||||
|
||||
The threshold OVERRIDES the inherited auto-inject value — code embeddings
|
||||
have a much higher similarity floor than prose, so the two surfaces need
|
||||
different bars. See WRITEPATH_DEFAULT_THRESHOLD for the measurements (#2223).
|
||||
top_k is still shared: "how many titles at once" means the same thing on
|
||||
both surfaces, and nothing suggests they want different ceilings.
|
||||
"""
|
||||
cfg = await get_autoinject_config(user_id)
|
||||
enabled_raw = await get_setting(
|
||||
user_id, WRITEPATH_ENABLED_KEY,
|
||||
"true" if WRITEPATH_DEFAULT_ENABLED else "false",
|
||||
)
|
||||
|
||||
try:
|
||||
threshold = float(await get_setting(
|
||||
user_id, WRITEPATH_THRESHOLD_KEY, str(WRITEPATH_DEFAULT_THRESHOLD)))
|
||||
except (TypeError, ValueError):
|
||||
threshold = WRITEPATH_DEFAULT_THRESHOLD
|
||||
threshold = min(1.0, max(0.0, threshold))
|
||||
|
||||
return {
|
||||
**cfg,
|
||||
"enabled": enabled_raw.strip().lower() in ("true", "1", "yes", "on"),
|
||||
"threshold": threshold,
|
||||
}
|
||||
|
||||
|
||||
@@ -332,11 +379,14 @@ async def build_write_path_hint(
|
||||
convention snippet locations are recorded in. `code` is what's about to be
|
||||
written, used only as the semantic query.
|
||||
|
||||
Same four anti-bloat gates as auto-inject (threshold, margin, session dedup
|
||||
via `exclude_ids`, titles-never-bodies) plus the shared top-k cap across BOTH
|
||||
Carries auto-inject's anti-bloat gates (margin, session dedup via
|
||||
`exclude_ids`, titles-never-bodies) plus the shared top-k cap across BOTH
|
||||
arms — so a file with a lot of recorded history can't turn one edit into a
|
||||
wall of text. Returns empty context when disabled, when there's no path, or
|
||||
when nothing is recorded — which is the common case, and the point.
|
||||
wall of text. Two gates are its OWN, because code is not prose: a stricter
|
||||
similarity threshold, and a minimum-substance floor on `code` below which the
|
||||
semantic arm doesn't run at all (#2223 — see WRITEPATH_DEFAULT_THRESHOLD and
|
||||
WRITEPATH_MIN_CODE_CHARS). Returns empty context when disabled, when there's
|
||||
no path, or when nothing is recorded — which is the common case, and the point.
|
||||
|
||||
Note the repo↔project mapping is deliberately one-way: the hook sends a git
|
||||
remote, which the ROUTE resolves to `project_id` through the repo bindings.
|
||||
@@ -394,6 +444,13 @@ async def build_write_path_hint(
|
||||
scored: list[tuple[str, dict]] = []
|
||||
remaining = top_k - len(placed)
|
||||
query = (code or "").strip()
|
||||
# Drop payloads too small to carry meaning before spending an embedding on
|
||||
# them — a one-line Edit is not a helper being rewritten, and its embedding
|
||||
# scores off the corpus floor rather than off any real resemblance (#2223).
|
||||
# Whitespace doesn't count: code is indentation-heavy, so raw length would
|
||||
# let a deeply-nested one-liner through on padding alone.
|
||||
if len("".join(query.split())) < WRITEPATH_MIN_CODE_CHARS:
|
||||
query = ""
|
||||
if remaining > 0 and query:
|
||||
t0 = time.perf_counter()
|
||||
hits = await semantic_search_notes(
|
||||
|
||||
Reference in New Issue
Block a user