fix(write-path): give the semantic arm its own threshold + a payload floor
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 6s
CI & Build / integration (push) Successful in 20s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / Python tests (push) Successful in 41s
CI & Build / Build & push image (push) Successful in 49s

Closes #2223. The write-path prior-art trigger's semantic arm shared
auto-inject's 0.55 threshold, which was tuned on prose. Code embeddings
sit on a much higher similarity floor — any two Python-shaped payloads
share keywords, indentation and structure — so 0.55 landed INSIDE the
noise band. Measured against the live instance:

  near-duplicate of a recorded helper   0.73-0.74   true positive
  unrelated colour math / Vue SFC / CSS 0.55-0.63   false positive
  `x = 1`                               0.58        false positive

6 of 8 probe payloads produced a nudge; 4 were noise. The margin gate
couldn't help — _AUTOINJECT_BAND is relative to the top hit, so with a
single hit it never engages.

Two gates are now the write-path arm's own:

- kb_writepath_threshold, default 0.68 — above every measured false
  positive, still 0.05 below both true positives. Auto-inject keeps
  0.55; it was tuned on prose and is not implicated. The comment this
  replaces explicitly reserved the split for when telemetry showed the
  surfaces wanted different values, so this is the change it described,
  not a reversal of it.
- WRITEPATH_MIN_CODE_CHARS = 48 non-whitespace chars, below which the
  semantic arm doesn't run at all. Whitespace is excluded so a deeply
  indented one-liner can't pass on padding. 48 sits under the smallest
  plausible reusable helper (~60) and well over a degenerate edit, so it
  errs toward keeping recall — precision is the threshold's job. This is
  the cheap half of the operator's #89 idea; the full length<->threshold
  curve stays open there, since they asked to brainstorm it rather than
  have a scale invented for them.

top_k stays shared — "how many titles at once" means the same thing on
both surfaces.

The existing tests were passing `code="x"` / `code="def f(): ..."` into
the semantic arm, i.e. exactly the payloads the floor now drops, so the
gate tests were never exercising a realistic payload. They now use a
REAL_CODE fixture, plus new coverage for the floor (trivial payload,
padding, place-arm unaffected, real helper passes) and a guard on the
constant itself.

Settings UI carries the new knob with the reasoning in its hint, and the
write-path checkbox no longer claims it shares the threshold.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
This commit is contained in:
2026-07-29 23:26:59 -04:00
co-authored by Claude Opus 5
parent 05b64ccacb
commit 1fb883b72f
3 changed files with 253 additions and 35 deletions
+70 -13
View File
@@ -56,16 +56,47 @@ AUTOINJECT_DEFAULT_ENABLED = True
AUTOINJECT_DEFAULT_THRESHOLD = 0.55
AUTOINJECT_DEFAULT_TOP_K = 3
# The write-path trigger (#2082) gets its own on/off switch but SHARES the
# threshold and top-k above. One knob for "how loud may Scribe be" is easier to
# reason about than two that drift; the switch is separate because wanting prior
# art on writes while keeping prompts quiet (or the reverse) is a real
# preference, and it's the only part of the gate an operator can judge without
# data. The two surfaces log under different `source` values, so if the
# telemetry ever shows they want different thresholds, splitting them is a
# data-backed change rather than a guess made up front.
# The write-path trigger (#2082) gets its own on/off switch, its own threshold,
# and shares only top-k. It originally shared the threshold too, on the argument
# that one "how loud may Scribe be" knob beats two that drift — and reserved the
# split for when telemetry showed the two surfaces wanted different values.
#
# #2223 is that evidence. Measured against the live instance, the semantic arm's
# scores for CODE sit far above what the same threshold means for PROSE:
# near-duplicate of a recorded helper 0.73-0.74 (true positive)
# unrelated colour math / Vue SFC / CSS 0.55-0.63 (false positive)
# `x = 1` 0.58 (false positive)
# Any two Python-shaped payloads share keywords, indentation and structure, so
# the floor for "some code" is ~0.55-0.63 — auto-inject's 0.55 lands INSIDE that
# noise band, and 6 of 8 probe payloads produced a nudge (4 of them noise). The
# margin gate can't rescue it either: _AUTOINJECT_BAND is relative to the top
# hit, so with a single hit it never engages.
#
# 0.68 clears every measured false positive with margin and still sits 0.05
# below both true positives. Auto-inject keeps 0.55 — it was tuned on prose and
# is not implicated. Tune from retrieval_logs (source='write_path') + note_usage
# pull-through (#2085) once a real corpus accrues; a cross-encoder rerank
# (#1038) would subsume this bump.
WRITEPATH_ENABLED_KEY = "kb_writepath_enabled"
WRITEPATH_THRESHOLD_KEY = "kb_writepath_threshold"
WRITEPATH_DEFAULT_ENABLED = True
WRITEPATH_DEFAULT_THRESHOLD = 0.68
# Minimum SUBSTANCE (non-whitespace chars) a payload must carry before the
# semantic arm will run at all — the cheap half of the operator's #89 idea
# ("a sliding scale between number of characters and semantic threshold").
#
# Deliberately NOT a settings knob and deliberately conservative. Its job is
# only to drop payloads too small to carry meaning, where an embedding is noise
# rather than signal: `x = 1`, a renamed variable, a changed string literal —
# which is what most single-line Edits look like, and the majority of Edits are
# single-line. 48 sits below the smallest plausible reusable helper (a one-line
# `def` with a body runs ~60), so it errs toward keeping recall and leaves
# precision to the threshold above, which is where the measured separation is.
# The full length↔threshold CURVE is still open in #89 — the operator flagged it
# as wanting a brainstorm, so this stays a flat floor rather than an invented
# scale. It also saves a pointless embedding round-trip on trivial edits.
WRITEPATH_MIN_CODE_CHARS = 48
# Margin gate: drop any hit more than this far below the top hit's score, so a
# single strong match doesn't drag in a wall of barely-passing neighbours.
@@ -307,15 +338,31 @@ def _prior_art_line(item: dict, marker: str, owner: str | None) -> str:
async def get_writepath_config(user_id: int) -> dict:
"""Write-path trigger settings: its own `enabled`, auto-inject's gates."""
"""Write-path trigger settings: its own `enabled` and `threshold`, auto-inject's top_k.
The threshold OVERRIDES the inherited auto-inject value — code embeddings
have a much higher similarity floor than prose, so the two surfaces need
different bars. See WRITEPATH_DEFAULT_THRESHOLD for the measurements (#2223).
top_k is still shared: "how many titles at once" means the same thing on
both surfaces, and nothing suggests they want different ceilings.
"""
cfg = await get_autoinject_config(user_id)
enabled_raw = await get_setting(
user_id, WRITEPATH_ENABLED_KEY,
"true" if WRITEPATH_DEFAULT_ENABLED else "false",
)
try:
threshold = float(await get_setting(
user_id, WRITEPATH_THRESHOLD_KEY, str(WRITEPATH_DEFAULT_THRESHOLD)))
except (TypeError, ValueError):
threshold = WRITEPATH_DEFAULT_THRESHOLD
threshold = min(1.0, max(0.0, threshold))
return {
**cfg,
"enabled": enabled_raw.strip().lower() in ("true", "1", "yes", "on"),
"threshold": threshold,
}
@@ -332,11 +379,14 @@ async def build_write_path_hint(
convention snippet locations are recorded in. `code` is what's about to be
written, used only as the semantic query.
Same four anti-bloat gates as auto-inject (threshold, margin, session dedup
via `exclude_ids`, titles-never-bodies) plus the shared top-k cap across BOTH
Carries auto-inject's anti-bloat gates (margin, session dedup via
`exclude_ids`, titles-never-bodies) plus the shared top-k cap across BOTH
arms — so a file with a lot of recorded history can't turn one edit into a
wall of text. Returns empty context when disabled, when there's no path, or
when nothing is recorded — which is the common case, and the point.
wall of text. Two gates are its OWN, because code is not prose: a stricter
similarity threshold, and a minimum-substance floor on `code` below which the
semantic arm doesn't run at all (#2223 — see WRITEPATH_DEFAULT_THRESHOLD and
WRITEPATH_MIN_CODE_CHARS). Returns empty context when disabled, when there's
no path, or when nothing is recorded — which is the common case, and the point.
Note the repo↔project mapping is deliberately one-way: the hook sends a git
remote, which the ROUTE resolves to `project_id` through the repo bindings.
@@ -394,6 +444,13 @@ async def build_write_path_hint(
scored: list[tuple[str, dict]] = []
remaining = top_k - len(placed)
query = (code or "").strip()
# Drop payloads too small to carry meaning before spending an embedding on
# them — a one-line Edit is not a helper being rewritten, and its embedding
# scores off the corpus floor rather than off any real resemblance (#2223).
# Whitespace doesn't count: code is indentation-heavy, so raw length would
# let a deeply-nested one-liner through on padding alone.
if len("".join(query.split())) < WRITEPATH_MIN_CODE_CHARS:
query = ""
if remaining > 0 and query:
t0 = time.perf_counter()
hits = await semantic_search_notes(