Write-path prior-art precision: own threshold + payload floor #85

Merged

bvandeusen merged 2 commits from dev into main 2026-07-30 08:07:48 -04:00

Conversation 0 Commits 2 Files Changed 4

+259 -35

bvandeusen commented 2026-07-30 08:07:38 -04:00

Owner

Copy Link

Copy Source

Two commits. CI green on dev head 1fb883b (run 3051, all six jobs).

1fb883b — fix(write-path): own threshold + payload floor

Closes #2223. The write-path prior-art trigger's semantic arm shared
auto-inject's 0.55 threshold, which was calibrated on prose. Code embeddings
sit on a much higher similarity floor, so 0.55 landed inside the noise band.
Measured against the live instance:

payload	score	verdict
near-duplicate of a recorded helper	0.73–0.74	true positive
unrelated colour math / Vue SFC / CSS	0.55–0.63	false positive
x = 1	0.58	false positive

6 of 8 probe payloads produced a nudge; 4 were noise. The margin gate couldn't
help — _AUTOINJECT_BAND is relative to the top hit, so with a single hit it
never engages.

Two gates are now the write-path arm's own:

• kb_writepath_threshold, default 0.68 — above every measured false
  positive, still 0.05 below both true positives. Auto-inject keeps 0.55. The
  comment this replaces explicitly reserved the split for when telemetry showed
  the surfaces wanted different values, so this is the change it anticipated.
• WRITEPATH_MIN_CODE_CHARS = 48 non-whitespace chars, below which the
  semantic arm doesn't run at all — no embedding call, no retrieval_logs row.
  Whitespace is excluded so a deeply indented one-liner can't pass on padding.
  48 sits under the smallest plausible reusable helper (63 measured) and well
  over a degenerate edit, so it errs toward keeping recall.

top_k stays shared. Only the flat floor from #89 is taken here; the full
length↔threshold curve stays open there by request.

Tests: the suite had been feeding the semantic arm 1- and 12-character payloads
— exactly what the floor now drops — so nothing exercised a realistic payload.
Now uses a REAL_CODE fixture, plus new coverage for the floor (trivial
payload, padding, place-arm unaffected, real helper passes), threshold clamping,
and a guard keeping the constant below the smallest real helper.

Settings UI carries the new knob; the write-path checkbox no longer claims a
shared threshold.

05b64cc — docs(knowledge)

Names tests/test_snippet_drift_check.py as the guard for the verification
filter's two dialects. Comment only.

Deploy notes

No migration (settings are generic key/value, no CHECK enum). No plugin change,
so no manifest bump — the threshold is entirely server-side and existing 0.1.20
installs pick it up with no re-install.

CI green proves the gates behave as specified; it can't prove the false
positives are gone, since the probe measures the deployed server. Re-running the
calibration after deploy is the actual confirmation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

Two commits. CI green on `dev` head `1fb883b` (run 3051, all six jobs). ## `1fb883b` — fix(write-path): own threshold + payload floor Closes #2223. The write-path prior-art trigger's semantic arm shared auto-inject's 0.55 threshold, which was calibrated on prose. Code embeddings sit on a much higher similarity floor, so 0.55 landed *inside* the noise band. Measured against the live instance: | payload | score | verdict | |---|---|---| | near-duplicate of a recorded helper | 0.73–0.74 | true positive | | unrelated colour math / Vue SFC / CSS | 0.55–0.63 | false positive | | `x = 1` | 0.58 | false positive | 6 of 8 probe payloads produced a nudge; 4 were noise. The margin gate couldn't help — `_AUTOINJECT_BAND` is relative to the top hit, so with a single hit it never engages. Two gates are now the write-path arm's own: - **`kb_writepath_threshold`, default 0.68** — above every measured false positive, still 0.05 below both true positives. Auto-inject keeps 0.55. The comment this replaces explicitly reserved the split for when telemetry showed the surfaces wanted different values, so this is the change it anticipated. - **`WRITEPATH_MIN_CODE_CHARS = 48`** non-whitespace chars, below which the semantic arm doesn't run at all — no embedding call, no `retrieval_logs` row. Whitespace is excluded so a deeply indented one-liner can't pass on padding. 48 sits under the smallest plausible reusable helper (63 measured) and well over a degenerate edit, so it errs toward keeping recall. `top_k` stays shared. Only the flat floor from #89 is taken here; the full length↔threshold curve stays open there by request. Tests: the suite had been feeding the semantic arm 1- and 12-character payloads — exactly what the floor now drops — so nothing exercised a realistic payload. Now uses a `REAL_CODE` fixture, plus new coverage for the floor (trivial payload, padding, place-arm unaffected, real helper passes), threshold clamping, and a guard keeping the constant below the smallest real helper. Settings UI carries the new knob; the write-path checkbox no longer claims a shared threshold. ## `05b64cc` — docs(knowledge) Names `tests/test_snippet_drift_check.py` as the guard for the verification filter's two dialects. Comment only. ## Deploy notes No migration (settings are generic key/value, no CHECK enum). No plugin change, so no manifest bump — the threshold is entirely server-side and existing 0.1.20 installs pick it up with no re-install. CI green proves the gates behave as specified; it can't prove the false positives are gone, since the probe measures the deployed server. Re-running the calibration after deploy is the actual confirmation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

bvandeusen added 2 commits 2026-07-30 08:07:38 -04:00

docs(knowledge): name the guard for the verification filter's two dialects ... 05b64ccacb

The location filter's section comment says THE TWO MUST CHANGE TOGETHER and
names tests/test_retrieval_scopes.py as what enforces it. The drift-check
filter added in #2086 has the same two-dialect structure and the same
hazard — a predicate applied in only one arm makes a record findable one
way and invisible another — but pointed at no guard, so the next person
had to discover that tests/test_snippet_drift_check.py walks both.

Also names the case that motivated `attention` existing at all: an ok
verdict whose code_sha has gone stale is neither `drifted` nor
`unverified`, and is the one shape a reader is likely to think redundant
and remove.

Written while verifying the plugin fixes end-to-end — this edit is what
the write-path trigger fired on, correctly surfacing snippet #2192 as
prior art at this exact path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

fix(write-path): give the semantic arm its own threshold + a payload floor ... 1fb883b72f

Closes #2223. The write-path prior-art trigger's semantic arm shared
auto-inject's 0.55 threshold, which was tuned on prose. Code embeddings
sit on a much higher similarity floor — any two Python-shaped payloads
share keywords, indentation and structure — so 0.55 landed INSIDE the
noise band. Measured against the live instance:

  near-duplicate of a recorded helper   0.73-0.74   true positive
  unrelated colour math / Vue SFC / CSS 0.55-0.63   false positive
  `x = 1`                               0.58        false positive

6 of 8 probe payloads produced a nudge; 4 were noise. The margin gate
couldn't help — _AUTOINJECT_BAND is relative to the top hit, so with a
single hit it never engages.

Two gates are now the write-path arm's own:

- kb_writepath_threshold, default 0.68 — above every measured false
  positive, still 0.05 below both true positives. Auto-inject keeps
  0.55; it was tuned on prose and is not implicated. The comment this
  replaces explicitly reserved the split for when telemetry showed the
  surfaces wanted different values, so this is the change it described,
  not a reversal of it.
- WRITEPATH_MIN_CODE_CHARS = 48 non-whitespace chars, below which the
  semantic arm doesn't run at all. Whitespace is excluded so a deeply
  indented one-liner can't pass on padding. 48 sits under the smallest
  plausible reusable helper (~60) and well over a degenerate edit, so it
  errs toward keeping recall — precision is the threshold's job. This is
  the cheap half of the operator's #89 idea; the full length<->threshold
  curve stays open there, since they asked to brainstorm it rather than
  have a scale invented for them.

top_k stays shared — "how many titles at once" means the same thing on
both surfaces.

The existing tests were passing `code="x"` / `code="def f(): ..."` into
the semantic arm, i.e. exactly the payloads the floor now drops, so the
gate tests were never exercising a realistic payload. They now use a
REAL_CODE fixture, plus new coverage for the floor (trivial payload,
padding, place-arm unaffected, real helper passes) and a guard on the
constant itself.

Settings UI carries the new knob with the reasoning in its hint, and the
write-path checkbox no longer claims it shares the threshold.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

bvandeusen merged commit 0550bf4687 into main 2026-07-30 08:07:48 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#85