Write-path semantic arm: query snippets by concept, not raw code #86

Merged

bvandeusen merged 1 commits from dev into main 2026-07-30 10:21:19 -04:00

Conversation 0 Commits 1 Files Changed 2

+304

bvandeusen commented 2026-07-30 10:20:58 -04:00

Owner

Copy Link

Copy Source

One commit, 5778177. CI green on dev (run 3073, all six jobs).

Implements #2242, from the operator's question: would it make more sense to
search by the concept of the snippet than by the code itself?

It would, measurably. A snippet's embedded text is f"{title}\n{body}", and a
snippet's body is composed markdown — When to use / Signature / Location, then
the fenced code — so when_to_use appears twice in the vector and the
document is prose-forward. The semantic arm was interrogating it with raw code
carrying no prose at all. Measured on the deployed instance against #2222:

query built from	score	best unrelated	separation
raw code body	0.743	0.630	0.11
name + docstring	0.823	0.602	0.22
hand-written concept prose	0.835	0.583	0.25

A 12-word description beats a near-verbatim reimplementation of the function, and
code-as-query raises the noise floor.

The query is now built from declarations plus the first docstring / JSDoc /
leading comment, shaped "name(params) — what it does" to mirror a snippet's own
title. Server-side, so installed 0.1.20 plugins get it with no manifest bump.

Two rules that are measured, not chosen:

• No doc, no rewrite. A bare identifier scored 0.671 against #2222 vs the code
  body's 0.743 — same separation from noise (0.113), but below the 0.68 bar, so
  preferring it would flip a comfortable hit to a miss.
• The 48-char floor still judges the RAW payload, before the rewrite. Concept
  queries are often shorter than the floor and those are the good ones; applying
  the floor afterward would discard exactly the best queries.

Regex not a parser: an Edit's new_string is rarely a valid module, so a miss
must cost only a fallback. Unrecognised inputs (Vue SFC, config files) behave
exactly as before.

Telemetry logs the concept query rather than the code. Note this puts a seam in
retrieval_logs — rows before this commit are code-shaped, rows after are
concept-shaped, and the two aren't comparable when tuning.

0.68 left untouched: signal rises to 0.82 while noise falls to 0.58, moving the
bar from near-the-edge to mid-gap. To be re-measured after deploy, not assumed.

Deploy notes

No migration. No plugin change, so no manifest bump — server-side only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

One commit, `5778177`. CI green on `dev` (run 3073, all six jobs). Implements #2242, from the operator's question: would it make more sense to search by the concept of the snippet than by the code itself? It would, measurably. A snippet's embedded text is `f"{title}\n{body}"`, and a snippet's body is composed markdown — When to use / Signature / Location, then the fenced code — so `when_to_use` appears **twice** in the vector and the document is prose-forward. The semantic arm was interrogating it with raw code carrying no prose at all. Measured on the deployed instance against #2222: | query built from | score | best unrelated | separation | |---|---|---|---| | raw code body | 0.743 | 0.630 | 0.11 | | name + docstring | **0.823** | 0.602 | **0.22** | | hand-written concept prose | 0.835 | 0.583 | 0.25 | A 12-word description beats a near-verbatim reimplementation of the function, and code-as-query *raises* the noise floor. The query is now built from declarations plus the first docstring / JSDoc / leading comment, shaped `"name(params) — what it does"` to mirror a snippet's own title. Server-side, so installed 0.1.20 plugins get it with no manifest bump. **Two rules that are measured, not chosen:** - **No doc, no rewrite.** A bare identifier scored 0.671 against #2222 vs the code body's 0.743 — same separation from noise (0.113), but below the 0.68 bar, so preferring it would flip a comfortable hit to a miss. - **The 48-char floor still judges the RAW payload**, before the rewrite. Concept queries are often shorter than the floor and those are the good ones; applying the floor afterward would discard exactly the best queries. Regex not a parser: an Edit's `new_string` is rarely a valid module, so a miss must cost only a fallback. Unrecognised inputs (Vue SFC, config files) behave exactly as before. Telemetry logs the concept query rather than the code. Note this puts a seam in `retrieval_logs` — rows before this commit are code-shaped, rows after are concept-shaped, and the two aren't comparable when tuning. 0.68 left untouched: signal rises to 0.82 while noise falls to 0.58, moving the bar from near-the-edge to mid-gap. To be re-measured after deploy, not assumed. ## Deploy notes No migration. No plugin change, so no manifest bump — server-side only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

bvandeusen added 1 commit 2026-07-30 10:20:59 -04:00

feat(write-path): query snippets by concept, not by raw code ... 57781770c3

Implements #2242, from the operator's question: would it make more sense to
search by the concept of the snippet than by the code itself?

It would, measurably. A snippet's embedded text is f"{title}\n{body}", and a
snippet's body is composed markdown — When to use / Signature / Location, then
the fenced code — so `when_to_use` appears TWICE in the vector and the document
is prose-forward. The semantic arm was interrogating it with raw code carrying
no prose at all. Measured on the deployed instance against snippet #2222:

  query built from            score   best unrelated   separation
  raw code body               0.743   0.630            0.11
  name + docstring            0.823   0.602            0.22
  hand-written concept prose  0.835   0.583            0.25

A 12-word description beats a near-verbatim reimplementation of the function,
and code-as-query RAISES the noise floor. It's also the cleanest explanation for
the fragment miss recorded on #2223: a short excerpt carries almost no prose to
match a document that is mostly prose.

So build the query from what the code says it's FOR — declarations plus the
first docstring / JSDoc / leading comment block — shaped as "name(params) — what
it does", mirroring a snippet's own title, which is the form that measured 0.823.

Server-side rather than in the hook: no manifest bump, so installed 0.1.20
plugins get this immediately; multi-language parsing in bash would be miserable;
and it's unit-testable here.

Two rules worth calling out, both measured rather than chosen:

- NO DOC, NO REWRITE. A bare identifier is not a concept and scored 0.671 vs the
  code body's 0.743. Separation from noise is identical either way (0.113), but
  the absolute drops under the 0.68 bar, so preferring a bare name would convert
  a comfortable hit into a miss. Undocumented code keeps the raw payload.
- The 48-char floor still judges the RAW payload, before the rewrite. A concept
  query is allowed to be shorter than the floor — that is the point, the best
  queries are short — but a sub-floor edit stays silent even with a docstring.
  Applying the floor after extraction would discard the best queries.

Regex, not a parser: this is on a PreToolUse critical path and an Edit's
new_string is rarely a valid module, so a miss must cost only a fallback. Every
unrecognised language (Vue SFC, config files) degrades to exactly the previous
behaviour.

Telemetry now logs the concept query rather than the code, since retrieval_logs
is what the threshold gets tuned from and the two aren't comparable.

0.68 is left alone: signal rises to 0.82 while noise FALLS to 0.58, so the bar
sits mid-gap instead of near the edge. To be re-measured against the deployed
instance rather than assumed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs

bvandeusen merged commit a36837c96a into main 2026-07-30 10:21:19 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#86