The wide net becomes a pull, and a tuned number carries the space it was measured in #164

Merged

bvandeusen merged 5 commits from dev into main 2026-09-17 18:50:50 -04:00

Conversation 0 Commits 5 Files Changed 19

+1577 -32

bvandeusen commented 2026-09-17 18:50:40 -04:00

Owner

Copy Link

Copy Source

Milestone 416 steps 5 and 6. CI green on a7d7368 (run 6980, 1742 passed).

#4103 — the wide net becomes a pull tool

what_might_apply(query) — up to fifty ranked candidates, no bar at all, graduated detail so the cost stays bounded. Before this, the only way to ask "is there a rule about what I'm about to do" was search at a threshold, which is exactly the surface that can hide the record you need.

The task claimed fifty candidates "cost nothing". _rule_hint_line had already measured ~143 tokens per full line, so fifty would be ~7,000. Resolved by reusing #3851's graduated shape: top full, tail compact.

The using-scribe skill's reflex now branches — what_might_apply before a consequential act, search(content_type="rule") when you already suspect a specific rule.

#4104 — a tuned number carries its calibration

A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor keeps applying while describing nothing.

• calibration_stamp() — embedding model + document shape. Two fields, never a fused string (rule 149): a mismatch has to say which half moved.
• retrieval_tuning_events gains embedding_model / shape_version (migration 0104), stamped on every write. Nullable and not backfilled — "unstamped" is the honest answer for older rows, and it reports as stale: null, never as fine.
• retrieval_surfaces and the Settings panel show staleness. The panel renders only when something is stale, so seeing it at all is the signal.
• migrate_retrieval_floor — the raw cosine can't cross models, but the percentile it represented can. Measures what fraction of a surface's logged calls the old floor admitted, re-scores those queries under the current model, proposes the value admitting the same fraction. Dry run by default.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be.

The load-bearing test is an absence: no chat-model identifier may appear anywhere in the calibration path. A Claude upgrade must trigger nothing, because a false alarm here costs the real alarm on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and back — a round trip must not turn "we don't know" into a stated fact.

After merging

1. Redeploy the server including alembic upgrade — migration 0104.
2. /plugin reload to pick up manifest 2026.09.17.1618; the executing cache only refreshes on a version change (#2209).

Note on what is already running

prompt_rule's floor is currently 0.60 (default 0.72) — a live experiment for #4105, recorded as tuning event id 1 with its reasoning attached. Settings live in the database, so the redeploy will not reset it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

Milestone 416 steps 5 and 6. CI green on `a7d7368` (run 6980, 1742 passed). ## #4103 — the wide net becomes a pull tool `what_might_apply(query)` — up to fifty ranked candidates, **no bar at all**, graduated detail so the cost stays bounded. Before this, the only way to ask "is there a rule about what I'm about to do" was `search` at a threshold, which is exactly the surface that can hide the record you need. The task claimed fifty candidates "cost nothing". `_rule_hint_line` had already measured ~143 tokens per full line, so fifty would be ~7,000. Resolved by reusing #3851's graduated shape: top full, tail compact. The `using-scribe` skill's reflex now branches — `what_might_apply` before a consequential act, `search(content_type="rule")` when you already suspect a specific rule. ## #4104 — a tuned number carries its calibration A retrieval floor is a cosine similarity, which only means something inside one embedding model's geometry over documents cut one particular way. Change either and every floor keeps applying while describing nothing. - `calibration_stamp()` — embedding model + document shape. Two fields, never a fused string (rule 149): a mismatch has to say *which* half moved. - `retrieval_tuning_events` gains `embedding_model` / `shape_version` (**migration 0104**), stamped on every write. Nullable and not backfilled — "unstamped" is the honest answer for older rows, and it reports as `stale: null`, never as fine. - `retrieval_surfaces` and the Settings panel show staleness. The panel renders **only** when something is stale, so seeing it at all is the signal. - `migrate_retrieval_floor` — the raw cosine can't cross models, but the percentile it represented can. Measures what fraction of a surface's logged calls the old floor admitted, re-scores those queries under the current model, proposes the value admitting the same fraction. **Dry run by default.** Nothing auto-retunes. A stale stamp says a number is no longer a measurement; it does not say what the number should be. The load-bearing test is an **absence**: no chat-model identifier may appear anywhere in the calibration path. A Claude upgrade must trigger nothing, because a false alarm here costs the real alarm on the day `bge-small` becomes `bge-base`. Backup **v17** carries both columns, unfilled on the way out and back — a round trip must not turn "we don't know" into a stated fact. ## After merging 1. Redeploy the server **including `alembic upgrade`** — migration 0104. 2. `/plugin` reload to pick up manifest `2026.09.17.1618`; the executing cache only refreshes on a version change (#2209). ## Note on what is already running `prompt_rule`'s floor is currently **0.60** (default 0.72) — a live experiment for #4105, recorded as tuning event id 1 with its reasoning attached. Settings live in the database, so the redeploy will not reset it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

bvandeusen added 5 commits 2026-09-17 18:50:40 -04:00

feat(retrieval): the wide net becomes a pull — fifty candidates, no bar (#4103) ... 381c90ca7e

Milestone 416 step 5. The operator's compromise — "if you're worried
about excluding potentially important data let limit it to 50 entries"
— moved to the surface where it is safe. Fifty in the push would be
milestone 394 with extra steps; fifty in a pull crowds nothing out.

`what_might_apply(query)` returns ranked rule candidates with NO
threshold. The moment it serves is the one where the caller does not
trust a bar to decide for them, so it does not have one — every row
carries its score and the reader judges.

WHY IT IS NOT A BIGGER `limit` ON `search`

`_search_rules` returns statement, why and how_to_apply in full, on the
stated reasoning that a caller who went looking deserves the whole
record. That is the DEEP pull and should stay that way. This is the
SHALLOW one — many candidates, each just enough to decide whether to
open it. Opposite trade-offs, so it is a second tool.

THE PREMISE THE STEP GOT WRONG

The task said fifty "costs nothing". `_rule_hint_line` had already
measured otherwise: ~143 tokens per line once the trigger is rendered,
and #3855 tripled trigger lengths across the corpus. Fifty is ~7,000
tokens — cheap next to an arm firing before every Bash call, but not
free, and a tool promising a free wide net gets reached for casually
and then regretted.

So it reuses the graduated shape #3851 measured for the push: the top
few carry their trigger whole, the rest carry a cut of it. TRUNCATED,
never dropped — the trigger is what lets a reader judge without
opening, and a teaser without one is just an id. The cut borrows
`_goal_line`'s technique including the fallback that matters (#4036):
`textwrap.shorten` returns a bare "…" for one unbroken word.

TELEMETRY

Logged under its own pull source, asserted absent from both the tunable
push registry and AMBIENT_SOURCES. That guard is load-bearing right
now: the push arms' near-miss distributions are the evidence #4121
argues from, and a pull folded into them would move those numbers.

INSTRUCTION SURFACES

The using-scribe reflex and the MCP instructions both pointed at
`search(content_type="rule")` for the consequential moment — the deep
tool, at the moment you want breadth. They now point here, and keep
`search` for reading a rule you already suspect. Written as a practice
rather than a prohibition (rule 165).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(plugin): a line break split the token a guidance guard matches on (#4103) ... dcf800ed65

CI 6973 — `test_guidance_ownership` on "rules are retrieved; ask before
a consequential act": the using-scribe skill was missing the marker
`content_type="rule"`.

It was there. I had hard-wrapped the new bullet across `content_type=`
/ `"rule"`, and that guard matches its MARKERS raw while normalising
whitespace only for the longer `statement` — which is why the failure
reported the statement present and the marker absent in the same
breath. Rewrapped so the token sits on one line.

Worth noting rather than just fixing: the markers are deliberately
raw-matched, since a marker IS a literal a reader copies. `tool_doc`
flattens for exactly the opposite reason one layer over. Both are
right; the trap is that they differ.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

feat(retrieval): a tuned number carries the space it was measured in (#4104) ... aee24c9c1c

Milestone 416 step 6. A retrieval floor is a cosine similarity, which only
means something inside one embedding model's geometry over documents cut one
particular way. Change either and every floor on the install keeps applying
while describing nothing — and nothing anywhere says so, because the scores
simply come out different and the bar goes on cutting.

`CHUNKER_VERSION` already solved this for documents: stamped per row, so the
backfill re-embeds precisely what is stale. The same idea, applied to the
numbers:

- `calibration_stamp()` — embedding model + document shape, one definition.
  TWO fields, never a fused string (rule 149): a mismatch has to say WHICH half
  moved, because they call for different responses.
- `retrieval_tuning_events` gains `embedding_model` / `shape_version`
  (migration 0104), stamped on every write. Nullable and NOT backfilled —
  "unstamped" is the honest answer for a row written before this existed, and
  it reports as `stale: null`, never as fine.
- `current_settings` reports calibration per dial: tuned rows from their event,
  untouched dials from the registry default's own stamp.
- `retrieval_surfaces` and the Settings panel show the mismatch. The panel
  renders ONLY when something is stale, so seeing it at all is the signal.
- `migrate_floor` / `migrate_retrieval_floor` answers "a path for thresholds to
  be inherited by the next model so that they don't have to recalibrate a lot":
  the raw cosine cannot cross models, but the PERCENTILE it represented can.
  Measure what fraction of a surface's logged calls the old floor admitted,
  re-score those queries under the current model, take the value admitting the
  same fraction. Dry run by default; applying writes an ordinary tuning event
  with the arithmetic in its reason.

Nothing auto-retunes. A stale stamp says a number is no longer a measurement;
it does not say what the number should be, and #4102 measured the one case
where the statistic and the correct action pointed opposite ways.

The load-bearing test is an ABSENCE: no chat-model identifier may appear
anywhere in the calibration path. Claude produces none of these scores, so a
Claude upgrade must trigger nothing — a false alarm here teaches the operator
to ignore the real one on the day bge-small becomes bge-base.

Backup v17 carries both columns, unfilled on the way out and on the way back:
a round trip must not turn "we don't know" into a stated fact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(tests): a Result's .all() is sync, and the module has a fourth tool (#4104) ... def144df06

Two failures in the tests added with the step, both mine.

`make_mock_session` is an AsyncMock and every child of an AsyncMock is one too,
so leaving `.all` as it came handed `migrate_floor` a coroutine where it reads a
list — the same trap the helper's own docstring already flags for `add`.

And the registration test enumerated three tools by name, which is exactly what
it is for: `migrate_retrieval_floor` made it four.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(retrieval): the newest two rows are not the newest row of each dial (#4104) ... a7d736860f

`current_settings` read a surface's history with one query and `limit(2)`, then
keyed the rows by dial. That is only the same thing while both dials have moved
equally often — and they do not. Floors get walked; budgets rarely move. Three
floor changes and one budget change returns two floor rows, and the budget
change vanishes.

Under #4102 that cost a missing reason. Since #4104 it is worse: the dial then
reports `source: "shipped"` — still on the value Scribe shipped — for a number
somebody deliberately tuned. A wrong calibration answer, in the direction a
reader has no cause to double-check.

One query per dial, `limit(1)` each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

bvandeusen merged commit f324be6b5a into main 2026-09-17 18:50:50 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#164