A rule must say when it applies; the ledger says what was read, not what was shown #162

Merged

bvandeusen merged 6 commits from dev into main 2026-09-16 17:50:08 -04:00

Conversation 0 Commits 6 Files Changed 29

+766 -48

bvandeusen commented 2026-09-16 17:50:02 -04:00

Owner

Copy Link

Copy Source

Six commits: the first two steps of milestone 416 "Retrieval stops guessing a bar — candidates ranked, a ledger of what is actually held, and numbers that survive a model change", plus a fix that milestone's opening investigation turned up.

#3860 — the completion-report arm gets its own bar (5ae6073)

report_preference was reading PROMPTRULE_THRESHOLD_KEY, so the two arms were one dial: tuning the bar for an operator's prose silently retuned the lookup that runs when a task closes.

That coupling is worse on this arm than anywhere else. Every other retrieval arm scores a query that varies per call, so a mis-set bar shows up as a changed clear-rate. COMPLETION_QUERY is a fixed string, so its best score against a given corpus is a constant — and a constant sitting under the bar is a dead arm rather than a quiet one. No volume of traffic reveals it.

Found by the first live read for milestone 394 step 9: 69 calls, 69 declines, every one naming the same record at the same score (0.7194 against a 0.72 bar). Reading best_available_id showed the record was about interpreting a REQUEST, not report shape — so the declines were correct and the arm is healthy. The percentile alone would have said "lower the bar", which would have delivered a false positive on every completion report ever written.

No behaviour change on any install: both defaults stay 0.72. What changes is that "leave this one where it is" became expressible.

#4099 — a rule cannot be created, or edited into, having no trigger (07bdbf1, bb6ab0f)

when_to_apply is not metadata. rule_document embeds a rule as {title} — {trigger} / When to apply: {trigger}\n\n{statement}, the trigger appearing twice so purpose dominates a short vector. Without one the document silently becomes title + statement — a DIFFERENT shape, ranked against a corpus it does not match, with nothing to report it.

create_preference has refused an empty trigger since it shipped; the two rule creators defaulted it to "". The shape was enforced for the record kind that guides and optional for the kind that binds.

The guard lives in the service, because both doors reach it — the MCP tools and the frontend's fast path in routes/rulebooks.py. Written in either alone, the other could still create a rule that never fires.

update_rule refuses to EMPTY an existing trigger, checked after the mutation so it covers clear=[...], an emptied form field, and any route added later. Asked as "did this edit remove one" rather than "does one exist": a rule predating the guard has none, and refusing to save it would freeze precisely the unreachable records that most need fixing.

15 test fixtures across 6 files were creating rules with no trigger. That they were is the argument for the guard — this install shows 0/65 empty only because they were fixed by hand.

#4100 — the ledger records what was OPENED (ad26b3f, ebd6cb2, 957a72c)

Milestone 386 made a repeat REFERENCED rather than withheld, and the line it chose says "You saw it earlier this session". Nothing ever checked that. The arms emit a TEASER — title, trigger, get_rule(N) — so a session can be shown a rule twenty times and never read a word of it, and a compaction summarises the teaser away leaving nothing behind.

Three states now:

ledger state	line
never surfaced	it is not in this session's loaded set
named, not opened	Mentioned earlier this session but not opened — read it with get_rule(N) …
opened	You opened it earlier this session; pull it with get_rule(N) again…

The middle one is the honest one and the one that was missing.

How "opened" became observable: a new PostToolUse hook watches the get_rule call itself. PostToolUse does fire for MCP tools — the event's own output schema carries updatedMCPToolOutput, which would be meaningless otherwise — and the matcher leaves the server segment unpinned because it varies by install.

This is not the self-report 386 rejected. That objection was to ASKING a model whether it holds a rule, which is unverifiable. A tool call is an event the harness reports whether anyone asks.

Both ledgers clear together on compact/clear: keeping .opened.ids across a compaction would tell a freshly-summarised session "you opened it earlier" about a rule now nowhere in its context — a more confident version of the bug being removed.

Also closes two smoke-coverage holes check_plugin.py was reporting as SKIP, including scribe_precompact_preserve.sh from #3680. That one needed STATIC_FLOOR to become a set: PreCompact's contract is inverted — its stdout BECOMES the summarizer's instructions, so silence is its failure mode and a generic read of it looks like a leak.

Verification

CI run 6947 green on 957a72c: Python lint, TypeScript typecheck, plugin hooks, Python tests, integration.

The CI arc for #4100 was 11 → 1 → 0 failures, and none was a defect in the shipped code — every one was an existing guard noticing that seen had stopped meaning "you saw it", which is exactly what the step set out to change.

Plugin minted 2026.09.16.2102. It ships from main, so this merge is what puts the new hook in front of a session; /plugin + /reload-plugins picks it up (#2209 — the cache only refreshes on a version bump). Expect 9 hooks, up from 8.

Not verified by this PR: the three-state ledger only proves itself in a session that opens a rule and later meets it again. That is milestone 416 step 7's acceptance (#4105).

🤖 Generated with Claude Code

https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

Six commits: the first two steps of milestone 416 "Retrieval stops guessing a bar — candidates ranked, a ledger of what is actually held, and numbers that survive a model change", plus a fix that milestone's opening investigation turned up. ## #3860 — the completion-report arm gets its own bar (`5ae6073`) `report_preference` was reading `PROMPTRULE_THRESHOLD_KEY`, so the two arms were one dial: tuning the bar for an operator's prose silently retuned the lookup that runs when a task closes. That coupling is worse on this arm than anywhere else. Every other retrieval arm scores a query that varies per call, so a mis-set bar shows up as a changed clear-rate. `COMPLETION_QUERY` is a fixed string, so its best score against a given corpus is a **constant** — and a constant sitting under the bar is a dead arm rather than a quiet one. No volume of traffic reveals it. Found by the first live read for milestone 394 step 9: 69 calls, 69 declines, every one naming the same record at the same score (0.7194 against a 0.72 bar). Reading `best_available_id` showed the record was about interpreting a REQUEST, not report shape — so the declines were correct and the arm is healthy. The percentile alone would have said "lower the bar", which would have delivered a false positive on every completion report ever written. **No behaviour change on any install**: both defaults stay 0.72. What changes is that "leave this one where it is" became expressible. ## #4099 — a rule cannot be created, or edited into, having no trigger (`07bdbf1`, `bb6ab0f`) `when_to_apply` is not metadata. `rule_document` embeds a rule as `{title} — {trigger}` / `When to apply: {trigger}\n\n{statement}`, the trigger appearing twice so purpose dominates a short vector. Without one the document silently becomes title + statement — a DIFFERENT shape, ranked against a corpus it does not match, with nothing to report it. `create_preference` has refused an empty trigger since it shipped; the two rule creators defaulted it to `""`. The shape was enforced for the record kind that guides and optional for the kind that binds. The guard lives in the **service**, because both doors reach it — the MCP tools and the frontend's fast path in `routes/rulebooks.py`. Written in either alone, the other could still create a rule that never fires. `update_rule` refuses to EMPTY an existing trigger, checked after the mutation so it covers `clear=[...]`, an emptied form field, and any route added later. Asked as "did this edit remove one" rather than "does one exist": a rule predating the guard has none, and refusing to save it would freeze precisely the unreachable records that most need fixing. 15 test fixtures across 6 files were creating rules with no trigger. That they were is the argument for the guard — this install shows 0/65 empty only because they were fixed by hand. ## #4100 — the ledger records what was OPENED (`ad26b3f`, `ebd6cb2`, `957a72c`) Milestone 386 made a repeat REFERENCED rather than withheld, and the line it chose says "You saw it earlier this session". Nothing ever checked that. The arms emit a TEASER — title, trigger, `get_rule(N)` — so a session can be shown a rule twenty times and never read a word of it, and a compaction summarises the teaser away leaving nothing behind. Three states now: | ledger state | line | |---|---| | never surfaced | *it is not in this session's loaded set* | | named, not opened | *Mentioned earlier this session but not opened — read it with `get_rule(N)` …* | | opened | *You opened it earlier this session; pull it with `get_rule(N)` again…* | The middle one is the honest one and the one that was missing. **How "opened" became observable**: a new PostToolUse hook watches the `get_rule` call itself. PostToolUse does fire for MCP tools — the event's own output schema carries `updatedMCPToolOutput`, which would be meaningless otherwise — and the matcher leaves the server segment unpinned because it varies by install. This is **not** the self-report 386 rejected. That objection was to ASKING a model whether it holds a rule, which is unverifiable. A tool call is an event the harness reports whether anyone asks. Both ledgers clear together on compact/clear: keeping `.opened.ids` across a compaction would tell a freshly-summarised session "you opened it earlier" about a rule now nowhere in its context — a more confident version of the bug being removed. Also closes two smoke-coverage holes `check_plugin.py` was reporting as SKIP, including `scribe_precompact_preserve.sh` from #3680. That one needed `STATIC_FLOOR` to become a set: PreCompact's contract is inverted — its stdout BECOMES the summarizer's instructions, so silence is its failure mode and a generic read of it looks like a leak. ## Verification CI run 6947 green on `957a72c`: Python lint, TypeScript typecheck, plugin hooks, Python tests, integration. The CI arc for #4100 was 11 → 1 → 0 failures, and none was a defect in the shipped code — every one was an existing guard noticing that `seen` had stopped meaning "you saw it", which is exactly what the step set out to change. **Plugin minted `2026.09.16.2102`.** It ships from `main`, so this merge is what puts the new hook in front of a session; `/plugin` + `/reload-plugins` picks it up (#2209 — the cache only refreshes on a version bump). Expect 9 hooks, up from 8. **Not verified by this PR**: the three-state ledger only proves itself in a session that opens a rule and later meets it again. That is milestone 416 step 7's acceptance (#4105). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

bvandeusen added 6 commits 2026-09-16 17:50:02 -04:00

fix(retrieval): the completion-report arm gets its own bar, not the prompt arm's (#3860) ... 5ae60734bb

`report_preference` shipped reading PROMPTRULE_THRESHOLD_KEY, so the two arms
were one dial: tuning the bar for an operator's prose silently retuned the
lookup that runs when a task closes.

That coupling is worse on this arm than it would be anywhere else. Every other
retrieval arm scores a query that varies per call, so a mis-set bar shows up as
a changed clear-rate. COMPLETION_QUERY is a fixed string, so this arm's best
score for a given corpus is a CONSTANT — and a constant sitting under the bar is
a dead arm rather than a quiet one. No volume of traffic reveals it.

Found by the first live read for milestone 394 step 9: 69 calls, 69 declines,
every one naming the same record at the same score (0.7194 against a 0.72 bar).
Reading `best_available_id` (#3807) showed the record was about interpreting a
REQUEST, not about report shape — so the declines were correct and the arm is
healthy. The percentile alone would have said "lower the bar", which would have
delivered a false positive on every completion report ever written.

The bar does not move; the key does. Both defaults stay 0.72, so this changes no
behaviour on any install — it makes "leave this one where it is" expressible,
which it was not before. Settings grows the control (rules 25, 27), and the
default-agreement check grows a row.

Deliberately NOT included: a change to PROMPTRULE_DEFAULT_THRESHOLD. The
evidence for moving it is this install's near-miss table, and rule 115 keeps a
shipped default from being justified by one instance's corpus. That bar is a
per-user setting and belongs in the operator's Settings, not in the product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

feat(rules)!: a rule cannot be created, or edited into, having no trigger (#4099) ... 07bdbf1647

`when_to_apply` is not metadata. `rule_document` embeds a rule as
`{title} — {trigger}` / `When to apply: {trigger}\n\n{statement}`, the trigger
appearing twice so purpose dominates a short vector — the shape note 2485
measured on snippets (a 0.153 top-to-second gap against 0.010–0.023 for
everything else). Without one the document silently becomes title + statement:
a DIFFERENT shape, ranked against a corpus it does not match, with nothing to
report it. Every bar and every rank in the system assumes one shape.

`create_preference` has refused an empty trigger since it shipped. The two rule
creators defaulted it to "" — so the shape was enforced for the record kind
that guides and optional for the kind that binds.

The guard lives in the SERVICE, because both doors reach it: the MCP tools and
the frontend's fast path in routes/rulebooks.py. Written in either alone, the
other could still create a rule that never fires. The route keeps a matching
check for the STATUS CODE only (400, not the 404 it maps ValueError to).

update_rule refuses to EMPTY an existing trigger, checked after the mutation so
it covers `clear=[...]`, an emptied form input, and any route added later.
Deliberately asked as "did this edit remove one" rather than "does one exist":
a rule predating the guard has none, and refusing to save it would freeze
precisely the unreachable records that most need fixing.

Deliberately not following arose_from_id, which the human door exempts itself
from because provenance is about auditing what the AGENT changed. That reasoning
does not reach this field — a missing trigger is not a missing explanation, it
is a rule that does not work, and it fails an operator as badly as a session.

15 test fixtures across 6 files were creating rules with no trigger. They now
pass one; that they did not is the point — curation is not a guarantee.

Step 1 of milestone 416 "Retrieval stops guessing a bar". First because every
later step assumes one document shape, and it is much cheaper to guarantee
before a corpus grows than to backfill after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(tests): the trigger backfill assumed keyword calls; three fixtures pass positionally (#4099) ... bb6ab0f2a8

CI run 6937 red — 3 collection errors, `positional argument follows keyword
argument`, in the three integration fixtures that call the service positionally
(`create_rule(topic.id, uid, "title", "statement")`). The script that added
`when_to_apply=` to 15 fixtures inserted it as the FIRST argument, which is
valid only where every other argument is already a keyword.

Moved to the last argument in every call, which is legal in both styles, and
the continuation indent now matches the surrounding arguments.

Also repairs self-inflicted damage: the same script added a trigger to the two
tests in test_rule_trigger_required.py whose whole purpose is to call the
creators WITHOUT one. They would have stopped raising and the guard's own
proof would have inverted — a test that passes for the opposite reason than
the one it names, which is worse than a failing one.

Caught by ast.parse across tests/ rather than by the next CI round trip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

feat(retrieval): the ledger records what was OPENED, not merely what was shown (#4100) ... ad26b3f458

Milestone 386 made a repeat REFERENCED rather than withheld, and the line it
chose says "You saw it earlier this session". Nothing ever checked that. The
arms emit a TEASER — title, trigger, get_rule(N) — so a session can be shown a
rule twenty times and never read a word of it, and a compaction summarises the
teaser away leaving nothing behind. The server was asserting something about
the reader's context it had no way to know.

Three states now, where there were two:

  never surfaced   "it is not in this session's loaded set"
  named, unopened  "Mentioned earlier this session but not opened — read it…"
  opened           "You opened it earlier this session; pull it… again"

The middle one is the honest one and the one that was missing. It keeps the
full invitation, because a session that skipped a teaser is in nearly the
position of one never shown it.

HOW "OPENED" BECOMES OBSERVABLE. A new PostToolUse hook watches the get_rule
call itself and appends to `<sid>.opened.ids`. PostToolUse does fire for MCP
tools — the event's own output schema carries `updatedMCPToolOutput`, which
would be meaningless otherwise — and the matcher is `mcp__.*__get_rule` so the
server segment, which varies by install, is not pinned.

This is NOT the self-report 386 rejected. That objection was to ASKING a model
whether it holds a rule, which is unverifiable. A tool call is an event the
harness reports whether anyone asks. Recording what a session DID and believing
what it SAYS about itself are different kinds of evidence.

Both ledgers clear together on compact/clear. Keeping `.opened.ids` across a
compaction would have the arms telling a freshly-summarised session "you opened
it earlier" about a rule now nowhere in its context — a more confident version
of the bug being removed. Same reader (scribe_rules_live) for both, so ageing,
last-entry-wins and the bare-id format are defined once.

Also closes two smoke-coverage holes the checker was reporting as SKIP: the new
recorder, and scribe_precompact_preserve.sh from #3680. The latter needed
STATIC_FLOOR to become a set — PreCompact's contract is inverted, its stdout
BECOMES the summarizer's instructions, so silence is its failure mode and a
generic read of it looks like a leak.

Step 2 of milestone 416, and a hard prerequisite for step 4: while suppression
keys on shown, widening k marks records "seen" faster than they are read, and
the ledger would degrade in proportion to the improvement.

Plugin minted 2026.09.16.1232 -> 2026.09.16.2102.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(tests): the repeat tail is the middle state now, not the opened one (#4100) ... ebd6cb203c

CI run 6943: 11 failures, two causes, both existing guards correctly catching
the behaviour change rather than defects in it.

TEN were `_SEEN_TAIL = "You saw it earlier this session"`. Every one of those
tests drives an arm with `exclude_rule_ids` alone — the NAMING ledger — which
since this change is the middle state, not the opened one. That is the whole
point of the step: `seen` stopped meaning "you saw it". The constant now holds
the middle tail, so each test keeps asserting exactly what it was written to
assert (a repeat gets a line distinct from a first surfacing) against the
wording that is now true.

Added `_HELD_TAIL` and a negative assertion with it: neither call names an
opened ledger, so neither may claim the session read anything. That catches
`held` defaulting true, which would have every repeat assert the strongest of
the three claims on no evidence — and none of the ten existing tests would
have noticed.

THE ELEVENTH was my own test edit being wrong, not the code. I asserted
`held_rule_ids` would appear in `re.findall(r"printf '([a-z_]+)=", defs)`, but
`scribe_held_query` spells it `printf '&held_rule_ids=`, and the leading `&`
means that regex never matches it. The regex enumerates SCOPE keys — the
alternatives that open a query — and the ledger key is appended to a query
that already has one. Two different contracts that happened to share a file,
so they are now asserted separately and the difference is written down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

fix(tests): the compact band pins three holding states, not two (#4100) ... 957a72c501

CI run 6945: 11 failures down to 1. The survivor is
test_shortening_a_line_does_not_decide_what_it_says_about_holding, which
asserted "no longer hold it" appears in the compact line for seen=True — the
phrase that now belongs to the OPENED state, not the named one.

Its subject is a property, not a string: `compact` and the ledger are
independent axes, and shortening a line must not change what it claims about
holding. So the fix follows the axis rather than swapping the phrase. The axis
grew from two states to three, and the test now checks all three are distinct
under compact — pinning only two would let the compact branch collapse the new
middle state into either neighbour, which is the same regression it was
written for with one more place to hide.

Added the assertion that matters most when room is short: a line the session
never opened must not imply it did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

bvandeusen merged commit 91cfa8da97 into main 2026-09-16 17:50:08 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#162