dev → main: rule-arm review, moment usage by rule, no-subject auto-inject, copy check, S3 test wording #215

Merged

bvandeusen merged 15 commits from dev into main 2026-10-22 20:26:40 -04:00

Conversation 0 Commits 15 Files Changed 38

+1243 -387

bvandeusen commented 2026-10-22 20:26:32 -04:00

Owner

Copy Link

Copy Source

• #4636: retrieval_telemetry rule_usage.by_source; list_rules with_usage and confirmed lesson counts
• #6247: pull points and rules-listing points are no longer listed as silent
• #6252: auto-inject skips the notes arm for a prompt with no subject ("yes", "go ahead")
• #6251: menus_to_review re-runs prompt_rule, pre_tool_rule and write_path_rule
• #6254: moment_usage gives each moment by_rule (delivered, opened), shown in Moments settings
• #6253: the rule-moment proposal queue lists a signal pair only once it has proposed
• #6255: backup Test destination says the address answered but not as S3
• #6249: the write-path hint asks whether a new definition copies a close snippet

CI green on dev (run 9706).

🤖 Generated with Claude Code

- #4636: retrieval_telemetry `rule_usage.by_source`; list_rules `with_usage` and confirmed lesson counts - #6247: pull points and rules-listing points are no longer listed as silent - #6252: auto-inject skips the notes arm for a prompt with no subject ("yes", "go ahead") - #6251: menus_to_review re-runs prompt_rule, pre_tool_rule and write_path_rule - #6254: moment_usage gives each moment `by_rule` (delivered, opened), shown in Moments settings - #6253: the rule-moment proposal queue lists a signal pair only once it has proposed - #6255: backup Test destination says the address answered but not as S3 - #6249: the write-path hint asks whether a new definition copies a close snippet CI green on dev (run 9706). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 15 commits 2026-10-22 20:26:32 -04:00

align(4909d): every rule arm that annotates reads one bar, kb_rule_threshold at 0.70 ... c8ede427ca

prompt_rule, pre_tool_rule and write_path_rule now share one settings key,
RULE_FLOOR_KEY = "kb_rule_threshold", default 0.70. The reply backstop keeps
its own 0.80, because there the floor is a stop bar. Moving the shared floor
through tune_retrieval or the Settings form records a tuning event on every
surface that reads it (set_dial, via retrieval_surfaces.sharing_floor), and
the result names the surfaces it carried as `moved_with`. Settings shows one
"Standing-rule confidence threshold" control where it showed three.

Operator ruling (2026-10-05): one shared bar for the rule arms, "especially
since we're only taking the first x number of them" -- the budget limits
what is shown; the bar only decides what is worth ranking. Kept a dial.

Why 0.70, from records read rather than percentiles:
- write_path_rule near-misses at 0.7199-0.72 were all false positives
  (rule 149 against a Lidarr comment, 157 against a retention Svelte file,
  121 against an Archives restore, 164 against a config test, 131 against a
  Go package), so the old 0.72 was not hiding real hits -- but the band does
  noise control on this arm, so dropping 0.02 costs little.
- prompt_rule: the operator set 0.70 by hand on 2026-09-18, after a 0.60
  experiment admitted junk and was reverted. 0.70 is the number with a human
  decision behind it.
- pre_tool_rule lines near 0.68 this session were mostly noise; rising to
  0.70 trims them, and the band still narrows what clears.

Retired keys: kb_promptrule_threshold, kb_toolrule_threshold and
kb_rulehint_threshold are no longer read. Rows an install stored under them
stay in the settings table, inert. On this instance the effective bars move
pre_tool_rule 0.68 -> 0.70, write_path_rule 0.72 -> 0.70, prompt_rule stays
0.70; record_release_defaults writes those per-surface release rows on boot,
so the telemetry shift has an event to explain it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(4909d): a shared floor move records the named surface first, then its siblings ... 6299335254

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

align(4909c): the command and write rule arms reserve a line for a preference ... c3933507c9

PRE_TOOL_RULE and WRITE_PATH_RULE now take preference_slot=True, like
PROMPT_RULE. When no preference placed on score, the arm asks once more for
the best preference over the same bar, and shows it as one extra line.

Operator ruling (2026-10-05): yes. A preference about how to act ("let each
action land before the next") is ranked against every rule whose trigger
names the same command; the act is its last chance to shape anything.

Cost: one kind-filtered search per call that placed no preference. The
query embedding is memoised per request on both hook routes, so no extra
model call. Expect a new stream of preference_slot rows from the two act
arms; preference_slot's call count will rise roughly by pre_tool_rule's and
write_path_rule's (~23k calls over 14 days before this change), and its
zero-result share will rise with it, since most commands have no matching
preference.

Tests: the tool-arm call-row tests count rows under pre_tool_rule only (the
slot logs under its own name), and the write-arm bar test reads the arm's
own search, which is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(4909c): the tool-arm suppression tests read the arm's own row, not the slot's ... 407f594a53

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

align(4909b): every write with a path gets its rule check, whether or not prior art matched ... 71260a28e2

The write-path rule arm ran only past the notes guard: a write whose code
matched no snippet, issue or lesson returned before any rule was searched.
It now runs before that guard, on `code or path`, and a write that matched
nothing but a rule still returns the rule lines.

Operator ruling (2026-10-05): yes. A rule about the file being written does
not depend on a snippet resembling it.

Cost: one more rule search per write that matched no prior art, with its
own embedding (the query is `code or path`, not the notes arm's concept
query, so the per-request memo does not cover it). Plus the preference slot
from (c) on those writes.

Expected telemetry shift, to be read on its own once this reaches main:
write_path_rule calls rise toward write_path's count (14 days to 2026-10-22:
write_path_rule 4,634 against write_path 3,786 — the notes arm skips trivial
payloads, the rule arm does not, so the rule arm will now exceed it). Its
zero-result share rises too, since the newly covered writes are the ones
that matched nothing in the notes corpus.

Tests: the gate test is inverted (a write with no prior art still gets its
rule check); the design-only write now runs the rule arm on its path; two
notes-arm tests read the write_path row instead of any row, since the rule
arm writes its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(4636): rule pull-through per surface, and per-rule usage with confirmed lessons on list_rules ... d05d4f3632

retrieval_telemetry gains rule_usage.by_source, the twin of usage.by_source:
distinct rules each surface showed, how many an agent opened, and a ratio
that is null on a bulk delivery. list_rules(with_usage=True, days=N) adds
each rule's usage, windowed, and how many lessons are confirmed instances
of it, so linked and unlinked rules can be compared through MCP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6247): a getter is seen by its pulls, so the silent list stops naming every pull door ... 1fa9245bbf

usage and rule_usage gain pulled_by_source, read from the per-source rows
the readout already had, and _silent_surfaces counts those doors as having
emitted. The rules-listing points were seen as of the previous commit,
through rule_usage.by_source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6252): a prompt that names nothing gets no notes menu ... dac4c59e07

"keep going", "yes please do", "is it deployed?": with no word of its own the
auto-inject query was the previous reply's tail alone, and the menu handed
back the records that reply was built from. In a sample of 59 logged calls,
28 came from short prompts; those with a subject of their own ("continue into
DRY 14", "fix the Windows gate") led with the record meant, while the ones
without were finished tasks already in context, with an open on 2 of about 15.

_has_subject checks the operator's words against a list of words that name
nothing; without one, the ranked menu is skipped and records named by number
are still shown. The prompt rule arm is unchanged and still searches with the
tail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6252): the named-record test keeps its number after "with" ... 4297c88cd2

"in 4448" is not read as a reference, so the subject went after the number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6251): the rule arms can be reviewed — menus_to_review re-runs prompt_rule, pre_tool_rule and write_path_rule ... 7c1d98dd2a

A rule arm logs its query, floor, budget and project, which is everything
semantic_search_rules takes, so a logged call is re-run to a depth past its
budget and judged like an auto_inject menu. within_budget is the arm's own
cut: inside the budget and, for the banded command and write arms, inside the
band of the top hit. "Opened after" reads rule pulls, and rules created
after the call are dropped as notes are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6251): the rules warning sends its reader to the rule-arm review ... 95ef4af746

surfaced_never_pulled told a rules reader there was no judged sample for the
rule arms. There is one now; the guard tying the text to REVIEWABLE caught it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6254): moment_usage says which rule each moment delivered, and whether it was opened ... 67d3c68fbf

Each moment gains by_rule: {rule_id, title, delivered, opened}, read from
the (moment, rule) grouping the readout already did. The moment's totals are
now summed from those rows, so the two cannot disagree. The Moments settings
page shows the list under each moment, closed by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6253): the moment-proposal queue lists a signal pair only once it has proposed ... d51ccd9b69

pending() listed every suggested signal row, including pairs still gathering
evidence below the three-situation bar, so the queue held 200 "proposals"
where about 25 had crossed it. A signal pair now needs proposed_at in its
evidence, the same test the misfire half already applies; proposals from a
pass are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6255): Test destination names an address that did not answer as S3 ... 0a4eb313fe

A 4xx with no S3 error body came from something else at that address, most
often a web server on the default port. The message now says so and to check
the scheme and port, beside the raw status. A 5xx without a body keeps its
own reason, since a busy S3 service can answer that way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6249): the write-path hint asks whether a new definition copies a close snippet ... db2427a3dc

The derive check matches by name, so a renamed copy of a recorded snippet
passed it and showed only as "similar" prior art. When a write introduces a
definition the file does not hold (shape_ledger.new_shapes_at) and an unseen
snippet on the menu clears the write-path floor by WRITEPATH_COPY_MARGIN (0.80
at the default floor), the hint now says to reuse it or extract the shared
shape before another copy. Names the derive check or the stamping feed already
spoke for are left to them. Covers recorded snippets only: the ledger keeps no
bodies to compare an unrecorded sibling against.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bvandeusen merged commit a7f3e0e972 into main 2026-10-22 20:26:40 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledGrimoire#215