Commit Graph
3 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5 003bfd7a0a fix(tests): two readers moved, and the settings guard now checks the registry (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 43s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Failing after 1m2s
CI & Build / Build & push image (push) Skipped
CI on 09b4845 caught both, and the second is an improvement rather than a
repair.

`test_services_reply_preferences` patched `reply_preferences.get_setting`,
which the registry refactor removed — its floor and budget are resolved through
`retrieval_surfaces` now. Its key-independence test also asserted the arm asks
for exactly one key; it asks for two, because both of its numbers are its own
since this step, so the assertion names both and adds a check that the registry
and the module constant still agree about the floor key. They writing different
keys is the failure where the Settings form saves one string and the arm reads
another.

`test_settings_defaults_agree` parsed `plugin_context.py` for a bare
module-level float, and those constants now alias the registry. Rather than
teach the regex about aliases, the six retrieval floors are keyed on their
SURFACE NAME and read from the registry directly — which is strictly better for
this guard: a surface name is also its telemetry source, so a row names the same
arm the readout does, and a floor cannot be checked against a stale constant
that happened to keep its old value. `PLAN_MATCH_DEFAULT_THRESHOLD` is not a
push surface and keeps the older shape, with a note saying why.

Added while there: `test_every_tunable_surface_has_a_control`, derived from the
registry, so a seventh surface arrives as a failing test rather than as a number
only the model can reach (rules 25, 27).

Also lands the audit trail this step needs — `retrieval_tuning_events` (model +
migration 0103) and `services/retrieval_tuning.py`. The tool layer on top is
the next commit; the table is here because `reason` being REQUIRED is the whole
guardrail, and the schema is where that starts. A number moved silently leaves
nothing for the operator to review or disagree with, and the operator's decision
is that the model moves these "9 times out of 10".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-17 11:20:47 -04:00
bvandeusenandClaude Opus 5 5ae60734bb fix(retrieval): the completion-report arm gets its own bar, not the prompt arm's (#3860)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 34s
`report_preference` shipped reading PROMPTRULE_THRESHOLD_KEY, so the two arms
were one dial: tuning the bar for an operator's prose silently retuned the
lookup that runs when a task closes.

That coupling is worse on this arm than it would be anywhere else. Every other
retrieval arm scores a query that varies per call, so a mis-set bar shows up as
a changed clear-rate. COMPLETION_QUERY is a fixed string, so this arm's best
score for a given corpus is a CONSTANT — and a constant sitting under the bar is
a dead arm rather than a quiet one. No volume of traffic reveals it.

Found by the first live read for milestone 394 step 9: 69 calls, 69 declines,
every one naming the same record at the same score (0.7194 against a 0.72 bar).
Reading `best_available_id` (#3807) showed the record was about interpreting a
REQUEST, not about report shape — so the declines were correct and the arm is
healthy. The percentile alone would have said "lower the bar", which would have
delivered a false positive on every completion report ever written.

The bar does not move; the key does. Both defaults stay 0.72, so this changes no
behaviour on any install — it makes "leave this one where it is" expressible,
which it was not before. Settings grows the control (rules 25, 27), and the
default-agreement check grows a row.

Deliberately NOT included: a change to PROMPTRULE_DEFAULT_THRESHOLD. The
evidence for moving it is this install's near-miss table, and rule 115 keeps a
shipped default from being justified by one instance's corpus. That bar is a
per-user setting and belongs in the operator's Settings, not in the product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-16 13:47:34 -04:00
bvandeusenandClaude Opus 5 921565696c feat(409): an operator's own reply shapes reach the reply they are about (#4013)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / Python tests (push) Successful in 1m24s
CI & Build / Build & push image (push) Successful in 23s
The reporting-back skill ships default shapes; an operator's adjustments to
them are preference records. Prompt-time retrieval matches the operator's
message, and a shape preference is about the reply, so those preferences were
on file and never arrived. Operator's decision (logged on #4013): the server
delivers them for a completion report, and the skill asks for every other kind.

- Completion reports (option C): closing a task with update_task runs a
  kind-filtered preference search for the moment "writing the completion
  report after finishing a task" and returns matches as `reply_preferences`
  ({id, title, statement, kind}), with a sentence added to `report_back`
  naming the key. A preference says it is about completion reports through
  its own when_to_apply; no tag or column. Omitted when nothing matches, and
  the lookup fails open.
- Telemetry: every call logs to retrieval_logs under `report_preference`
  (empty calls included; a search that never ran writes no row) and hits are
  recorded surfaced. The source is ranked, so it counts toward pull-through.
  The bar is the prompt arm's setting until step 6 reads this source's near
  misses.
- Every other reply (option A): reporting-back gains "The operator's own
  shapes come first". Before a finding, decision, handoff or "where are we",
  search(content_type="rule") in the words of that moment and follow what
  comes back. Registered in the ownership guard with reporting-back as owner.
- Loading reply shapes at session start (option B) was rejected: it would be
  a small copy of the preloading milestone 394 retired.

Domain-neutral query (pinned); works on an install with no preferences.
Plugin version minted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 17:43:16 -04:00