Commit Graph
2 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5 4e59af380a fix(tests): the config stand-in fell behind the real one, and ten arms silently no-opped (#4214)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / Python tests (push) Failing after 1m7s
CI & Build / Build & push image (push) Skipped
CI & Build / integration (push) Successful in 44s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI 7124 Python tests: 14 failed. One root cause behind ten of them, and the
failure was the exact one `tests/helpers.writepath_cfg`'s docstring already
warns about in prose — while being unable to prevent this instance of it.

Three arms read their numbers out of that config dict inside a fail-open
`except`. A missing key raises where nobody sees it, so the arm becomes a
silent no-op, indistinguishable from the arm working and finding nothing. The
helper derives its keys from `retrieval_surfaces.SURFACES` precisely to stop
that — and `checkpoint_threshold` is deliberately NOT a surface, because
everything in that table is a floor/budget pair belonging to one query and the
checkpoint runs none. The derivation therefore could not see it, the write-path
rule arm died before `record_retrieval`, and ten tests went red at once.

Fixed at the helper, from the module constant, so there is still exactly one
literal and it lives in the product. And the guard the docstring claimed now
exists: `test_the_config_stand_in_carries_every_key_the_real_one_does`
compares the stand-in's key set against the real `get_writepath_config`, so
the next key that is not a surface fails loudly here instead of quietly
disabling an arm under test. `test_retrieval_surfaces`'s hand-written key list
gains it for the same reason, spelled out in place.

The other four were the contract widening itself: `checkpoint` is present on
every return of the tool arm, including its early ones, so four assertions
comparing the whole dict needed it. That key is deliberately always present —
the two arms feed one shell reader where an absent key and an empty one are
read the same, so the difference is invisible exactly where it would bite.

Verified statically: every cfg in the suite now routes through `writepath_cfg`
(test_write_path_trigger's local `_cfg` delegates to it), no hand-written
config dict survives, and the real config's eight keys match the stand-in's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-21 00:30:20 -04:00
bvandeusenandClaude Opus 5 09b48457ff refactor(retrieval): one registry for every surface's floor and budget (#4102)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 50s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / Python tests (push) Failing after 1m3s
CI & Build / Build & push image (push) Skipped
Groundwork for the step's real change. The operator's decision is that the
floor is chosen and adjusted by the model using it, not shipped as a value
somebody has to defend:

  "we need a model consistent surface for the adjustment of these floor values.
   the user should be able to touch it but the model should be the thing
   handling it 9 times out of 10."

A tuning surface cannot be consistent across six arms that each spell their
configuration differently, so the arms stop owning their numbers.
`retrieval_surfaces.SURFACES` names each one, its floor key and default, its
budget key and default, and — because they are rendered by the tuning tool and
the Settings UI — what it asks, over what corpus, and how often it fires. A
floor cannot be moved responsibly by anyone who does not know those three.

Three things fall out:

- **`k` becomes a real budget everywhere.** Only auto-inject had a configurable
  one; `RULEHINT_LIMIT`, `PROMPTRULE_LIMIT` and `reply_preferences.LIMIT` were
  constants. `k` is what binds under a low floor, so it has to be settable per
  surface — and per surface is the point, since `pre_tool_rule` fires before
  every Bash call while `prompt_rule` fires once a turn.
- **`write_path` gets its own budget, inherited not reset.** It shared
  auto-inject's outright on the argument that "how many titles at once" means
  the same thing on both. It does not, for the same reason. Unset, it still
  reads auto-inject's key, so an install that tuned the shared knob does not
  silently drop to a new default.
- **The duplicated read-and-clamp goes.** That shape is canon #2860 across 295
  of 372 judged siblings. Survivable while the numbers were constants; not once
  they are meant to move.

The long measurement comments stay exactly where they are — #2223's noise-floor
probe, #3853's command-vs-code split, #3851's band measurement. The constants
they annotate now alias the registry, so there is one value and the reasoning
still sits beside it.

Tests build the write-path config from the registry (`helpers.writepath_cfg`)
instead of from hand-written dicts. That is not tidiness: the rule arms read
their numbers inside a fail-open `except`, so a dict missing one key does not
raise where a reader would see it — the arm silently becomes a no-op that reads
exactly like "fired and found nothing". Ten hand-written dicts each looked
complete on the day they were typed.

tests/test_retrieval_surfaces.py pins the identity everything rests on: a
surface's name IS its telemetry source. Nothing in the type system says so —
`record_retrieval(source="pre_tool_rule")` is a literal in another file — and
renaming one without the other yields an arm that can be tuned and not
measured, or measured and not tuned, with no symptom either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-17 11:17:23 -04:00