Files
FabledScribe/tests/test_settings_defaults_agree.py
T
bvandeusenandClaude Opus 5 5ae60734bb
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 48s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 34s
fix(retrieval): the completion-report arm gets its own bar, not the prompt arm's (#3860)
`report_preference` shipped reading PROMPTRULE_THRESHOLD_KEY, so the two arms
were one dial: tuning the bar for an operator's prose silently retuned the
lookup that runs when a task closes.

That coupling is worse on this arm than it would be anywhere else. Every other
retrieval arm scores a query that varies per call, so a mis-set bar shows up as
a changed clear-rate. COMPLETION_QUERY is a fixed string, so this arm's best
score for a given corpus is a CONSTANT — and a constant sitting under the bar is
a dead arm rather than a quiet one. No volume of traffic reveals it.

Found by the first live read for milestone 394 step 9: 69 calls, 69 declines,
every one naming the same record at the same score (0.7194 against a 0.72 bar).
Reading `best_available_id` (#3807) showed the record was about interpreting a
REQUEST, not about report shape — so the declines were correct and the arm is
healthy. The percentile alone would have said "lower the bar", which would have
delivered a false positive on every completion report ever written.

The bar does not move; the key does. Both defaults stay 0.72, so this changes no
behaviour on any install — it makes "leave this one where it is" expressible,
which it was not before. Settings grows the control (rules 25, 27), and the
default-agreement check grows a row.

Deliberately NOT included: a change to PROMPTRULE_DEFAULT_THRESHOLD. The
evidence for moving it is this install's near-miss table, and rule 115 keeps a
shipped default from being justified by one instance's corpus. That bar is a
per-user setting and belongs in the operator's Settings, not in the product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2026-09-16 13:47:34 -04:00

94 lines
4.2 KiB
Python

"""The Settings UI shows the default the server actually uses (#3927).
WHY THIS EXISTS
A retrieval threshold lives in two places by necessity: a Python constant the
arm reads when no row is stored, and a Vue `ref` the Settings form shows when
the operator has never touched it. Neither can import the other.
So the form's initial value is a CLAIM about the server's behaviour, and it is
the kind of claim that rots quietly. Retune the Python constant and the input
keeps rendering the old number — the operator reads it as the bar in force,
sees no reason to change anything, and the form has misinformed them about the
one fact it exists to convey. Nothing errors, nothing looks wrong, and the
value they are shown is simply not the value being applied.
WHAT THIS PINS
The RELATIONSHIP, never the number: for each threshold, the Vue ref's initial
string equals the Python default. Retuning either stays free as long as both
move — which is the point, because these are tuning values and #3853 moved one
of them the day this was written.
Both sides are read from SOURCE rather than imported. The Vue file cannot be
imported at all, and reading the Python constant through an import would tie
this to module-load side effects it has no interest in.
It cannot check that the form WRITES the right key — that is behaviour, and
the keys are asserted where they are built. It catches the drift that has no
other alarm.
"""
from __future__ import annotations
import pathlib
import re
import pytest
ROOT = pathlib.Path(__file__).resolve().parents[1]
_SERVICES = ROOT / "src" / "scribe" / "services"
_VUE = ROOT / "frontend" / "src" / "views" / "SettingsView.vue"
# (services module, python constant, vue ref). Hand-written because the
# pairing is an editorial fact — the names do not share a convention either
# side could derive — but every entry is asserted to EXIST on both sides, so a
# rename fails loudly here rather than silently dropping that threshold from
# the check.
_PAIRS = (
("plugin_context.py", "AUTOINJECT_DEFAULT_THRESHOLD", "kbInjectThreshold"),
("plugin_context.py", "WRITEPATH_DEFAULT_THRESHOLD", "kbWritePathThreshold"),
("plugin_context.py", "RULEHINT_DEFAULT_THRESHOLD", "kbRuleHintThreshold"),
("plugin_context.py", "TOOLRULE_DEFAULT_THRESHOLD", "kbToolRuleThreshold"),
("plugin_context.py", "PROMPTRULE_DEFAULT_THRESHOLD", "kbPromptRuleThreshold"),
("plugin_context.py", "REPORTPREF_DEFAULT_THRESHOLD", "kbReportPrefThreshold"),
# The plan gate (milestone 415): it blocks a create, so a form showing a
# looser bar than the one in force would be the more misleading drift.
("dedup.py", "PLAN_MATCH_DEFAULT_THRESHOLD", "kbPlanMatchThreshold"),
)
def _python_default(module: str, name: str) -> float:
m = re.search(rf"^{re.escape(name)}\s*=\s*([0-9.]+)\s*$",
(_SERVICES / module).read_text(), re.M)
assert m, (
f"{name} is no longer a bare module-level float in "
f"services/{module}. If it moved or was renamed, update "
f"_PAIRS; if it was retired, drop its row — leaving it here checks "
f"nothing while looking like coverage."
)
return float(m.group(1))
def _vue_default(ref_name: str) -> float:
m = re.search(rf"const {re.escape(ref_name)} = ref\(\"([0-9.]+)\"\)",
_VUE.read_text())
assert m, (
f"{ref_name} is no longer a `ref(\"<number>\")` in SettingsView.vue. "
f"If the control was renamed, update _PAIRS; if it was removed, the "
f"setting has lost its UI and that is the thing to fix (rule 25)."
)
return float(m.group(1))
@pytest.mark.parametrize(("module", "constant", "ref_name"), _PAIRS,
ids=[p[1] for p in _PAIRS])
def test_the_form_shows_the_default_the_server_uses(module, constant, ref_name):
"""An untouched control must render the bar actually in force."""
server, form = _python_default(module, constant), _vue_default(ref_name)
assert form == server, (
f"SettingsView shows {form} for {ref_name} while the server defaults "
f"to {server} ({constant}). An operator who has never set this reads "
f"the form as the value in force, so the two must move together — "
f"retune both, or neither."
)