fix(planning): the plan gate's default drops to 0.80, where reworded plans actually score (#4079)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m38s
CI & Build / Build & push image (push) Successful in 36s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m38s
CI & Build / Build & push image (push) Successful in 36s
Measured live after deploy: three plans reworded from existing active milestones scored 0.83-0.87 against the milestone they restated, and the nearest distinct plans 0.72-0.77. At the inherited 0.90 the semantic arm matched nothing, so only an identical title was caught. 0.80 sits in the gap. The Settings form's default moves with it (test_settings_defaults_agree). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -105,7 +105,7 @@ const kbDupThresholdSnippet = ref("0.82");
|
|||||||
const kbDupThresholdNote = ref("0.93");
|
const kbDupThresholdNote = ref("0.93");
|
||||||
const kbDupThresholdTask = ref("0.93");
|
const kbDupThresholdTask = ref("0.93");
|
||||||
// PLAN_MATCH_DEFAULT_THRESHOLD in services/dedup.py.
|
// PLAN_MATCH_DEFAULT_THRESHOLD in services/dedup.py.
|
||||||
const kbPlanMatchThreshold = ref("0.90");
|
const kbPlanMatchThreshold = ref("0.80");
|
||||||
const savingKbInject = ref(false);
|
const savingKbInject = ref(false);
|
||||||
const kbInjectSaved = ref(false);
|
const kbInjectSaved = ref(false);
|
||||||
|
|
||||||
@@ -159,7 +159,7 @@ async function saveKbInject() {
|
|||||||
const dupTask = Math.min(1, Math.max(0, Number(kbDupThresholdTask.value) || 0.93));
|
const dupTask = Math.min(1, Math.max(0, Number(kbDupThresholdTask.value) || 0.93));
|
||||||
// Same `|| default` guard: a floor of 0 would hand back an existing plan
|
// Same `|| default` guard: a floor of 0 would hand back an existing plan
|
||||||
// for every new one, and no plan could be started without force.
|
// for every new one, and no plan could be started without force.
|
||||||
const planT = Math.min(1, Math.max(0, Number(kbPlanMatchThreshold.value) || 0.9));
|
const planT = Math.min(1, Math.max(0, Number(kbPlanMatchThreshold.value) || 0.8));
|
||||||
// Same `|| default` reasoning: falling back to 0 would surface every
|
// Same `|| default` reasoning: falling back to 0 would surface every
|
||||||
// snippet in the corpus on every edit, which is the failure this knob fixes.
|
// snippet in the corpus on every edit, which is the failure this knob fixes.
|
||||||
const wpT = Math.min(1, Math.max(0, Number(kbWritePathThreshold.value) || 0.68));
|
const wpT = Math.min(1, Math.max(0, Number(kbWritePathThreshold.value) || 0.68));
|
||||||
|
|||||||
@@ -739,16 +739,22 @@ async def find_duplicate_rule(
|
|||||||
# project is exactly what adding steps to its existing plan needs, so a caller
|
# project is exactly what adding steps to its existing plan needs, so a caller
|
||||||
# who reaches this gate can act on whatever it returns.
|
# who reaches this gate can act on whatever it returns.
|
||||||
#
|
#
|
||||||
# Its own threshold, as a setting (rule 25). The note gate's 0.90 is where it
|
# Its own threshold, as a setting (rule 25), and lower than the note gate's
|
||||||
# starts, because it asks the same question ("the same thing, reworded") with
|
# 0.90 because plan documents are shaped differently: a milestone is embedded
|
||||||
# the same embedder. Plan documents are shaped differently, though: a milestone
|
# as title, description and plan (embeddings.milestone_document), while the
|
||||||
# is embedded as title, description and plan (embeddings.milestone_document),
|
# candidate usually has no description and carries its steps instead, so even
|
||||||
# while the candidate usually has no description and carries its steps
|
# a faithful rewording never scores like a copy.
|
||||||
# instead. That difference has not been measured, and a gate that blocks on
|
#
|
||||||
# noise teaches sessions to pass force=true every time, which is worse than no
|
# Measured after the first deploy (2026-09-15, bge-small-en-v1.5, #4079): three
|
||||||
# gate. So the default is conservative and the operator can lower it.
|
# plans reworded from existing active milestones scored 0.83, 0.85 and 0.87
|
||||||
|
# against the milestone they restated; the nearest DIFFERENT plan for each, and
|
||||||
|
# a distinct plan on a neighbouring topic, scored 0.72-0.77. At 0.90 the
|
||||||
|
# semantic arm matched nothing, leaving only the title arm. 0.80 sits in the
|
||||||
|
# gap. A gate that blocks on noise teaches sessions to pass force=true every
|
||||||
|
# time, which is why it is not lower; four samples is thin, which is why the
|
||||||
|
# operator can move it. Retune alongside the embedder, not the corpus.
|
||||||
PLAN_MATCH_THRESHOLD_KEY = "kb_plan_match_threshold"
|
PLAN_MATCH_THRESHOLD_KEY = "kb_plan_match_threshold"
|
||||||
PLAN_MATCH_DEFAULT_THRESHOLD = 0.90
|
PLAN_MATCH_DEFAULT_THRESHOLD = 0.80
|
||||||
|
|
||||||
|
|
||||||
async def get_plan_match_threshold(user_id: int) -> float:
|
async def get_plan_match_threshold(user_id: int) -> float:
|
||||||
|
|||||||
Reference in New Issue
Block a user