10 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	5fb41af9b0	feat(search): the agent's search can ask for every kind the corpus has (#4250) ... The engine took `note_type` and `task_kind` all along. What was missing was a way to say them: the MCP tool's `content_type` knew `note`, `task` and `all`, and `/api/search` knew the same two — so an agent could not ask "has this snippet already been recorded" or "what lessons apply here" without searching everything and reading past the rest. Browse offered nine kinds from the same data. The cause is that each door kept its own map. `_FACETS` in services/knowledge is where a kind is declared, and #3161 made adding one a single edit by generating the SQL filter, the Python predicate and the door's validation from it — but the two search doors were written before that and never joined. So this adds the third dialect, `search_filters_for`, and one composition over it, `content_type_filters`, and both doors now derive instead of listing. Two names keep a meaning of their own, and the docstrings say so: `all` is no filter, and `note` is BROAD — any non-task, snippets and lessons included — where the browse facet of the same name is narrow (`note_type == 'note'`). They are left different deliberately; narrowing this one would stop returning snippets to every caller that already asks this way. An unrecognised kind is now refused rather than answered. Both doors used to fall through: the MCP tool into a filter matching no row, the route into no filter at all, so `?content_type=snippets` returned the whole corpus while looking like a narrowed search. An empty result set is a claim — "the corpus holds nothing like this" — and an agent acts on that claim by building the thing it could not find, so a typo must not be able to make it. The docstring is the agent-facing contract (#2846), and a test now holds it to the table: every kind `_FACETS` declares has to appear in it, because a filter an agent has not been told about is unreachable however well it is wired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy	2026-09-21 11:12:48 -04:00
bvandeusenandClaude Opus 5	d5ac8408f6	feat(telemetry): record WHAT the bar turned away, not only how close it came (#3807) ... #3670 added `best_available_score` so a threshold could be judged from its rejections. It records how CLOSE the bar came to firing and not WHAT it refused, and that is the half a decision actually needs. Live, pre_tool_rule sits at a ~0.72 bar with a near-miss p90 of 0.7071 — about 117 declines a day within 0.013 of firing. Dropping to 0.707 would take that arm from 22 hits a day to roughly 139: six-fold, on a surface that runs before every Bash call. The percentile says the mass is there. Nothing said whether it was worth showing. NEITHER OBVIOUS INSTRUMENT ANSWERS IT. Pull-through cannot: the injected rule line already carries title and trigger, so a session can comply without ever calling get_rule, and rule pull-through understates usefulness by construction. Reading the rejected records can — and `result_ids` holds only what was RETURNED, so on a zero-result call the near-missed record had no name at all. So the id, from the SAME ranked candidate as the score. Both searches unpack `best` once and read both fields off it, because splitting that into two expressions is exactly how a later edit pairs a score with its neighbour's id — and a score attached to the wrong record is worse than no id, since it invites judging the wrong one and concluding the bar is fine. write_path withholds the id on the same condition it withholds the score (#3739): a surviving id beside a null score names a record without saying what it scored, the pair disagreeing in the other direction. THE READ PATH IS A LISTING, NOT A STATISTIC — an id cannot be percentiled, and a reader tuning a bar needs to go and read the records. Opt-in via `near_miss_samples` (0-20, default 0) so the ordinary readout keeps its size, and deliberately NOT a window function: this module's one production outage was a grouped query Postgres rejected, swallowed by the broad except, every counter reading zero while the mocked tests passed (#2663). One flat ordered query, overfetched, bucketed in Python — the shape that lesson prescribes. Migration 0097, nullable and unbackfilled. Not a foreign key: the table spans record types and `source` says which, exactly as result_ids works. The integration guard pins the listing as PER SOURCE. A global LIMIT would let a noisy source eat the whole quota and leave the surface being tuned showing nothing — which reads as "nothing was close", the misreading this milestone has spent itself correcting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 21:24:32 -04:00
bvandeusenandClaude Opus 5	623464323e	fix(telemetry): a search that never ran is not a decline (#3765) ... `best_available_score` was added by #3670 so a bar could be judged from what it rejected, and it arrived null on four unrelated causes: the corpus offered nothing, the query was empty, the embedder was down, or the DATABASE QUERY FAILED. Only the first is a measurement. The fourth is the #2663 shape — a swallowed failure rendering as a clean zero — inside the field added to fix an instance of the #2663 shape. Found while trying to explain why reuse_slot returned nothing on 45 of 45 calls, and auto_inject on 153 of 161. That investigation is still open; what it established first is that the readout could not answer it. THE FIX IS NOT A NEW COLUMN. A call that never searched writes no row, so every remaining null means one thing: searched, and nothing came close. That is the convention the pre-tool arm already follows for a blank command — "a row here would report a call that never happened and drag the clear-rate down with phantom declines" — extended from the case a caller can see in advance to the ones only the search knows about. Both searches stamp `report["searched"]` FALSE before anything can return and True only where a real result set exists, so every early return leaves it false. It has to be the first thing done to the dict: a return added above that line would leave the key absent. ABSENT IS A THIRD STATE AND IT DEFAULTS TO TRUE. A caller that passes no report cannot know, and the safe reading there is the old behaviour. Only a real search can report False, so absent means "nobody asked" and never "it failed" — which is also why 66 existing mocked searches across twelve test files keep working unchanged rather than being rewritten to simulate a flag they do not care about. A FAILURE IS NOT MADE INVISIBLE. semantic_search_notes already logs a WARNING on a query failure, which is where a broken search belongs: a counter cannot say "I am broken" without a reader already trusting it. Three tests, and the middle one is what makes them discriminate — a blanket `return` passes the first and fails the second, because a call that searched and came back empty is the only evidence a threshold is too high (#3497). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-09 12:47:46 -04:00
bvandeusenandClaude Opus 5	e7c1af32a0	fix(telemetry): the bar can only be judged from what it rejected (#3670) ... `cleared_threshold` was documented as the number to read first. It was a tautology. The search applies the threshold before returning, so every returned result cleared it by construction and a call with no results has no top_score to compare — the condition was true exactly when `result_count > 0`. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen source/window readings ever taken, today's live seven included. The reading procedure built on it asked the reader to compare a number with itself, and a threshold change was unobservable through it: raise the bar and both numbers move together, so the field could never show a bar set too high. REPLACED, NOT JUST REMOVED. The question the table exists to answer is whether the bar is in the right place, and that is only answerable from the calls that returned NOTHING: how close did the best rejected candidate come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair; the same bar turning away 0.30s is working. Both render as a zero-result call today and nothing separates them, because the losing score is discarded inside the search. So both searches now rank WITHOUT the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one, and an over-fetch that returned N above-bar rows returns the same N plus some losers. What changes is that the losers are visible instead of dropped in the query. `report` carries the score out without changing what a search RETURNS: eight of eleven call sites want hits and nothing else. New column (migration 0096), nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence, which is the substitution this whole milestone corrects. The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step from the shape that produced #2663, where a rejected query was swallowed by the broad except and every counter read zero. It carries an integration guard for that reason: only real Postgres can say it parses, and the symptom of failure is silence. Also adds a guard that no int field in a bucket equals `calls - zero_result_calls`. That identity is what `cleared_threshold` satisfied for its whole life, and it survived because it had its own name and nobody added the two numbers beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ	2026-09-08 13:50:05 -04:00
bvandeusen	45ba4aab25	fix(telemetry): the two invisible retrievals log, and /api/search takes scope ... #2463, the remaining findings (finding 3 landed with f11a547). ## The reuse slot logs (source: reuse_slot) A real semantic query competing for an auto-inject menu slot, and the ledger was asymmetric: the scored hit it DISPLACED was in retrieval_logs, the query that displaced it was not — so the slot could never be evaluated against what it replaced. #1038 and #2085 are gated on this ledger being complete. ## Browse search logs (source: browse_search) The human's main search surface, and it logged nothing — so retrieval_logs claimed the web's search was /api/search. Measured while fixing: /api/search has ZERO frontend consumers; the web UI searches through /api/knowledge exclusively. The table wasn't just under-describing the UI, it was describing a surface the UI never touches. The task's check — does folding human queries into the corpus skew the precision signal thresholds are tuned against? — is answered by the source column: distinct values (browse_search, reuse_slot beside the existing four) mean tuning includes or excludes human traffic deliberately rather than by accident. Same resolution as the mcp_/rest_ split in note_usage_events. ## /api/search takes project_id (and logs it) The route logged project_id=None unconditionally while the MCP tool insists the agent pass the active project. Now optional, default global: with no frontend consumer, this route serves API callers, and an API caller states its scope explicitly — the default-scope UI decision the task flagged is moot until a UI actually consumes the route, which is recorded rather than guessed. Refs #2463	2026-08-08 20:00:22 -04:00
bvandeusen	f11a547cd2	fix(lists): shared-project records appear, and a list's q means what search means ... #2462, both decided halves. ## The ACL defect list_notes filtered on Note.user_id == user_id with no scope parameter at all — never a per-call decision, the capability was absent. query_knowledge beside it was deliberately browse-scoped with a comment saying why. So a task in a shared project was invisible in list_tasks, enter_project's open-task list, the SessionStart todo count and the web UI's task views, while the same project's notes and snippets appeared. Now the shared clause: notes_visibility_clause(user_id, "browse"). Browse and not read, per decision #2094 — an ambient list must never surface a record someone shared one-to-one; those stay search-only. This is the remaining half of a fix made twice before (#2159 widened fetch, #2092 widened meaning), and the guard below is what stops a fourth half appearing. Guarded by source inspection in test_retrieval_scopes: every list-shaped service references a shared visibility clause AND carries no bare Note.user_id comparison that would quietly re-narrow it. An owner-only list returns correct-looking rows and simply omits the shared ones — the shape no behavioural test catches. ## q is semantic (operator: "make it match") The UI's note list keyword-matched while the UI's Browse search semantic-matched, over the same records, with nothing saying so. Now one meaning: q joins the embedding index and orders by cosine distance at the interactive floor, with every lifecycle filter still applied in the same indexed query. Relevance ordering wins over `sort` when q is present — a query is a relevance claim, and sorting its results by date would shuffle the answer. ILIKE survives only as the embedder-down fallback: degraded, never empty. Stated position: superseded records are NOT demoted in list-q. The penalty reorders a top-k; reordering a paginated, counted list would make page boundaries lie. The search surfaces carry the demotion. ## The interactive floor becomes one constant INTERACTIVE_SEARCH_THRESHOLD = 0.3 in embeddings.py, consumed by routes/search.py (was a commented constant), knowledge.py (was a bare literal), and the new list-q. This closes #2463's finding 3 early — the same number lived in two files with the reasoning attached to only one. ## Deferred, deliberately The return-shape unification (ORM objects vs dicts) stays undone. It is a 14-caller refactor whose motivation — callers being unable to swap paths — shrinks now that both paths share the clause and the meaning of q. If swap pressure recurs it deserves its own change, not a rider on an ACL fix. Refs #2462, #2463	2026-08-08 19:58:01 -04:00
bvandeusen	3f1523b19f	feat(systems): read-side teeth — the vocabulary at session start, a search filter, and the state/chronicle instructions ... Step 4 of #278, product half. The audit that motivated it: one System in project 2, thirty records tagged, nothing since July 28 — three days after the feature landed. Not a discipline failure; retrieval was completely blind to the association (zero references in embeddings, knowledge, search, auto-inject, or enter_project), so tagging was a write-side label with no read-side payoff, and labels nobody reads don't get maintained. Three changes, ordered by what makes the others workable: 1. enter_project returns the project's Systems (id, name, first line of the charter). Load-bearing for the tagging instruction: you cannot ask an agent to check a record against a vocabulary it never sees. Trimmed because it rides on every session start; the full charter stays get_system's job. Present-and-empty rather than absent when a project has none — "no named areas yet" is information the create-the-System instruction acts on. 2. search accepts system_id, MCP and REST (#33). Implemented once in semantic_search_notes as an EXISTS against record_systems — an association filter deciding candidate-set membership before scoring, like project_id, not a ranking signal. The REST route's missing project filter stays #2463's: it carries a default-scope UI decision this change must not preempt. 3. The instructions (#119, _INSTRUCTIONS + using-scribe skill; plugin 0.1.25 for the cache): - Tag as you write, with an executable test — "would someone investigating that subsystem want this in the pile list_system_records returns?" — rather than "tag appropriately", which is what died. - Create the System when the area has no record: the two-or-more test snippets use, plus "don't wait to be asked to name an area that plainly exists", because the agent's default was leaving un-modelled areas un-modelled forever. - State vs chronicle: dev-logs are written once and never rewritten; durable findings live in the System's reference note, updated in place — safe because note versions are the changelog, which has existed since the feature shipped and was never named as one. list_system_records' docstring now sells it as the way to READ a subsystem, reference note first. No auto-inject boost by System — vocabulary and filter first, measure before adding ranking behaviour (the #2486 lesson). Refs #278, #2546	2026-08-08 18:19:58 -04:00
bvandeusenandClaude Opus 5	ef1dbdfc86	feat(acl): unify the retrieval scopes; mark shared rows in Knowledge browse ... Closes #2092 and the Knowledge-browse provenance gap. The two halves of a hybrid search disagreed: the keyword half honoured shares while the semantic half was pinned to NoteEmbedding.user_id, so a shared record was findable by wording and invisible by meaning — the case a semantic search exists to serve. semantic_search_notes now scopes on Note via a `scope` parameter, and each of its five callers declares which kind of act it is: mcp/tools/search.py read the agent asked routes/search.py read the user typed it knowledge.py (semantic) read matches the keyword half beside it plugin_context.py browse nobody asked; never a one-to-one share dedup.py own a verdict that blocks a write must not hinge on another person's notes That last one is the reason this isn't a single global widening: the dedup gate returns "update the existing one instead", so matching a stranger's record would refuse a legitimate create and point at something the caller can't edit. Scope defaults to "own" so a caller that forgets is wrong in the safe direction, and an unknown scope raises rather than falling back — a typo there would be a data-exposure bug. Auto-inject keeps the browse scope, which still admits a collaborator's note via a shared project. Its menu line is the only provenance an agent sees, so a foreign hit now reads: #12 "Title" (0.71) - shared by alex, treat as a suggestion. MCP and REST search results carry shared/owner too. Knowledge browse: the feed hydrates cards from /api/knowledge/batch rather than the list route, so both paths label rows now, and KnowledgeView shows "by <owner>" on records the viewer doesn't own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt	2026-07-25 23:06:52 -04:00
bvandeusenandClaude Opus 4.8	807f478cac	feat(search): retrieval telemetry — log every semantic retrieval ... Add retrieval_logs (migration 0068) + services/retrieval_telemetry with a fire-and-forget record_retrieval(), wired into the MCP search tool (source=mcp_search) and the REST search route (source=rest_search). Captures query, effective params, and the per-result score distribution so KB-injection thresholds can be tuned from data rather than guessed. Scribe: project 2, milestone 93, task 1032. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xz4j1H7pjYSjKsEpgcNH5E	2026-06-22 20:10:15 -04:00
bvandeusenandClaude Opus 4.8	b255a0f90e	refactor: rename package fabledassistant -> scribe (code-only) ... Renames src/fabledassistant -> src/scribe and all imports, plus the default DB name and DB user/password (fabled -> scribe) in config + compose. 952 refs / 154 files. Reverses the old 'internal name stays fabledassistant' convention. Code-only: live databases are still physically named 'fabledassistant'. Deployed environments must set POSTGRES_DB / POSTGRES_USER (or rename the DB) since the defaults now resolve to 'scribe'. Repo (FabledScribe), git host (fabledsword), MCP (fabled-git) and the image name (fabledscribe) are intentionally unchanged. ruff check src/ clean locally; CI (typecheck + pytest) is the gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 15:48:35 -04:00