Commit Graph
18 Commits
Author SHA1 Message Date
bvandeusen 9d8104f7a5 fix(embeddings): key_share alone is FOR NO KEY UPDATE, not FOR KEY SHARE (#3262)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 10s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / integration (push) Successful in 24s
CI & Build / Build & push image (push) Successful in 24s
SQLAlchemy spells Postgres's four row locks as a read/key_share pair, so
`with_for_update(key_share=True)` renders FOR NO KEY UPDATE — an
exclusive lock that two refreshes of the same record would fight over,
and that an ordinary concurrent edit would block. The claim needs
`read=True` as well to be the FOR KEY SHARE the docstring describes.

Caught by the unit test that compiles the statement, which is the whole
reason it asserts on the rendered lock mode rather than on behaviour
that looks identical either way.
2026-08-31 08:13:29 -04:00
bvandeusen 7827b4ce63 fix(embeddings): the index refresh loses the race it used to deadlock (#3262)
CI & Build / integration (push) Successful in 38s
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 37s
CI & Build / Python tests (push) Failing after 55s
CI & Build / Build & push image (push) Skipped
An embedding refresh replaces a record's vectors as delete-then-insert,
which takes the chunk rows first and the parent row second (via the
insert's foreign key). A cascading delete of the parent takes exactly
those two locks in the other order. Postgres calls the cycle a deadlock
and kills one side: sometimes the detached embedder, silently, and
sometimes the user's delete, as a 500 on an operation that should have
worked.

Both upserts now claim the parent row with FOR KEY SHARE NOWAIT before
touching any chunk row. That removes the cycle instead of narrowing it —
either the embedder is first and the delete queues behind it, or the
delete already holds the row and the embedder loses at once, which is
the side designed to lose. FOR KEY SHARE is the lock the insert would
take anyway, so an ordinary edit is unaffected.

The note twin, recorded as unverified on the issue, has the same shape
and the same fix; a trash purge is the hard delete that reaches it.

Unit tests pin the ORDER and the lock mode by compiling the statement;
the integration pair holds a real delete open in one transaction and
proves the embedder returns having written nothing, with a deadline so
a regression fails instead of hanging.
2026-08-31 08:09:09 -04:00
bvandeusenandClaude Opus 5 02c1e37620 feat(rules): the write path can notice a standing rule it was never given (#3031, milestone 307 step 5, hook arm)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m7s
CI & Build / Build & push image (push) Successful in 24s
A conditional rule is not resident, so a session can be about to violate one it
was never handed. This arm notices: when what is being written resembles a
rule's trigger, the hint names it and says to read it before deciding it does
not apply.

A SUGGESTION, and the plan was wrong about why it could be more. It claimed the
hook "already resolves a path to an area" — it does not, and nothing in Scribe
maps a path to a System or a canonical area (build_write_path_hint resolves
paths against snippet LOCATIONS, a different index; the learned-alias idea
belongs to another project). Correction logged on the task. Rather than invent
path→area inference to make a stale claim true, the arm does what D7 already
decided and what this surface already IS: tags bind at enter_project, meaning
suggests here. The header of the hook says NEVER BLOCKS; dressing a hint up as
binding would have been the actual mistake.

CONDITIONAL RULES ONLY. An always-on rule is already in the session, so
re-offering it is noise — and noise on a hint that fires on every write is how
a hint gets ignored.

Telemetry goes to retrieval_logs, NOT note_usage_events, and that is a
correctness call rather than a preference: note_usage ids are REMAPPED on a
backup restore, so a rule id written there would come back attached to whatever
note took that number — silently corrupting the evidence the next true-up is
supposed to read. retrieval_logs is never restored and `source` already
separates surfaces. record_retrieval's `results` type widened to match what it
actually needs (an `.id`), instead of passing a Rule to something annotated Note.

The rule dedup gets its OWN state file and query parameter, like the three
channels before it — #2708's lesson was that one shared channel lets a hint of
one class silence a different class that had never been shown. Plugin version
bumped: a hook change clients cannot see did not ship (#1040).

The stub is autouse in conftest rather than added to forty-odd call sites: the
arm loads an embedding model, and every existing test that stubs the NOTES
search would otherwise pull a real model in through the one arm it had no way
to know about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 16:35:36 -04:00
bvandeusenandClaude Opus 5 6ada97bb0b fix(rules): resolve the Rule forward ref for ruff and readers (#3030)
CI & Build / Python lint (push) Successful in 6s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / integration (push) Successful in 33s
CI & Build / TypeScript typecheck (push) Successful in 44s
CI & Build / Python tests (push) Successful in 1m11s
CI & Build / Build & push image (push) Successful in 23s
F821: the quoted `"Rule"` in semantic_search_rules' return annotation
evaluates fine at runtime — a string inside a subscript is a value, not a name
lookup — but it points at nothing a reader or a type checker can follow, which
is what ruff is objecting to and it is right to.

A TYPE_CHECKING import resolves it at zero runtime cost, while the real import
stays inside the function so this module still does not pull in the rulebook
models. Placed after the import block, where isort wants the guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 15:03:40 -04:00
bvandeusenandClaude Opus 5 95a37318fc feat(rules): rules become findable by meaning (#3030, milestone 307 step 4)
CI & Build / Python lint (push) Failing after 9s
CI & Build / Plugin hooks (push) Successful in 12s
CI & Build / TypeScript typecheck (push) Successful in 44s
CI & Build / integration (push) Successful in 45s
CI & Build / Python tests (push) Successful in 1m26s
CI & Build / Build & push image (push) Skipped
Rules were the only major record type with no vector, so `search` could never
return one and a rule could arrive only by being preloaded. That single fact is
what made every rule compete for one always-on budget.

THE DECISION THE TASK ASKED FOR, made explicitly: a sibling rule_embeddings
table, not a polymorphic embedding row. The ROW could have been generalised;
the SEARCH could not. semantic_search_notes is Note-specific scoping end to end
— the visibility clause, the supersession penalty, note_type/task_kind/system
filters — and a rule shares none of it, scoping instead by rulebook ownership
or project. Generalising the row while still needing two searches is the worst
of both: a key with referential integrity to neither table, on the path every
session start runs, to share four columns. What is genuinely common is
BEHAVIOUR — get_embedding, chunk_document, embedding_text, CHUNKER_VERSION —
and those are reused as-is. Sharing them is the DRY win; sharing the table
would have been the DRY costume.

The document shape is measured, not chosen (note 2485). That pass found the
snippet was the only discriminative record in the corpus — a 0.153
top-to-second gap against 0.010-0.023 — and that the cause was its SHAPE:
purpose stated twice in a short single-topic document. rule_document
reproduces it: the trigger in the title AND as the body's first line.

And it excludes `why`, which matters more than any of it. `why` is dated
incident narrative — rule 46's runs to 4,300 characters — and long multi-topic
prose is exactly what made sixteen dev-logs mutually indistinguishable. Adding
it would not give the vector more to work with; it would give every rule the
SAME thing to work with. rule_document takes no `why` parameter at all, so a
well-meaning caller cannot pass one.

A rule with no trigger degrades to title + statement — findable, less sharp.
That is an argument for backfilling triggers (step 6), not for padding the
document with whatever text is nearby.

search(content_type="rule") returns the rule WITH its why and how_to_apply:
they are its operational half, the session payload never carries them, and a
caller who went looking should not have to re-fetch. Writes re-index
fire-and-forget like notes; startup backfills in its own try block so neither
backfill can skip the other. rule_embeddings is derived, so it joins
note_embeddings in the backup's explicitly-NOT-included list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 15:00:49 -04:00
bvandeusenandClaude Fable 5 041d8defbc feat(embeddings): best-chunk-per-note on every retrieval surface (#280 step 4)
CI & Build / Plugin hooks (push) Failing after 1s
CI & Build / Python lint (push) Failing after 3s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Build & push image (push) Skipped
A note's relevance is now its best chunk's similarity, everywhere:

- semantic_search_notes keeps the indexed raw-distance top-k and over-fetches
  chunk rows (x4, composing with the x3 supersession over-fetch), then
  collapses to first-appearance-per-note — rows arrive distance-ordered, so
  first is best. Every ranked consumer (MCP/REST search, Browse, auto-inject,
  write-path, gate) inherits through the one function.
- list_notes semantic q swaps its join for a correlated MIN-distance
  subquery — the join would have repeated a long note once per matching chunk
  and made total count chunks.
- the duplicate report groups its self-join by note pair on MIN(distance):
  pair similarity = closest chunk pair, and the < join now also drops
  cross-chunk self-pairs that would flag every long note against itself.
- the write gate queries once per chunk of the candidate (capped at 8), so a
  note duplicating an existing record in ONE SECTION is caught — the
  whole-document query diluted exactly the section that mattered.

Integration test now seeds a two-chunk note and pins the collapse against
real pgvector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
2026-08-08 23:51:01 -04:00
bvandeusenandClaude Fable 5 0e70a3896b feat(embeddings): per-chunk rows — schema, write path, version-aware backfill (#280 steps 2+3)
note_embeddings becomes one row per chunk: PK (note_id, chunk_index), plus
chunk_text (what this vector actually encodes) and chunker_version. Migration
0077 clears the table — embeddings are derived (0067 precedent) and the old
whole-document rows are indistinguishable from single-chunk notes, so the
startup backfill regenerates the corpus at the new shape. The backfill is now
version-aware: a future shape change is a CHUNKER_VERSION bump that re-embeds
exactly the stale notes, not another wipe.

upsert_note_embedding takes (title, body) and chunks internally — one path
for the write path, the recurrence spawn and the backfill. The recurrence
spawn's own embed call is deleted outright: create_note already embeds via
embed_note (#2056), so the spawn was a second copy of the rule. An emptied
record now CLEARS its stale vectors instead of leaving them findable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
2026-08-08 23:47:34 -04:00
bvandeusenandClaude Fable 5 6b5043a69c feat(embeddings): chunk_document — the chunked document shape (#280 step 1)
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / Python tests (push) Successful in 46s
CI & Build / Python lint (push) Successful in 4s
CI & Build / integration (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Build & push image (push) Successful in 25s
bge-small reads 512 tokens and fastembed truncates silently, so a single
whole-document vector permanently lost everything past ~400 words. The new
shape: split at markdown headings (fence-aware), merge small sections, split
oversize ones at paragraph boundaries, title-anchor every chunk, repeat the
section heading on continuation pieces. A record that fits the window yields
exactly one chunk identical to the historical title\nbody shape, so the
corpus's sharpest records are byte-for-byte unaffected. CHUNKER_VERSION added
so later shape changes re-embed by version comparison instead of a table wipe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
2026-08-08 23:43:51 -04:00
bvandeusen f11a547cd2 fix(lists): shared-project records appear, and a list's q means what search means
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / TypeScript typecheck (push) Successful in 12s
CI & Build / integration (push) Successful in 18s
CI & Build / Python tests (push) Successful in 48s
CI & Build / Build & push image (push) Successful in 28s
#2462, both decided halves.

## The ACL defect

list_notes filtered on Note.user_id == user_id with no scope parameter at all
— never a per-call decision, the capability was absent. query_knowledge beside
it was deliberately browse-scoped with a comment saying why. So a task in a
shared project was invisible in list_tasks, enter_project's open-task list,
the SessionStart todo count and the web UI's task views, while the same
project's notes and snippets appeared.

Now the shared clause: notes_visibility_clause(user_id, "browse"). Browse and
not read, per decision #2094 — an ambient list must never surface a record
someone shared one-to-one; those stay search-only. This is the remaining half
of a fix made twice before (#2159 widened fetch, #2092 widened meaning), and
the guard below is what stops a fourth half appearing.

Guarded by source inspection in test_retrieval_scopes: every list-shaped
service references a shared visibility clause AND carries no bare
Note.user_id comparison that would quietly re-narrow it. An owner-only list
returns correct-looking rows and simply omits the shared ones — the shape no
behavioural test catches.

## q is semantic (operator: "make it match")

The UI's note list keyword-matched while the UI's Browse search
semantic-matched, over the same records, with nothing saying so. Now one
meaning: q joins the embedding index and orders by cosine distance at the
interactive floor, with every lifecycle filter still applied in the same
indexed query. Relevance ordering wins over `sort` when q is present — a query
is a relevance claim, and sorting its results by date would shuffle the
answer. ILIKE survives only as the embedder-down fallback: degraded, never
empty.

Stated position: superseded records are NOT demoted in list-q. The penalty
reorders a top-k; reordering a paginated, counted list would make page
boundaries lie. The search surfaces carry the demotion.

## The interactive floor becomes one constant

INTERACTIVE_SEARCH_THRESHOLD = 0.3 in embeddings.py, consumed by
routes/search.py (was a commented constant), knowledge.py (was a bare
literal), and the new list-q. This closes #2463's finding 3 early — the same
number lived in two files with the reasoning attached to only one.

## Deferred, deliberately

The return-shape unification (ORM objects vs dicts) stays undone. It is a
14-caller refactor whose motivation — callers being unable to swap paths —
shrinks now that both paths share the clause and the meaning of q. If swap
pressure recurs it deserves its own change, not a rider on an ACL fix.

Refs #2462, #2463
2026-08-08 19:58:01 -04:00
bvandeusen 3f1523b19f feat(systems): read-side teeth — the vocabulary at session start, a search filter, and the state/chronicle instructions
CI & Build / Python tests (push) Successful in 46s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 10s
CI & Build / integration (push) Successful in 18s
CI & Build / Build & push image (push) Successful in 25s
Step 4 of #278, product half. The audit that motivated it: one System in
project 2, thirty records tagged, nothing since July 28 — three days after the
feature landed. Not a discipline failure; retrieval was completely blind to
the association (zero references in embeddings, knowledge, search, auto-inject,
or enter_project), so tagging was a write-side label with no read-side payoff,
and labels nobody reads don't get maintained.

Three changes, ordered by what makes the others workable:

1. enter_project returns the project's Systems (id, name, first line of the
   charter). Load-bearing for the tagging instruction: you cannot ask an agent
   to check a record against a vocabulary it never sees. Trimmed because it
   rides on every session start; the full charter stays get_system's job.
   Present-and-empty rather than absent when a project has none — "no named
   areas yet" is information the create-the-System instruction acts on.

2. search accepts system_id, MCP and REST (#33). Implemented once in
   semantic_search_notes as an EXISTS against record_systems — an association
   filter deciding candidate-set membership before scoring, like project_id,
   not a ranking signal. The REST route's missing project filter stays #2463's:
   it carries a default-scope UI decision this change must not preempt.

3. The instructions (#119, _INSTRUCTIONS + using-scribe skill; plugin 0.1.25
   for the cache):
   - Tag as you write, with an executable test — "would someone investigating
     that subsystem want this in the pile list_system_records returns?" —
     rather than "tag appropriately", which is what died.
   - Create the System when the area has no record: the two-or-more test
     snippets use, plus "don't wait to be asked to name an area that plainly
     exists", because the agent's default was leaving un-modelled areas
     un-modelled forever.
   - State vs chronicle: dev-logs are written once and never rewritten; durable
     findings live in the System's reference note, updated in place — safe
     because note versions are the changelog, which has existed since the
     feature shipped and was never named as one.

list_system_records' docstring now sells it as the way to READ a subsystem,
reference note first. No auto-inject boost by System — vocabulary and filter
first, measure before adding ranking behaviour (the #2486 lesson).

Refs #278, #2546
2026-08-08 18:19:58 -04:00
bvandeusen f20c019f2a feat(supersession): demote what a later note overtook, and label it
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / TypeScript typecheck (push) Successful in 10s
CI & Build / integration (push) Successful in 18s
CI & Build / Python tests (push) Failing after 30s
CI & Build / Build & push image (push) Skipped
Step 3 of #278. First step with visible effect.

## Where the demotion happens, and why not in SQL

Applied AFTER the pgvector fetch, over an over-fetched candidate set, not as
part of the ORDER BY.

Ordering by `distance + penalty` would be exact and would turn an indexed
top-k into a scan-and-sort of every embedded note — the HNSW index from
migration 0067 can only serve a raw-distance ordering. So the query fetches
3x the requested rows by raw distance and the re-rank happens in Python.

Demoting after a LIMIT k with no over-fetch would have been theatre: the cut
already happened, so a superseded record pushed down still sits in the results
and the live record that should have replaced it was never fetched.

The cost is stated in the code: a live record outside the over-fetch window
cannot be promoted in. With a 0.05 penalty against neighbours ~0.014 apart,
that needs the true answer more than three ranks down, which no observed query
approaches.

## Demote, never hide — enforced in three places

The penalty applies to the RANKING score, not to the relevance threshold. The
floor decides whether a record is relevant at all; the penalty decides which
relevant record comes first. Applying it to the floor would drop a superseded
record out of the results entirely, which is the one thing this must not do.

It is small on purpose. Supersession is a claim about SOME of a record's
content, so one that strongly answers a question nothing else answers still
surfaces — just behind anything comparable that is current. Its test asserts
both bounds, the upper one citing the operator's constraint rather than an
optimisation.

And every test here that could be satisfied by dropping a record instead
asserts the record is still present.

## The dedup gate opts out

A superseded record is still a duplicate of what you are about to write — the
claim is that it is no longer current, not that it is gone. Demoting it there
would let the same note be recorded a second time, and the second copy would be
the one nothing warns about.

## The label

Auto-inject marks a superseded line SUPERSEDED with a pointer to check the
later record. One query for the whole menu. An agent handed stale material with
nothing marking it acts on it with full confidence, which is worse than never
having surfaced it — the ranking is only half the fix.

Fails open: a supersession lookup error returns unpenalised results rather than
none, because unpenalised ranking is the behaviour that shipped for months and
a broken search is not.

Refs #278
2026-08-08 02:00:06 -04:00
bvandeusen bbba0b3ae3 refactor(embeddings): one definition of the document a record is embedded as
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 34s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 26s
`f"{title}\n{body}"` was written out four times. #2486 found three — the write
path, the recurring-task spawn, the startup backfill. The guard added here
found the fourth immediately, and it was the one that mattered most.

`dedup.find_duplicate_note` built the same string as a QUERY, compared against
embedded documents. Shaped differently from the corpus it searches, the gate
degrades silently: it still returns neighbours, just less apt ones, and nothing
says the query and the index stopped agreeing. The spawn path has the same
shape of risk — a recurring task embedded differently from everything else is
ranked against documents it doesn't match.

None of the four had diverged. That is what makes this worth doing now rather
than after: they are identical today, so collapsing them is a no-op, and the
whole point is that the next change to the shape can't hit three of four.

Which is imminent. #2486 measured a dev-log separating from five unrelated
dev-logs by 0.023 where a snippet separates by 0.153 — the difference being
that a snippet states its purpose twice in a short document. Whether that shape
is right is the open question; testing an alternative against four copies would
mean testing a shape that isn't the one in production. This is the precondition
the issue named.

The guard is source inspection, matching the f-string pattern rather than a
variable name, so a copy that renames its locals is still caught. A behavioural
test cannot see this: an inlined copy produces the same string today and
diverges the day the shape changes.

Refs #2486
2026-08-07 13:01:45 -04:00
bvandeusenandClaude Opus 5 f8522fb28f fix(retrieval): give reuse a slot, and let experience reach the write path
CI & Build / TypeScript typecheck (push) Failing after 2s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 22s
CI & Build / Python tests (push) Failing after 35s
CI & Build / Build & push image (push) Skipped
Two mirror-image scoping mistakes, neither deliberate (#2246).

AUTO-INJECT let every kind compete on raw cosine. That is fatal rather than
merely imperfect here, because Scribe's project records are ABOUT software
work: a task titled "surface snippets before the agent writes code" is a
near-perfect lexical match for "write a function…" while answering none of it.
Measured live, a prompt asking for a helper returned three records about
BUILDING the retrieval system and zero snippets. Snippets are ~0.5% of the
corpus, and the ratio worsens as the project record grows — which is the
direction Scribe is meant to grow, so no threshold tuning fixes it.

Now the best snippet or process takes the LAST slot when none won on score.
Deliberately NOT held to the margin band: that band measures distance from the
top overall score, and the top score is the very thing snippets lose to. It
still must clear the configured threshold, so a weak snippet cannot buy the
slot — silence stays the default. Skipped entirely when reuse already won,
so the fix is invisible in the case it isn't needed.

WRITE-PATH was snippets-only — the same mistake inverted. An issue recording
"we tried this and it deadlocked" could never reach the moment that code was
about to be written, though it is arguably the better prior art: it says what
NOT to do. Widened to snippets plus recorded experience.

That needed a filter the search layer couldn't express. "Experience" is issues
plus dev-logs, which differ on is_task, so neither note_type nor is_task alone
covers it. semantic_search_notes gains task_kind, which restricts TASKS to the
given kinds while leaving non-task notes untouched — so note_type=("snippet",
"note") + task_kind="issue" yields snippets, fixed problems and durable notes,
without the open to-do list. note_type now accepts a sequence too.

Non-snippet hits are labelled with their kind, because an unlabelled issue on
that menu reads as "here is code to reuse", the opposite of what it says.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
2026-07-31 22:56:31 -04:00
bvandeusen e0328f2b1c feat(plugin): write-path trigger — offer prior art before code is rewritten
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python lint (push) Successful in 4s
CI & Build / integration (push) Successful in 27s
CI & Build / Python tests (push) Successful in 47s
CI & Build / Build & push image (push) Successful in 1m2s
The milestone headline. Auto-inject fires on the operator's prompt; the
moment reuse is actually lost is later, when the agent decides mid-task to
write a helper. A PreToolUse hook on Write|Edit now fires there.

Channel: `additionalContext` with NO permissionDecision, so the note reaches
Claude beside the tool result and the write is never blocked — a recall aid
must not be able to stop the operator's work. Plain stdout would have been
invisible to the model, and deny/ask would have made a nudge into a gate.

Two arms, different in kind:
- BY PLACE — a snippet recorded at this path (or its directory) is prior art
  by definition, not resemblance, so it is neither scored nor thresholded.
  This is what #2083's reverse lookup was built to answer.
- BY MEANING — semantic search restricted to snippets (new `note_type` filter
  on semantic_search_notes) over the code about to be written.
Place ranks first; the top-k cap spans both arms.

Gates carried over from milestone 93 verbatim: threshold, margin, session
dedup, titles-never-bodies. Own `source='write_path'` in retrieval_logs so
precision is tunable separately — the docstring records that the place arm
is unlogged and hands that to #2085.

Its own on/off in Settings but the SAME threshold/top-k: one "how loud may
Scribe be" knob is easier to reason about than two that drift, and splitting
them later is then a data-backed change rather than a guess.

Details worth keeping: the hook sends a REPO-RELATIVE path because that is
how locations are recorded; the git remote resolves to a project and is never
used as the location `repo` filter (different namespaces, would silently
match nothing); the endpoint stays a GET because a read-scoped API key cannot
POST and every other hook depends on that.

plugin.json 0.1.17 -> 0.1.18. Refs #2082, milestone #232.
2026-07-28 09:08:17 -04:00
bvandeusenandClaude Opus 5 ef1dbdfc86 feat(acl): unify the retrieval scopes; mark shared rows in Knowledge browse
CI & Build / Build & push image (push) Has been skipped
CI & Build / Python lint (push) Successful in 3s
CI & Build / Python tests (push) Failing after 31s
CI & Build / TypeScript typecheck (push) Successful in 34s
CI & Build / integration (push) Successful in 34s
Closes #2092 and the Knowledge-browse provenance gap.

The two halves of a hybrid search disagreed: the keyword half honoured shares
while the semantic half was pinned to NoteEmbedding.user_id, so a shared record
was findable by wording and invisible by meaning — the case a semantic search
exists to serve. semantic_search_notes now scopes on Note via a `scope`
parameter, and each of its five callers declares which kind of act it is:

  mcp/tools/search.py     read    the agent asked
  routes/search.py        read    the user typed it
  knowledge.py (semantic) read    matches the keyword half beside it
  plugin_context.py       browse  nobody asked; never a one-to-one share
  dedup.py                own     a verdict that blocks a write must not hinge
                                  on another person's notes

That last one is the reason this isn't a single global widening: the dedup gate
returns "update the existing one instead", so matching a stranger's record would
refuse a legitimate create and point at something the caller can't edit. Scope
defaults to "own" so a caller that forgets is wrong in the safe direction, and an
unknown scope raises rather than falling back — a typo there would be a
data-exposure bug.

Auto-inject keeps the browse scope, which still admits a collaborator's note via
a shared project. Its menu line is the only provenance an agent sees, so a
foreign hit now reads: #12 "Title" (0.71) - shared by alex, treat as a
suggestion. MCP and REST search results carry shared/owner too.

Knowledge browse: the feed hydrates cards from /api/knowledge/batch rather than
the list route, so both paths label rows now, and KnowledgeView shows "by
<owner>" on records the viewer doesn't own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLwAaV4DQEmVyn496HnEvt
2026-07-25 23:06:52 -04:00
bvandeusenandClaude Opus 4.8 513019786e feat(search): pgvector substrate — vector(384) + HNSW for semantic search
Move semantic_search_notes off the full-table Python cosine scan onto a native
pgvector column: indexed ORDER BY embedding <=> :q LIMIT k (HNSW, cosine).
Migration 0067 enables the extension, converts the JSONB embedding column to
vector(384) (stale-dim rows dropped and regenerated by the startup backfill),
and builds the HNSW cosine index. Postgres image moves postgres:16-alpine ->
pgvector/pgvector:pg17 across prod, quickstart, and CI.

Scribe: project 2, milestone 93, task 1031.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xz4j1H7pjYSjKsEpgcNH5E
2026-06-22 20:10:15 -04:00
bvandeusenandClaude Opus 4.8 70ab3f38c6 chore: remove pre-pivot dead code + finish Scribe rebrand (#599 t1-3)
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 51s
CI & Build / Build & push image (push) Successful in 1m1s
- Header wordmark Fabled -> Scribe; fable:calendar-changed event ->
  scribe:calendar-changed; SettingsView CSS comment.
- Drop dead Project.auto_summary + summary_updated_at columns (migration
  0063) -- the Ollama-era summarizer is gone; model + 2 frontend types +
  projects test updated.
- Remove pivot vestiges: diagnostics _curator_busy()/curator_busy
  heartbeat field, tz BRIEFING_DAY_START_HOUR/user_briefing_date dead
  aliases, the ignored 'model' param on get_embedding (+ its test).

ruff src/ clean; CI is the gate. Part of scribe plan #599.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 16:16:44 -04:00
bvandeusenandClaude Opus 4.8 b255a0f90e refactor: rename package fabledassistant -> scribe (code-only)
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 54s
CI & Build / Build & push image (push) Successful in 1m14s
Renames src/fabledassistant -> src/scribe and all imports, plus the
default DB name and DB user/password (fabled -> scribe) in config +
compose. 952 refs / 154 files. Reverses the old 'internal name stays
fabledassistant' convention.

Code-only: live databases are still physically named 'fabledassistant'.
Deployed environments must set POSTGRES_DB / POSTGRES_USER (or rename the
DB) since the defaults now resolve to 'scribe'. Repo (FabledScribe), git
host (fabledsword), MCP (fabled-git) and the image name (fabledscribe)
are intentionally unchanged.

ruff check src/ clean locally; CI (typecheck + pytest) is the gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:48:35 -04:00