refactor(notes): a snippet's and lesson's stored title is its name; the trigger joins it only in the embedded document (milestone 427)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / integration (push) Successful in 52s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 32s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / integration (push) Successful in 52s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 32s
The title was `subject — trigger` because the stored title WAS the embedded one, and the join is what makes these kinds rank on the situation they apply to (#2485). Every surface that shows a title then showed the trigger too -- menus, lists and search rows ran to kilobytes. - embeddings.document_title(title, note_type, data, body) joins the trigger from `data` (body fallback) at embed time. Idempotent: an un-migrated composed title comes out the same, never doubled. The embed path, the startup backfill and the dedup gate's semantic signal all use it, so the embedded text -- and every vector -- is unchanged. - Writers store the subject: snippet create/update (service, REST, MCP) and lesson_document. Both compose_title helpers are removed. - Readers: dedup takes `data`; the menus strip the embedded title from a passage; list rows project `when_to_use`, which SnippetListView reads. - 0108 rewrites existing rows on an exact `' — ' || <own trigger>` suffix with raw SQL, leaving updated_at alone so the backfill does not re-embed the corpus for identical vectors. Downgrade recomposes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -214,10 +214,12 @@ TRIGGER_SEP = " — "
|
||||
def trigger_title(subject: str | None, trigger: str | None) -> str:
|
||||
"""`{subject} — {trigger}` — the title half of a situation-keyed document.
|
||||
|
||||
ONE definition, because this join had three. `rule_document` built it for
|
||||
rules, `snippets.compose_title` for snippets, and milestone 385 needed a
|
||||
fourth for lessons — the shape #3207 records, where a fix or an improvement
|
||||
then has to be found in N places by someone who does not know N.
|
||||
ONE definition, because this join had three — rules, snippets, and a
|
||||
fourth for lessons (milestone 385) — the shape #3207 records, where a fix
|
||||
or an improvement then has to be found in N places by someone who does not
|
||||
know N. Since milestone 427 it builds EMBEDDED titles only: `rule_document`
|
||||
for rules and `document_title` for snippets and lessons. No stored title
|
||||
carries it.
|
||||
|
||||
WHY THE JOIN MATTERS AT ALL, measured in note #2485: the snippet was the
|
||||
only sharp record in the corpus — a 0.153 top-to-second gap against
|
||||
@@ -265,6 +267,51 @@ def untrigger_title(title: str | None, trigger: str | None) -> str:
|
||||
return title
|
||||
|
||||
|
||||
# The `data` key each trigger-keyed note kind mirrors its trigger under. Rules
|
||||
# are not here: they keep the trigger in a column and `rule_document` builds
|
||||
# their document from it.
|
||||
_TRIGGER_DATA_KEYS = {"snippet": "when_to_use", "lesson": "when_to_apply"}
|
||||
|
||||
|
||||
def document_title(
|
||||
title: str | None, note_type: str | None, data: dict | None = None,
|
||||
body: str | None = None,
|
||||
) -> str | None:
|
||||
"""The title a note is EMBEDDED under — its stored title, plus its trigger.
|
||||
|
||||
Milestone 427. A snippet's or lesson's STORED title is its subject alone;
|
||||
the trigger lives in `data` (decision #4157). It still has to reach the
|
||||
vector — the `subject — trigger` join is what makes these kinds rank on
|
||||
the situation they apply to (#2485) — so it is joined HERE, at embed time,
|
||||
rather than being carried in a title every listing then has to show.
|
||||
|
||||
IDEMPOTENT, and that is what makes the migration safe: a title that is
|
||||
already composed (a row not yet migrated, an old backup restored) is
|
||||
untriggered first, so it comes out the same and never doubled. The text is
|
||||
byte-identical to what these kinds were embedded as before, so no vector
|
||||
moves and the floors tuned against them stay calibrated.
|
||||
|
||||
`body` is the fallback when the mirror is missing, read by the kind's own
|
||||
parser — the same degrade-to-the-body each kind's reader already has.
|
||||
Every other kind, and a record with no trigger, keeps its title as-is.
|
||||
"""
|
||||
key = _TRIGGER_DATA_KEYS.get(note_type or "")
|
||||
if key is None:
|
||||
return title
|
||||
trigger = ((data or {}).get(key) or "").strip() if isinstance(data, dict) else ""
|
||||
if not trigger and body:
|
||||
from types import SimpleNamespace
|
||||
if note_type == "lesson":
|
||||
from scribe.services.lessons import lesson_trigger
|
||||
trigger = lesson_trigger(SimpleNamespace(data=None, body=body))
|
||||
else:
|
||||
from scribe.services.snippets import parse_snippet_fields
|
||||
trigger = parse_snippet_fields(title or "", body).get("when_to_use", "")
|
||||
if not trigger:
|
||||
return title
|
||||
return trigger_title(untrigger_title(title, trigger), trigger)
|
||||
|
||||
|
||||
# --- chunking (#280): the document shape ------------------------------------
|
||||
#
|
||||
# bge-small reads at most 512 tokens and fastembed silently truncates the rest,
|
||||
@@ -1081,10 +1128,14 @@ async def backfill_note_embeddings() -> None:
|
||||
)
|
||||
success = 0
|
||||
for note_id in notes_to_embed:
|
||||
row = await _current_row((Note.user_id, Note.title, Note.body), Note.id, note_id)
|
||||
row = await _current_row(
|
||||
(Note.user_id, Note.title, Note.body, Note.note_type, Note.data), Note.id, note_id,
|
||||
)
|
||||
if row is None:
|
||||
continue # deleted between the scan and here
|
||||
user_id, title, body = row
|
||||
user_id, title, body, note_type, data = row
|
||||
# The EMBEDDED title, as the write path builds it (milestone 427).
|
||||
title = document_title(title, note_type, data, body)
|
||||
if not chunk_document(title, body):
|
||||
continue
|
||||
await upsert_note_embedding(note_id, user_id, title, body)
|
||||
|
||||
Reference in New Issue
Block a user