refactor(notes): a snippet's and lesson's stored title is its name; the trigger joins it only in the embedded document (milestone 427)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / integration (push) Successful in 52s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 32s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / integration (push) Successful in 52s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / Python tests (push) Successful in 1m35s
CI & Build / Build & push image (push) Successful in 32s
The title was `subject — trigger` because the stored title WAS the embedded one, and the join is what makes these kinds rank on the situation they apply to (#2485). Every surface that shows a title then showed the trigger too -- menus, lists and search rows ran to kilobytes. - embeddings.document_title(title, note_type, data, body) joins the trigger from `data` (body fallback) at embed time. Idempotent: an un-migrated composed title comes out the same, never doubled. The embed path, the startup backfill and the dedup gate's semantic signal all use it, so the embedded text -- and every vector -- is unchanged. - Writers store the subject: snippet create/update (service, REST, MCP) and lesson_document. Both compose_title helpers are removed. - Readers: dedup takes `data`; the menus strip the embedded title from a passage; list rows project `when_to_use`, which SnippetListView reads. - 0108 rewrites existing rows on an exact `' — ' || <own trigger>` suffix with raw SQL, leaving updated_at alone so the backfill does not re-embed the corpus for identical vectors. Downgrade recomposes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -26,20 +26,23 @@ while staying a note in every other respect.
|
||||
WHERE THE TRIGGER LIVES (decision #4157, milestone 385 step 1)
|
||||
|
||||
In ``notes.data`` under ``when_to_apply``, written through a named parameter and
|
||||
mirrored into the title and the head of the body — the shape snippets already
|
||||
use for ``when_to_use``. Not a column on ``notes``.
|
||||
mirrored into the head of the body — the shape snippets already use for
|
||||
``when_to_use``. Not a column on ``notes``. Since milestone 427 it is NOT in the
|
||||
stored title: the title is the lesson's subject, and the trigger joins it only
|
||||
in the embedded document (``embeddings.document_title``).
|
||||
|
||||
That decision was measured rather than assumed. The whole snippet corpus —
|
||||
164 of 164 — carries a ``when_to_use`` with **no guard anywhere**, which refutes
|
||||
the premise that an unenforced field gets skipped. What it does NOT show is that
|
||||
an agent types a title convention correctly: ``compose_title`` builds the title
|
||||
an agent types a title convention correctly: the service composed the title
|
||||
from the parameter, so what is at 100% is a named structured field. A column
|
||||
would have bought enforceability at the price of deciding, for every note kind
|
||||
at once, a question nothing had measured.
|
||||
|
||||
The mirror is what makes the vector sharp, and it is why nothing re-embeds:
|
||||
``chunk_document`` is untouched, so ``CHUNKER_VERSION`` does not move. The
|
||||
trigger reaches the document by being in the text, exactly as a snippet's is.
|
||||
The trigger in the document is what makes the vector sharp. It reaches it
|
||||
twice — in the embedded title, joined at embed time, and in the body's first
|
||||
line — which is the text these kinds were always embedded as, so nothing
|
||||
re-embeds and ``CHUNKER_VERSION`` does not move.
|
||||
|
||||
WHAT A LESSON INHERITS, AND THE CELLS LEFT EMPTY ON PURPOSE (#3163)
|
||||
|
||||
@@ -207,36 +210,17 @@ def sole_source(sources: list[int] | None) -> int | None:
|
||||
return ids[0] if len(ids) == 1 else None
|
||||
|
||||
|
||||
def compose_title(what: str, when_to_apply: str = "") -> str:
|
||||
"""`{what} — {when it applies}`, the half of the document that ranks.
|
||||
|
||||
Built HERE rather than asked of the caller, and that distinction is the
|
||||
whole evidence base for this design: the snippet corpus is at 100% on its
|
||||
trigger because a service composes the title from a named parameter, not
|
||||
because agents type separators reliably. A caller made to spell the
|
||||
convention is the option milestone 385 step 1 rejected.
|
||||
|
||||
The join is `embeddings.trigger_title` — shared with rules and snippets, so
|
||||
the three kinds that rank on a trigger cannot drift apart in how they say
|
||||
so.
|
||||
"""
|
||||
from scribe.services.embeddings import trigger_title
|
||||
|
||||
return trigger_title(what, when_to_apply)
|
||||
|
||||
|
||||
def compose_body(
|
||||
insight: str, when_to_apply: str = "", learned_from: list[int] | None = None,
|
||||
) -> str:
|
||||
"""The lesson body — the trigger line first, the insight after.
|
||||
|
||||
The mirror of `compose_title` on the other half of the document, and the
|
||||
reason the pair is what makes a lesson findable: `chunk_document` joins
|
||||
them as `{title}\\n{body}`, so a lesson composed here states WHEN IT
|
||||
APPLIES in the title and again in the first line of the body. That is the
|
||||
twice-in-a-short-document shape note #2485 measured as the only sharp one
|
||||
in the corpus, reached the way a snippet reaches it — by being in the text
|
||||
— rather than by a second document builder at embed time.
|
||||
The trigger's home in the text of the document, and half of what makes a
|
||||
lesson findable: `chunk_document` joins `{embedded title}\\n{body}`, and
|
||||
the embedded title is `what — when it applies` (`embeddings.document_title`,
|
||||
milestone 427), so the document states WHEN IT APPLIES in the title and
|
||||
again in the first line of the body. That is the twice-in-a-short-document
|
||||
shape note #2485 measured as the only sharp one in the corpus.
|
||||
|
||||
`**When to apply:**` rather than plain text: the body is the READABLE
|
||||
form, `data` is the queryable mirror, and `_BODY_TRIGGER_RE` reads this
|
||||
@@ -275,23 +259,23 @@ def lesson_document(
|
||||
what: str, when_to_apply: str = "", insight: str = "",
|
||||
learned_from: list[int] | None = None,
|
||||
) -> tuple[str, str]:
|
||||
"""The (title, body) a lesson is STORED — and therefore embedded — as.
|
||||
"""The (title, body) a lesson is STORED as.
|
||||
|
||||
One call so the two halves cannot be composed apart. A lesson whose title
|
||||
carried the trigger and whose body did not would embed as an ordinary
|
||||
note wearing a label, and nothing would report it: the record would look
|
||||
right in every listing and simply never be retrieved at the moment it
|
||||
applies.
|
||||
One call so the two halves cannot be composed apart. The body's first line
|
||||
carries the trigger; a lesson without it (and without the `data` mirror)
|
||||
would embed as an ordinary note wearing a label, and nothing would report
|
||||
it: the record would look right in every listing and simply never be
|
||||
retrieved at the moment it applies.
|
||||
|
||||
Deliberately returns what is STORED, not a separate embed-time shape.
|
||||
Rules need `rule_document` because a rule keeps its trigger in a column
|
||||
and its title is a plain name, so the sharp document has to be synthesised
|
||||
for the ranker and exists nowhere else. A lesson follows the snippet
|
||||
instead — the stored record IS the sharp document — which is why nothing
|
||||
re-embeds and `CHUNKER_VERSION` does not move.
|
||||
The title is the SUBJECT alone (milestone 427). It used to carry the
|
||||
trigger too, so the stored record was itself the sharp document — and every
|
||||
listing, menu and search row then showed a title that ran to kilobytes.
|
||||
The trigger now joins the title at embed time (`embeddings.document_title`,
|
||||
reading `data`), producing the same text as before, so nothing re-embeds
|
||||
and `CHUNKER_VERSION` does not move.
|
||||
"""
|
||||
return (
|
||||
compose_title(what, when_to_apply),
|
||||
(what or "").strip(),
|
||||
compose_body(insight, when_to_apply, learned_from),
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user