refactor(embeddings): one definition of the document a record is embedded as
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 34s
CI & Build / Python tests (push) Successful in 50s
CI & Build / Build & push image (push) Successful in 26s

`f"{title}\n{body}"` was written out four times. #2486 found three — the write
path, the recurring-task spawn, the startup backfill. The guard added here
found the fourth immediately, and it was the one that mattered most.

`dedup.find_duplicate_note` built the same string as a QUERY, compared against
embedded documents. Shaped differently from the corpus it searches, the gate
degrades silently: it still returns neighbours, just less apt ones, and nothing
says the query and the index stopped agreeing. The spawn path has the same
shape of risk — a recurring task embedded differently from everything else is
ranked against documents it doesn't match.

None of the four had diverged. That is what makes this worth doing now rather
than after: they are identical today, so collapsing them is a no-op, and the
whole point is that the next change to the shape can't hit three of four.

Which is imminent. #2486 measured a dev-log separating from five unrelated
dev-logs by 0.023 where a snippet separates by 0.153 — the difference being
that a snippet states its purpose twice in a short document. Whether that shape
is right is the open question; testing an alternative against four copies would
mean testing a shape that isn't the one in production. This is the precondition
the issue named.

The guard is source inspection, matching the f-string pattern rather than a
variable name, so a copy that renames its locals is still caught. A behavioural
test cannot see this: an inlined copy produces the same string today and
diverges the day the shape changes.

Refs #2486
This commit is contained in:
2026-08-07 13:01:45 -04:00
parent 3f26aa9485
commit bbba0b3ae3
5 changed files with 120 additions and 8 deletions
+26 -1
View File
@@ -86,6 +86,31 @@ def _cosine_similarity(a: list[float], b: list[float]) -> float:
return dot / (mag_a * mag_b)
def embedding_text(title: str | None, body: str | None) -> str:
"""The document a record is embedded AS.
One definition, deliberately. This was written out three times — the write
path (`notes.embed_note`), the recurring-task spawn, and the startup
backfill — and identical copies of a formatting rule are three chances to
change one and not the others. The spawn path is the dangerous one: a
recurring task embedded to a different shape than everything else would be
ranked against a corpus it doesn't match, and nothing would report it.
It is also a PRECONDITION for changing the shape at all (#2486). Measured,
a dev-log's vector separates from five unrelated dev-logs by 0.023 while a
snippet's separates by 0.153 — the difference being that a snippet states
its purpose twice in a short document, so the purpose dominates. Testing an
alternative shape against three copies would mean testing a shape that is
not the one in production.
Whether `title\\n{body}` is the RIGHT shape is the open question. That it is
one shape is what makes the question answerable.
"""
title = title or ""
body = body or ""
return f"{title}\n{body}".strip() if body else title
async def upsert_note_embedding(note_id: int, user_id: int, text: str) -> None:
"""Generate and persist an embedding for a note. Safe to fire-and-forget."""
if not text or not text.strip():
@@ -248,7 +273,7 @@ async def backfill_note_embeddings() -> None:
logger.info("Embedding backfill: generating embeddings for %d notes", len(notes_to_embed))
success = 0
for note_id, user_id, title, body in notes_to_embed:
text = f"{title}\n{body}".strip() if body else (title or "")
text = embedding_text(title, body)
if not text:
continue
await upsert_note_embedding(note_id, user_id, text)