fix(embeddings): the index refresh loses the race it used to deadlock (#3262)

An embedding refresh replaces a record's vectors as delete-then-insert,
which takes the chunk rows first and the parent row second (via the
insert's foreign key). A cascading delete of the parent takes exactly
those two locks in the other order. Postgres calls the cycle a deadlock
and kills one side: sometimes the detached embedder, silently, and
sometimes the user's delete, as a 500 on an operation that should have
worked.

Both upserts now claim the parent row with FOR KEY SHARE NOWAIT before
touching any chunk row. That removes the cycle instead of narrowing it —
either the embedder is first and the delete queues behind it, or the
delete already holds the row and the embedder loses at once, which is
the side designed to lose. FOR KEY SHARE is the lock the insert would
take anyway, so an ordinary edit is unaffected.

The note twin, recorded as unverified on the issue, has the same shape
and the same fix; a trash purge is the hard delete that reaches it.

Unit tests pin the ORDER and the lock mode by compiling the statement;
the integration pair holds a real delete open in one transaction and
proves the embedder returns having written nothing, with a deadline so
a regression fails instead of hanging.

src/scribe/services/embeddings.py

@@ -344,6 +344,49 @@ def chunk_document(title: str | None, body: str | None) -> list[str]:
     return chunks
 
 
+async def _claim_parent_row(session, id_column, row_id: int, label: str) -> bool:
+    """Lock the record a vector belongs to BEFORE rewriting that vector (#3262).
+
+    An embedding write and a cascading delete of the same record take the same
+    two row locks in OPPOSITE orders. The embedder deletes the old chunk rows
+    and then, on INSERT, needs the foreign key's lock on the parent; a delete
+    of the parent — or of the rulebook, topic or project above it — locks the
+    parent first and cascades down into the chunk rows. That is a cycle, and
+    Postgres breaks it by killing one side at random: sometimes the embedding
+    write, which is swallowed and invisible, and sometimes the operator's
+    delete, which surfaces as a 500 on an operation that should have worked.
+
+    Claiming the parent first REMOVES the cycle rather than narrowing it.
+    Either the embedder arrives first and the delete waits its turn behind it,
+    or the delete already holds the row and NOWAIT makes the embedder lose at
+    once. The embedder is the side that should lose: a skipped refresh costs a
+    stale vector until the next write or the startup backfill, and the other
+    outcome costs a person their request.
+
+    FOR KEY SHARE, not FOR UPDATE — it is precisely the lock the INSERT's
+    foreign key would take anyway, so it conflicts with a delete of the parent
+    and with nothing else. An ordinary edit of the same record, or a second
+    refresh racing this one, is unaffected.
+
+    Returns False when the row is locked or already gone; the caller skips.
+    """
+    try:
+        held = (await session.execute(
+            select(id_column)
+            .where(id_column == row_id)
+            .with_for_update(key_share=True, nowait=True)
+        )).scalar_one_or_none()
+    except Exception:
+        # LockNotAvailable: this record is being deleted right now. Not an
+        # error — the delete wins by design.
+        logger.debug("Skipping embedding for %s %d — row is being deleted", label, row_id)
+        return False
+    if held is None:
+        logger.debug("Skipping embedding for %s %d — row is gone", label, row_id)
+        return False
+    return True
+
+
 async def upsert_note_embedding(
     note_id: int, user_id: int, title: str | None, body: str | None
 ) -> None:
@@ -380,6 +423,8 @@ async def upsert_note_embedding(
 
     try:
         async with async_session() as session:
+            if not await _claim_parent_row(session, Note.id, note_id, "note"):
+                return
             await session.execute(
                 delete(NoteEmbedding).where(NoteEmbedding.note_id == note_id)
             )
@@ -666,6 +711,8 @@ async def upsert_rule_embedding(
     replacement is atomic per rule so a concurrent read sees the old chunk set
     or the new one, never a mixture.
     """
+    from scribe.models.rulebook import Rule  # runtime import: see TYPE_CHECKING above
+
     doc_title, doc_body = rule_document(title, statement, when_to_apply)
     chunks = chunk_document(doc_title, doc_body)
     try:
@@ -688,6 +735,8 @@ async def upsert_rule_embedding(
 
     try:
         async with async_session() as session:
+            if not await _claim_parent_row(session, Rule.id, rule_id, "rule"):
+                return
             await session.execute(
                 delete(RuleEmbedding).where(RuleEmbedding.rule_id == rule_id)
             )