feat(retrieval): a task's work logs join the document it is embedded as (#4251)

Step 1 of #4251. A work log is the richest prose Scribe holds about WHY
something is the way it is — written during the work, recording what was tried
and ruled out. #4241 made it readable from the agent's door. It was still not
findable, so "has anyone tried this approach?" — precisely the question a log
answers — could not reach one. The cost is not hypothetical: #4208 was rebuilt
in this session because its logs were unreachable.

THE DESIGN QUESTION the issue left open was whether logs embed as part of their
task's document or as rows of their own. As their own rows, a hit has to be
resolved back to a task to be worth anything, and it needs a fourth search, a
fourth result shape and a fourth arm. As part of the task, the objection is
that a long log drowns a short title.

That objection was true before #280 and is not true now. Chunking made one
record into one vector per section, so each log becomes its own title-anchored
chunk, scored separately, and the task's own prose keeps the chunk it always
had — a task is as findable as its best-matching log rather than as the average
of everything in it. The other half is this session's other build: a search
hands back the chunk that won (#4243), so a hit earned by a log shows that
log's passage under the task's title. Without that a reader would have got
body[:240] of the task — the opening of a record whose relevance lives three
hundred lines further down.

So `task_document(title, body, logs)` sits beside `rule_document`: a
synthesised embed-time shape, because the stored record is the task row and the
logs live in their own table, so the document that should be searchable exists
nowhere until it is built. A task with no logs is returned untouched — most
notes are not tasks and most tasks carry no log, and their vectors are the
corpus every tuned number here was measured against.

CHUNKER_VERSION 1 → 2, and its comment now says what the version actually
means. It used to read "whenever chunk_document's output can change for the
same input", which this change would slip past: `chunk_document` is untouched
and every task with a log now embeds differently while its title, body and the
chunker all stand still. The invariant is the document a record is embedded as.
The startup backfill re-embeds on that.

Create, edit and delete of a log all refresh the task through `embed_note`, the
one path every writer shares — an edited log whose vectors still carry its old
wording keeps matching what it no longer says.

No new kind enters the auto-inject menu: tasks were always in it, and this
makes recall on them better rather than changing what the menu spans. The
calibration stamp will now report shape_version 2 against numbers measured at
1, which is exactly the report #4104 built it to make.

Also promoted `session_returning` into tests/helpers beside `make_mock_session`
(#2834) — two files had spelled it out identically and a third was about to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/embeddings.py

@@ -23,6 +23,7 @@ from sqlalchemy import delete, or_, select
 from scribe.models import async_session
 from scribe.models.embedding import NoteEmbedding, RuleEmbedding
 from scribe.models.note import Note
+from scribe.models.task_log import TaskLog
 from scribe.services.access import can_read_project, notes_visibility_clause
 
 if TYPE_CHECKING:  # resolves forward refs without importing at runtime
@@ -271,11 +272,20 @@ def untrigger_title(title: str | None, trigger: str | None) -> str:
 # search. The fix is the document shape: one vector per meaningful chunk, and a
 # record is as findable as its best-matching section.
 
-# Bumped whenever chunk_document's output can change for the same input. Stored
-# on every note_embeddings row so the startup backfill can re-embed exactly the
-# notes whose stored shape is stale — a version comparison instead of the table
-# wipe migrations 0067/0077 had to do.
-CHUNKER_VERSION = 1
+# Bumped whenever THE DOCUMENT A RECORD IS EMBEDDED AS can change for a record
+# that itself has not changed. Stored on every note_embeddings row so the
+# startup backfill re-embeds exactly the stale notes — a version comparison
+# instead of the table wipe migrations 0067/0077 had to do.
+#
+# Stated that way rather than as "chunk_document's output for the same input",
+# which is what it used to say: `chunk_document` is only the last step, and
+# version 2 moves without touching it. A task's document now carries its work
+# logs (#4251), so every task that has one embeds differently than it did while
+# its title, body and the chunker are all untouched — exactly the case the
+# narrower wording would have read as "nothing to re-embed".
+#
+#   1 → 2: work logs joined the task document.
+CHUNKER_VERSION = 2
 
 
 # The public name of the space every score lives in, and the two facts that
@@ -483,6 +493,47 @@ async def _claim_parent_row(session, id_column, row_id: int, label: str) -> bool
     return True
 
 
+async def _work_log_sections(note_id: int) -> list[tuple[object, str | None]]:
+    """A task's work logs, oldest first, for its embedded document (#4251).
+
+    Reads the table directly rather than through `task_logs.logs_for_task`,
+    because that function asks a PERMISSION question — may this user read this
+    task — and there is no user here. An index build acts for the record, and
+    the record's vectors carry the owner's `user_id`, so the access decision is
+    made once at search time by the clause that already scopes every hit.
+
+    That also settles what happens on a shared task: a collaborator's log is
+    part of the task's document, so it becomes findable by everyone who can
+    read the task and by nobody else. The same answer `logs_for_task` gives a
+    reader (#4241), which is the point — a log that can be read and not found
+    is the half-surface that issue was about.
+
+    Asked for every note, not only tasks. `upsert_note_embedding` is handed a
+    note_id and no kind — and the three writers that call it would each have to
+    learn to pass one — so a non-task simply has no rows and gets []. One
+    indexed lookup beside an ONNX forward pass over every chunk is not the
+    expense worth adding a parameter to three call sites for.
+
+    Returns [] on any failure. A task whose logs could not be read should embed
+    as its own prose rather than not embed at all: less findable is recoverable
+    at the next write, unindexed is not.
+    """
+    try:
+        async with async_session() as session:
+            result = await session.execute(
+                select(TaskLog.created_at, TaskLog.content)
+                .where(TaskLog.task_id == note_id)
+                .order_by(TaskLog.created_at.asc(), TaskLog.id.asc())
+            )
+            return list(result.all())
+    except Exception:
+        logger.warning(
+            "Could not read work logs for note %d; embedding its own prose only",
+            note_id, exc_info=True,
+        )
+        return []
+
+
 async def upsert_note_embedding(
     note_id: int, user_id: int, title: str | None, body: str | None
 ) -> None:
@@ -496,6 +547,7 @@ async def upsert_note_embedding(
     inserted in one transaction, so a concurrent read sees the old shape or the
     new one, never a mixture.
     """
+    title, body = task_document(title, body, await _work_log_sections(note_id))
     chunks = chunk_document(title, body)
     try:
         if not chunks:
@@ -908,6 +960,71 @@ async def backfill_note_embeddings() -> None:
 
 # ── Rules (milestone 307, note 3026) ────────────────────────────────────
 
+# The heading a work log gets inside its task's embedded document. A CONSTANT
+# because it is load-bearing twice over: `_split_sections` splits on it, so it
+# is what keeps a log from being merged into the task's own prose, and it is
+# what a reader sees at the top of a matched passage — "this is a log entry,
+# not the task's description". Changing it changes the chunk boundaries of
+# every task that has one, which is a CHUNKER_VERSION move.
+WORK_LOG_HEADING = "## Work log"
+
+
+def task_document(
+    title: str | None,
+    body: str | None,
+    logs: "Sequence[tuple[object, str | None]]" = (),
+) -> tuple[str | None, str | None]:
+    """The (title, body) a TASK is EMBEDDED as — its prose plus its work logs.
+
+    A synthesised embed-time shape, like `rule_document` and unlike a lesson:
+    the stored record is the task row, and the logs live in their own table, so
+    the document that should be searchable exists nowhere until it is built
+    here (#4251).
+
+    WHY THE LOGS BELONG IN THE TASK'S DOCUMENT rather than in rows of their
+    own. "Has anyone tried this before?" is answered by a log and asked of a
+    task — a hit on a bare log would have to be resolved back to its task to be
+    worth anything, so the useful result is the task either way. The objection
+    to folding them in is that a long log drowns a short title, and that was
+    true before #280: one vector per record meant a 2,000-word log averaged the
+    task's own subject away, and everything past ~400 words was truncated
+    unread. Chunking removed both. Each log becomes its own section, each
+    section its own title-anchored vector, each scored separately — so a task
+    is as findable as its best-matching log, and the task's own prose keeps the
+    chunk it always had.
+
+    That the result is LEGIBLE is the other half, and it is this session's
+    other build: a search now hands back the chunk that won (#4243), so a hit
+    earned by a log shows that log's passage under the task's title. Without it
+    the reader would get `body[:240]` of the task — the opening of a record
+    whose relevance lives three hundred lines further down.
+
+    Ordering is oldest-first, matching how the web renders the narrative. Only
+    the heading date distinguishes the sections, so it is part of the shape:
+    "when was this tried" is half of what a log answers.
+
+    An entry with no content is skipped rather than emitted as a bare heading —
+    an empty section is a vector with nothing in it but the task's title, which
+    competes with the task's real chunk and says nothing.
+    """
+    sections = []
+    for created_at, content in logs:
+        text = (content or "").strip()
+        if not text:
+            continue
+        stamp = getattr(created_at, "date", None)
+        heading = (
+            f"{WORK_LOG_HEADING} — {stamp()}" if callable(stamp)
+            else WORK_LOG_HEADING
+        )
+        sections.append(f"{heading}\n\n{text}")
+    if not sections:
+        return title, body
+    prose = (body or "").strip()
+    joined = "\n\n".join(sections)
+    return title, f"{prose}\n\n{joined}" if prose else joined
+
+
 def rule_document(
     title: str | None, statement: str | None, when_to_apply: str | None,
 ) -> tuple[str | None, str | None]:

src/scribe/services/task_logs.py

@@ -14,6 +14,36 @@ logger = logging.getLogger(__name__)
 _UNSET = object()
 
 
+async def _refresh_task_document(session, task_id: int) -> None:
+    """Re-embed the task whose work log just changed (#4251).
+
+    A task's embedded document carries its logs, so a log written and not
+    indexed is #4241's half-surface wearing different clothes: the entry is
+    readable and unfindable, and the next session rebuilds what this one ruled
+    out. Create, edit and delete all go through here — an edited log that keeps
+    matching its old wording is the stale-vector problem `upsert_note_embedding`
+    already refuses to leave behind for a body.
+
+    Loads the NOTE rather than synthesising one. `embed_note` reads
+    `title`, `body` and the OWNER's `user_id` off what it is handed, so a
+    stand-in row would index the logs under an empty title — throwing away the
+    per-chunk topical anchor that makes any of this discriminative — and file
+    the vectors under the wrong user.
+
+    Failure is swallowed the way `embed_note`'s own is: a log that saved must
+    not fail on its index refresh. The startup backfill is the backstop.
+    """
+    from scribe.services.notes import embed_note
+
+    try:
+        result = await session.execute(select(Note).where(Note.id == task_id))
+        note = result.scalars().first()
+        if note is not None:
+            embed_note(note)
+    except Exception:  # noqa: BLE001 - indexing never breaks a write
+        logger.exception("embedding refresh failed for task %s", task_id)
+
+
 async def create_log(
     user_id: int,
     task_id: int,
@@ -36,6 +66,7 @@ async def create_log(
         session.add(log)
         await session.commit()
         await session.refresh(log)
+        await _refresh_task_document(session, task_id)
         return log
 
 
@@ -144,6 +175,7 @@ async def update_log(
         log.updated_at = datetime.now(timezone.utc)
         await session.commit()
         await session.refresh(log)
+        await _refresh_task_document(session, log.task_id)
         return log
 
 
@@ -155,6 +187,8 @@ async def delete_log(user_id: int, log_id: int) -> bool:
         log = result.scalars().first()
         if log is None:
             return False
+        task_id = log.task_id
         await session.delete(log)
         await session.commit()
+        await _refresh_task_document(session, task_id)
         return True