fix(backup): a restore stops flattening the record vocabulary, and a column guard stops the next one (#3182)

`_note_rows` carried 16 of the `notes` table's 27 columns. A backup -> restore
cycle reported success and handed back a corpus with every snippet and process
flattened into a plain note, every issue and spike into `work`, every
provenance edge gone, and recurring tasks no longer recurring. The record-type
and kind vocabulary is what #3128 and milestone 312 were about, and a restore
erased it.

Two more found by auditing every row helper rather than only the one being
edited: `_milestone_rows` dropped `body` — a milestone IS the plan (0066), so
every plan restored as a title with no reasoning behind it — and
`_repo_binding_rows` dropped `ref`, the branch a ledger follows (#2873), so a
restored binding silently accounts for a different tree.

`arose_from_id` is deferred to a second pass beside `parent_id`, never written
in the constructor: it is an id in the SOURCE database, so copying it through
lands the edge on whatever record happens to hold that number here. An edge
whose target did not survive stays NULL rather than being guessed at. This is
the trap that kept the fix out of milestone 317 step 1.

THE STRUCTURAL HALF. The coverage guard from #2293 checks TABLES against
Base.metadata; nothing checked COLUMNS, which is how nine went missing from a
table that had been "covered" for years — added to the model and the migration,
both of which fail loudly, and never to the serialiser, which fails silently.
`_COLUMN_EXCLUSIONS` now declares, per table, every column deliberately not
exported and why, and a parameterised guard walks all 23 helpers and asserts
the two agree. Forgetting is no longer expressible.

Reconciling all 23 turned up one more deliberate exclusion worth naming: the
`code_shapes` proposal columns are the machine's standing suggestion, cleared
by judgment and recomputed by every refresh, so carrying them would restore
stale guesses over a tree the proposer has not seen.

Tests: the round trip drives the REAL restore_full_backup against Postgres,
not a reimplementation of its loop — a test that re-derives the remap it is
checking would agree with whatever the product does, including nothing.

Backup v12.

src/scribe/services/backup.py

@@ -51,11 +51,17 @@ logger = logging.getLogger(__name__)
 # v10 (2026-08) added projects.inception + project_rulebook_exclusions
 # (milestone 297): the WHY a project inherits what it does, and its opt-outs.
 # v11 (2026-08) added the note verification trio — notes.verify_with /
-# expires_when / verified_at (milestone 317). NOT an audit of the notes
-# section: it carries 16 of the 27 `notes` columns, and #3182 tracks the nine that
-# have been missing since long before this.
+# expires_when / verified_at (milestone 317).
+# v12 (2026-08) closed #3182: the nine Note columns that had been missing for
+# years (note_type, task_kind, arose_from_id, data, description, the
+# recurrence pair, started_at, completed_at), plus milestones.body — which IS
+# the plan — and repo_bindings.ref. Until v12 a restore reported success and
+# handed back a corpus with every snippet and process flattened into a plain
+# note, every issue and spike into `work`, and every plan reduced to a title.
+# _COLUMN_EXCLUSIONS and its guard landed with it, so the next such column
+# fails the build instead.
 # Bump when the serialized schema changes.
-BACKUP_VERSION = 11
+BACKUP_VERSION = 12
 
 # Every table this backup carries, by its REAL name. Paired with _NOT_INCLUDED
 # below, these two lists must together account for the entire schema — which is
@@ -112,6 +118,82 @@ _NOT_INCLUDED = [
 ]
 
 
+# Columns a backed-up table deliberately does NOT export, per table. Paired
+# with the column-coverage guard in tests/test_services_backup.py, this is
+# _NOT_INCLUDED's shape one level down — and it exists because the table guard
+# could not see the failure it was written to stop.
+#
+# #2293 was six whole tables missing. #3182 was NINE COLUMNS missing from a
+# table that had been "covered" for years: note_type and task_kind, so every
+# snippet and process restored as a plain note and every issue and spike as
+# `work`; arose_from_id, so every provenance edge vanished; the recurrence
+# pair, so recurring tasks stopped recurring; milestones.body, which IS the
+# plan; repo_bindings.ref, the branch a ledger follows. Every one arrived the
+# same way — a column added to the model and the migration, both of which fail
+# loudly, and then never added to the serialiser, which fails silently.
+#
+# So: a new column on a backed-up table now fails the build unless it is either
+# exported or named here with a reason. "I forgot" is no longer expressible.
+_COLUMN_EXCLUSIONS: dict[str, set[str]] = {
+    # Trash is not exported at all, so neither is the batch id that groups a
+    # deletion for restore(). Uniform across every soft-deletable table.
+    "users": set(),
+    "projects": {
+        "deleted_at", "deleted_batch_id",
+        # Credentials-adjacent, same reasoning as api_keys and
+        # forge_connections: a restored project falls back to keyring-by-host
+        # resolution, which is the documented unpinned behaviour (#2778).
+        "forge_connection_id",
+    },
+    "milestones": {"deleted_at", "deleted_batch_id"},
+    "notes": {"deleted_at", "deleted_batch_id"},
+    "task_logs": set(),
+    "note_drafts": set(),
+    "note_versions": set(),
+    "settings": set(),
+    "rulebooks": {"deleted_at", "deleted_batch_id"},
+    "rulebook_topics": {"deleted_at", "deleted_batch_id"},
+    "rules": {"deleted_at", "deleted_batch_id"},
+    "systems": {
+        "deleted_at", "deleted_batch_id",
+        # Travels as `canonical_slug`: the catalog is global and its ids are
+        # per-install, so an id would restore pointing at whatever area
+        # happened to land on that number.
+        "canonical_id",
+        "created_at", "updated_at",
+    },
+    "canonical_systems": {
+        "deleted_at", "deleted_batch_id",
+        # Matched on SLUG at restore, so a target install that already seeded
+        # the standard vocabulary reuses its own rows rather than colliding.
+        "id", "created_at", "updated_at",
+    },
+    # Edge tables: the id is regenerated on insert, and the pair IS the row.
+    "record_systems": {"id", "created_at"},
+    "note_supersessions": {"id", "created_at"},
+    "rule_relations": {"id", "created_at"},
+    "note_usage_events": {"id"},
+    "design_systems": {"deleted_at", "deleted_batch_id", "created_at", "updated_at"},
+    "design_tokens": {"deleted_at", "deleted_batch_id", "created_at", "updated_at"},
+    "repo_bindings": {"id", "created_at", "updated_at"},
+    # Serialised via the model's own to_dict(), so a column reaches the backup
+    # the moment it reaches that method — and the guard still catches one that
+    # reaches neither.
+    "code_shapes": {
+        # The PROPOSER's standing suggestion for an unclassified row, not a
+        # judgment: "looks like an instance of #N", or "repeats with no canon".
+        # Every refresh recomputes it and a judgment clears it, so carrying it
+        # would restore stale machine guesses over a tree the proposer has not
+        # seen. Same reasoning as code_shape_consumers in _NOT_INCLUDED —
+        # derived data is regenerated, never restored.
+        "proposed_snippet_id", "proposal_basis", "proposal_score",
+        "proposal_group", "proposed_at", "proposed_sha",
+    },
+    "code_shape_events": set(),
+    "code_shape_uses": set(),
+}
+
+
 def _dt(val: str | None) -> datetime:
     return datetime.fromisoformat(val) if val else datetime.now(timezone.utc)
 
@@ -244,7 +326,13 @@ def _code_shape_use_rows(rows) -> list[dict]:
 
 def _repo_binding_rows(rows) -> list[dict]:
     return [
-        {"user_id": r.user_id, "project_id": r.project_id, "repo_key": r.repo_key}
+        # `ref` is the branch this binding's ledger follows (#2873). Without
+        # it a restored binding silently falls back to the default branch, and
+        # the shape ledger starts accounting for a different tree (#3182).
+        {
+            "user_id": r.user_id, "project_id": r.project_id,
+            "repo_key": r.repo_key, "ref": r.ref,
+        }
         for r in rows
     ]
 
@@ -286,6 +374,11 @@ def _milestone_rows(rows) -> list[dict]:
         {
             "id": m.id, "user_id": m.user_id, "project_id": m.project_id,
             "title": m.title, "description": m.description, "status": m.status,
+            # THE PLAN. A milestone IS the plan (0066) and `body` is its
+            # design and intent; `description` is only the one-line summary.
+            # Dropping this restored every plan as a title with no reasoning
+            # behind it (#3182).
+            "body": m.body,
             "order_index": m.order_index,
             "created_at": m.created_at.isoformat(),
             "updated_at": m.updated_at.isoformat(),
@@ -295,31 +388,45 @@ def _milestone_rows(rows) -> list[dict]:
 
 
 def _note_rows(rows) -> list[dict]:
-    # INCOMPLETE, and knowingly so — see #3182. This carries 16 of the
-    # `notes` table's 27 columns. `note_type`, `task_kind`, `arose_from_id`, `data`,
-    # `description`, `recurrence_rule`, `recurrence_next_spawn_at`,
-    # `started_at` and `completed_at` are all absent, so a restore flattens
-    # every snippet and process into a plain note and every issue and spike
-    # into `work`. That predates the verification trio below and is tracked
-    # separately rather than fixed in passing: `arose_from_id` points at
-    # another note and needs the same second pass `parent_id` gets, which is
-    # a change with its own trap and deserves its own tests.
-    #
-    # The table-coverage guard cannot see this. It asserts that every TABLE in
-    # Base.metadata is either backed up or declared not-included; nothing
-    # checks columns, which is exactly how nine of them went missing quietly.
     return [
         {
             "id": n.id, "user_id": n.user_id, "title": n.title, "body": n.body,
+            "description": n.description,
             "tags": n.tags or [], "parent_id": n.parent_id,
             "project_id": n.project_id, "milestone_id": n.milestone_id,
             "status": n.status, "priority": n.priority,
             "due_date": n.due_date.isoformat() if n.due_date else None,
             "created_at": n.created_at.isoformat(),
             "updated_at": n.updated_at.isoformat(),
-            # The verification trio (milestone 317, migration 0092). These
-            # travel because they are operator judgment — "somebody checked
-            # this fact, and this is when" — which nothing can recompute.
+            # WHAT KIND OF RECORD THIS IS — both typing axes (#3182). Missing
+            # until now, which meant a restore reported success and handed back
+            # a corpus with every snippet and process flattened into a plain
+            # note and every issue and spike into `work`. Nothing recomputes
+            # these; the vocabulary is simply gone.
+            "note_type": n.note_type,
+            "task_kind": n.task_kind,
+            # Provenance — which record caused this one. Re-mapped in the
+            # second pass beside parent_id, never here: the value is an id in
+            # the SOURCE database.
+            "arose_from_id": n.arose_from_id,
+            # The queryable mirror. The only one of these that would self-heal
+            # (backfill_snippet_data rebuilds it from the body at startup), but
+            # a restore should not hand back a corpus that needs a restart to
+            # become searchable by location.
+            "data": n.data,
+            # Lifecycle: when the work actually started and finished, and the
+            # recurrence rule that makes a task come back. Without these a
+            # restored recurring task simply stops recurring.
+            "started_at": n.started_at.isoformat() if n.started_at else None,
+            "completed_at": n.completed_at.isoformat() if n.completed_at else None,
+            "recurrence_rule": n.recurrence_rule,
+            "recurrence_next_spawn_at": (
+                n.recurrence_next_spawn_at.isoformat()
+                if n.recurrence_next_spawn_at else None
+            ),
+            # The verification trio (milestone 317, migration 0092). Operator
+            # judgment — "somebody checked this fact, and this is when" —
+            # which nothing can recompute.
             "verify_with": n.verify_with,
             "expires_when": n.expires_when,
             "verified_at": n.verified_at.isoformat() if n.verified_at else None,
@@ -770,6 +877,17 @@ async def _restore_v1(data: dict) -> dict:
                 body=n_data.get("body", ""),
                 tags=n_data.get("tags", []),
                 parent_id=None,  # patched below
+                arose_from_id=None,  # patched below, same reason
+                description=n_data.get("description"),
+                note_type=n_data.get("note_type") or "note",
+                task_kind=n_data.get("task_kind") or "work",
+                data=n_data.get("data"),
+                started_at=_dt_or_none(n_data.get("started_at")),
+                completed_at=_dt_or_none(n_data.get("completed_at")),
+                recurrence_rule=n_data.get("recurrence_rule"),
+                recurrence_next_spawn_at=_dt_or_none(
+                    n_data.get("recurrence_next_spawn_at")
+                ),
                 status=n_data.get("status"),
                 priority=n_data.get("priority"),
                 due_date=_d(n_data.get("due_date")),
@@ -789,14 +907,25 @@ async def _restore_v1(data: dict) -> dict:
                 note_id_map[old_id] = note.id
             stats["notes"] += 1
 
-        # Patch parent_id now that all notes have new IDs
+        # Patch the two note->note edges now that every note has a new id.
+        # Both are ids in the SOURCE database, so writing either straight into
+        # the constructor would point at whatever record happens to hold that
+        # number here — a restore that succeeds and silently re-parents (#3182).
+        # An edge whose target did not survive the import is left NULL rather
+        # than guessed at.
         for n_data in data.get("notes", []):
             old_id = n_data.get("id")
+            if not old_id or old_id not in note_id_map:
+                continue
+            note_row = await session.get(Note, note_id_map[old_id])
+            if note_row is None:
+                continue
             old_parent = n_data.get("parent_id")
-            if old_id and old_parent and old_id in note_id_map and old_parent in note_id_map:
-                note_row = await session.get(Note, note_id_map[old_id])
-                if note_row:
-                    note_row.parent_id = note_id_map[old_parent]
+            if old_parent and old_parent in note_id_map:
+                note_row.parent_id = note_id_map[old_parent]
+            old_origin = n_data.get("arose_from_id")
+            if old_origin and old_origin in note_id_map:
+                note_row.arose_from_id = note_id_map[old_origin]
 
         for s_data in data.get("settings", []):
             mapped_user_id = user_id_map.get(s_data.get("user_id", 0))
@@ -889,6 +1018,7 @@ async def _restore_v2(data: dict) -> dict:
                 project_id=mapped_pid,
                 title=m_data.get("title", ""),
                 description=m_data.get("description"),
+                body=m_data.get("body"),
                 status=m_data.get("status", "active"),
                 order_index=m_data.get("order_index", 0),
                 created_at=_dt(m_data.get("created_at")),
@@ -901,6 +1031,7 @@ async def _restore_v2(data: dict) -> dict:
 
         # 4a. Notes — first pass (no parent_id yet)
         notes_with_parents: list[tuple[int, int]] = []  # (new_note_id, old_parent_id)
+        notes_with_origins: list[tuple[int, int]] = []  # (new_note_id, old_arose_from_id)
         for n_data in data.get("notes", []):
             mapped_uid = user_id_map.get(n_data.get("user_id", 0))
             if mapped_uid is None:
@@ -911,6 +1042,17 @@ async def _restore_v2(data: dict) -> dict:
                 body=n_data.get("body", ""),
                 tags=n_data.get("tags", []),
                 parent_id=None,
+                arose_from_id=None,  # patched below, same reason as parent_id
+                description=n_data.get("description"),
+                note_type=n_data.get("note_type") or "note",
+                task_kind=n_data.get("task_kind") or "work",
+                data=n_data.get("data"),
+                started_at=_dt_or_none(n_data.get("started_at")),
+                completed_at=_dt_or_none(n_data.get("completed_at")),
+                recurrence_rule=n_data.get("recurrence_rule"),
+                recurrence_next_spawn_at=_dt_or_none(
+                    n_data.get("recurrence_next_spawn_at")
+                ),
                 project_id=project_id_map.get(n_data["project_id"]) if n_data.get("project_id") else None,
                 milestone_id=milestone_id_map.get(n_data["milestone_id"]) if n_data.get("milestone_id") else None,
                 status=n_data.get("status"),
@@ -928,15 +1070,25 @@ async def _restore_v2(data: dict) -> dict:
             note_id_map[n_data["id"]] = note.id
             if n_data.get("parent_id"):
                 notes_with_parents.append((note.id, n_data["parent_id"]))
+            if n_data.get("arose_from_id"):
+                notes_with_origins.append((note.id, n_data["arose_from_id"]))
             stats["notes"] += 1
 
-        # 4b. Patch parent_id
+        # 4b. Patch the note->note edges. Deferred for the same reason
+        # parent_id always has been: these are ids in the SOURCE database
+        # (#3182). An edge whose target did not survive stays NULL.
         for new_note_id, old_parent_id in notes_with_parents:
             new_parent_id = note_id_map.get(old_parent_id)
             if new_parent_id:
                 note_row = await session.get(Note, new_note_id)
                 if note_row:
                     note_row.parent_id = new_parent_id
+        for new_note_id, old_origin_id in notes_with_origins:
+            new_origin_id = note_id_map.get(old_origin_id)
+            if new_origin_id:
+                note_row = await session.get(Note, new_note_id)
+                if note_row:
+                    note_row.arose_from_id = new_origin_id
 
         # 5. TaskLogs
         for tl_data in data.get("task_logs", []):
@@ -1289,6 +1441,7 @@ async def _restore_v2(data: dict) -> dict:
             session.add(RepoBinding(
                 user_id=mapped_uid, project_id=mapped_pid,
                 repo_key=rb_data.get("repo_key", ""),
+                ref=rb_data.get("ref"),
             ))
             stats["repo_bindings"] += 1