bvandeusen/FabledScribe
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases 53 Wiki Activity

docs(6205): backup comments state the reason, not the instance that taught it
CI & Build / Python lint (push) Successful in 2s
Details
CI & Build / Plugin hooks (push) Successful in 13s
Details
CI & Build / TypeScript typecheck (push) Successful in 52s
Details
CI & Build / integration (push) Successful in 1m11s
Details
CI & Build / Python tests (push) Successful in 1m55s
Details
CI & Build / Build & push image (push) Successful in 22s
Details

Browse Source 
The format-version log keeps what each version adds, because restore depends
on what an older archive can carry; dates, milestone and issue ids, and the
stories of what a missing column once cost are removed. Compatibility
comments now name the archive version instead of a milestone. Two leftover
history-phrased comments in embeddings and shape_ledger are restated as
present-tense reasons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-21 20:08:12 -04:00
co-authored by Claude Opus 5.5
parent 0c24fa392c
commit be9a659e9b
3 changed files with 196 additions and 234 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/scribe/services/backup.py
+188 -225
View File
@@ -42,92 +42,75 @@ from scribe.models.user_profile import UserProfile
logger = logging.getLogger(__name__)
# Backup format version. v3 (2026-06) added rulebooks/topics/rules + their
# project subscription/suppression join tables. v4 (2026-07) dropped events
# when the calendar surface was retired — old v3 events are skipped on restore.
# v5 (2026-08) added the six tables that had accumulated outside the backup
# entirely (#2293), and the coverage guard that stops the seventh.
# v6 (2026-08) added note_supersessions — and the guard did stop the seventh:
# the table shipped without a backup section and the coverage test failed the
# build, which is the whole reason that list was written.
# v7 (2026-08) added code_shapes — the shape ledger (#2787). Classifications
# are judgment data worth carrying; a restore keeps a judgment only when its
# snippet target survives the id re-mapping, else the row rejoins the todo.
# v8 (2026-08) added code_shape_events — the ledger's history (#2793): what
# was used where, when, and why is not recomputable, so it travels.
# v9 (2026-08) added code_shape_uses — the ledger's consumption edges (#2870):
# judgment-grade edges (agent/audit/import) are operator records; mechanical
# ones (reference/hook) travel too, cheaply, and the next refresh refreshes them.
# v10 (2026-08) added projects.inception + project_rulebook_exclusions
# (milestone 297): the WHY a project inherits what it does, and its opt-outs.
# v11 (2026-08) added the note verification trio — notes.verify_with /
# expires_when / verified_at (milestone 317).
# v12 (2026-08) closed #3182: the nine Note columns that had been missing for
# years (note_type, task_kind, arose_from_id, data, description, the
# recurrence pair, started_at, completed_at), plus milestones.body — which IS
# the plan — and repo_bindings.ref. Until v12 a restore reported success and
# handed back a corpus with every snippet and process flattened into a plain
# note, every issue and spike into `work`, and every plan reduced to a title.
# _COLUMN_EXCLUSIONS and its guard landed with it, so the next such column
# fails the build instead.
# v13 (2026-08) added rule_versions — a rule's edit history (milestone 323).
# v14 (2026-09) added rule_usage_events — the rule twin of note_usage_events
# (milestone 333). Carrying it is the WHOLE REASON the table is separate: the
# note importer maps note_id through note_id_map, so a rule id parked there
# would restore attached to whatever note took that number.
# v15 (2026-09) dropped rulebook_subscriptions / rule_suppressions /
# topic_suppressions with their tables (milestone 414): a rule's scope is its
# home now. Older archives carrying those sections still restore — the keys are
# simply not read — as do the subscribe_rulebooks inception choices they hold.
# v16 (2026-09) added retrieval_tuning_events (milestone 416): the REASON each
# retrieval floor and budget is where it is. `settings` already carried the
# numbers, so leaving this behind would restore six moved dials with the
# argument for them silently dropped — and from this step on those dials are
# moved by the model, which is exactly the case where the operator needs the
# argument to review.
# v17 (2026-09) added retrieval_tuning_events.embedding_model / shape_version
# (milestone 416 step 6): a floor is a distance in ONE embedding model's
# geometry over documents cut one particular way, so the number alone cannot
# say whether it still measures anything. Both travel NULLABLE and unfilled —
# a row written before the stamp existed restores unstamped, because inventing
# the model it was measured under would turn "unknown" into a stated fact.
# v18 (2026-10) added lesson_rule_links (milestone 440): which rule each lesson
# is an instance of, and the judgments that confirmed or rejected each pair. A
# confirmed link is a judgment nothing else records, and a rejected one is
# what stops the pair being proposed again — losing either undoes work.
# v19 (2026-10) added lesson_no_rule (#4631): the "no rule fits" answer and its
# reason. Without it a restored lesson that was judged to stand alone reads as
# never judged, and lands back on the unjudged list.
# v20 (2026-10) added systems.path_patterns and system_usage_events (milestone
# 444): the files that are each area, and whether an area's rulings were read
# once shown. The usage rows restore through the SYSTEM map, for the reason
# the rule twin restores through the rule map.
# v21 (2026-10) added moment_mappings (milestone 458): which of this install's
# actions reach which moment, and the shipped defaults it switched off. Each
# row is a correction an operator made in-session; a restore that dropped them
# would silently put every misfire back.
# v22 (2026-10) added rule_moments (milestone 458): the moments each rule
# arrives at. A mount is a judgment about when a rule applies that nothing in
# the rule's text records, and losing it would put back exactly the misses the
# mounts were made to fix.
# v23 (2026-10) added rule_moment_judgments (milestone 458 step 7): the
# proposals waiting on a mount, and the rejections and "no moment fits"
# answers that stop a pass or the open-after-moment signal proposing the same
# pair again. Losing them puts every answered question back on the list.
# v24 (2026-10) added rule_moment_judgments.misfire (milestone 458 step 7b):
# the reports that a mounted rule arrived where it did not apply, and the
# operator's "keep it" that stops them being proposed as an unmount again.
# v25 (2026-10) added family canon (milestone 463): the platform catalog, each
# project's platforms, family ideas with their platforms and reference
# implementations, the adoption ledger and the decision log. The ledger is
# every project's recorded answer to a shared idea, and the log is the
# precedent each later answer follows. Lose either and every assessment is
# owed again, made fresh, with nothing to keep it consistent with the last.
# v26 (2026-10) added user_profiles (#5879): the Profile page — name, role,
# expertise, response style, tone, interests, working hours. Every field is
# typed by its user and none is derived or a credential, so leaving it out made
# a restore drop it without a word.
# Bump when the serialized schema changes.
# Backup format version. Bump when the serialized schema changes. What each
# version added, so a restore knows what an older archive can carry:
#
# v3 rulebooks, topics and rules.
# v4 drops calendar events; a v3 archive's events are skipped on restore.
# v5 systems, record_systems, design systems and tokens, note usage events,
# repo bindings — and the coverage guard below.
# v6 note_supersessions.
# v7 code_shapes, the shape ledger. A restore keeps a classification only
# when its snippet target survives the id re-mapping, else the row rejoins
# the todo.
# v8 code_shape_events, the ledger's history: what was used where, when and
# why is not recomputable, so it travels.
# v9 code_shape_uses, the ledger's consumption edges. Judgment-grade edges
# (agent/audit/import) are operator records; mechanical ones
# (reference/hook) travel too, cheaply, and the next refresh refreshes
# them.
# v10 projects.inception and project_rulebook_exclusions: why a project
# inherits what it does, and its opt-outs.
# v11 the note verification trio — verify_with / expires_when / verified_at.
# v12 the Note typing and provenance columns (note_type, task_kind,
# arose_from_id, data, description, the recurrence pair, started_at,
# completed_at), milestones.body — which IS the plan — and
# repo_bindings.ref; and the column guard below. An archive older than
# v12 restores every snippet and process as a plain note, every issue and
# spike as `work`, and every plan as a title.
# v13 rule_versions, a rule's edit history.
# v14 rule_usage_events, the rule twin of note_usage_events. Carrying it is the
# WHOLE REASON the table is separate: the note importer maps note_id
# through note_id_map, so a rule id parked there would restore attached to
# whatever note took that number.
# v15 drops the rule subscription and suppression tables: a rule's scope is
# its home. Older archives carrying those sections still restore — the
# keys are simply not read.
# v16 retrieval_tuning_events: the REASON each retrieval floor and budget is
# where it is. `settings` carries the numbers; without this a restore
# keeps moved dials and drops the argument for them, which an operator
# needs to review a dial the model moved.
# v17 retrieval_tuning_events.embedding_model / shape_version: a floor is a
# distance in ONE embedding model's geometry over documents cut one
# particular way. Both travel NULLABLE — an unstamped row restores
# unstamped, because inventing the model it was measured under would turn
# "unknown" into a stated fact.
# v18 lesson_rule_links: which rule each lesson is an instance of, and the
# judgments on each pair. A confirmed link is a judgment nothing else
# records, and a rejected one stops the pair being proposed again.
# v19 lesson_no_rule: the "no rule fits" answer and its reason. Without it a
# lesson judged to stand alone lands back on the unjudged list.
# v20 systems.path_patterns and system_usage_events. The usage rows restore
# through the SYSTEM map, for the reason the rule twin restores through the
# rule map.
# v21 moment_mappings: which of this install's actions reach which moment, and
# the shipped defaults it switched off. Each row is a correction an
# operator made; dropping them puts every misfire back.
# v22 rule_moments: the moments each rule arrives at — a judgment about when a
# rule applies that nothing in the rule's text records.
# v23 rule_moment_judgments: proposals waiting on a mount, and the rejections
# and "no moment fits" answers that stop the same pair being proposed
# again.
# v24 rule_moment_judgments.misfire: reports that a mounted rule arrived where
# it did not apply, and the operator's "keep it" that stops them being
# proposed as an unmount again.
# v25 family canon: the platform catalog, each project's platforms, family
# ideas with their platforms and reference implementations, the adoption
# ledger and the decision log. Lose the ledger or the log and every
# assessment is owed again, with nothing to keep it consistent with the
# last.
# v26 user_profiles: what each user wrote on their Profile page. Every field
# is typed by its user, and none is derived or a credential.
BACKUP_VERSION = 26
# Every table this backup carries, by its REAL name. Paired with _NOT_INCLUDED
@@ -137,55 +120,49 @@ BACKUP_VERSION = 26
# The point is the ABSENCE case. A new table gets a model and a migration, both
# of which fail loudly if wrong, and then silently never gets a backup section:
# no error, no warning, and a restore that reports success. Naming the coverage
# explicitly turns "someone forgot" into a failing test (#2293).
# explicitly turns "someone forgot" into a failing test.
_BACKED_UP = [
"users", "projects", "milestones", "notes", "task_logs", "note_drafts",
"note_versions", "settings", "rulebooks", "rulebook_topics", "rules",
# v5 (2026-08): the five-year gap this list was written to stop.
"systems", "record_systems", "design_systems", "design_tokens",
"note_usage_events", "repo_bindings", "note_supersessions",
# v7 (2026-08): the shape ledger (#2787); v8: its history (#2793).
# The shape ledger, its history and its consumption edges.
"code_shapes", "code_shape_events", "code_shape_uses",
# v9 (2026-08): the global area catalog (milestone 307). Global, not
# The global area catalog. Global, not
# user-scoped, so it rides in EVERY export — including a single-user
# one, whose Systems would otherwise restore unmapped.
"canonical_systems",
# v10 (2026-08): a rule's area tag and its typed edges (milestone 307).
# A rule's area tag and its typed edges.
"rule_systems", "rule_relations",
# v13 (2026-08): a rule's edit history (milestone 323). note_versions has
# always travelled; its sibling has no excuse not to.
# A rule's edit history, for the reason note_versions travels.
"rule_versions",
# v14 (2026-09): rule usage telemetry (milestone 333). Same argument
# Rule usage telemetry. Same argument
# note_usage_events makes for itself — pull-through is only ever
# accumulated, so a restore that dropped it would silently reset the
# measurement to zero while everything still looked fine.
"rule_usage_events",
# v16 (2026-09): why each retrieval floor and budget is where it is
# (milestone 416). The values live in `settings` and already travelled; the
# argument for them had nowhere to go. Now that the model moves these dials
# on the operator's behalf, a restore that kept the numbers and dropped the
# reasons would leave an install tuned by nobody it can name.
# Why each retrieval floor and budget is where it is. The values live in
# `settings`; the model moves these dials on the operator's behalf, so a
# restore that kept the numbers and dropped the reasons would leave an
# install tuned by nobody it can name.
"retrieval_tuning_events",
# v18 (2026-10): lesson → rule links and their judgments (milestone 440).
# Lesson → rule links and their judgments, and "no rule fits" answers.
"lesson_rule_links",
# v19 (2026-10): "no rule fits" answers (#4631).
"lesson_no_rule",
# v20 (2026-10): System usage telemetry (milestone 444), for the reason
# its note and rule twins travel.
# System usage telemetry, for the reason its note and rule twins travel.
"system_usage_events",
# v21 (2026-10): an install's action → moment corrections (milestone 458).
# An install's action → moment corrections, the moments each rule arrives
# at, and the proposals and judgments about those mounts.
"moment_mappings",
# v22 (2026-10): the moments each rule arrives at (milestone 458).
"rule_moments",
# v23 (2026-10): proposals and judgments about those mounts (step 7).
"rule_moment_judgments",
# v25 (2026-10): family canon (milestone 463). `platforms` is global like
# Family canon. `platforms` is global like
# canonical_systems and rides every export for the same reason: the rows
# below name platforms by slug, and a partial catalog restores partial
# memberships.
"platforms", "project_platforms", "family_ideas", "family_idea_platforms",
"family_idea_references", "family_adoptions", "family_decisions",
# v26 (2026-10): what each user wrote on their Profile page (#5879).
# What each user wrote on their Profile page.
"user_profiles",
]
@@ -197,9 +174,8 @@ _BACKED_UP = [
# reads for correctness and that grows per query; the rest are
# transient/operational.
#
# REAL table names, deliberately. This list used to read "embeddings",
# "invitations", "password_resets" — none of which are tables — so it looked
# like coverage while naming nothing the schema could confirm.
# REAL table names, deliberately: a name that is not a table looks like
# coverage while naming nothing the schema can confirm.
_NOT_INCLUDED = [
"groups", "group_memberships", "project_shares", "note_shares",
"api_keys", "note_embeddings", "rule_embeddings", "milestone_embeddings",
@@ -207,7 +183,7 @@ _NOT_INCLUDED = [
"app_logs", "notifications",
"invitation_tokens", "password_reset_tokens",
"retrieval_logs",
# Verdicts on retrieval_logs rows (#4772), so they go where their subject
# Verdicts on retrieval_logs rows, so they go where their subject
# goes: a restore that carried a judgment of a call it did not carry would
# leave a verdict on nothing. A sample, re-takeable by the review pass.
"retrieval_judgments",
@@ -215,9 +191,9 @@ _NOT_INCLUDED = [
# forge tokens is a token-exfiltration file. Users re-add connections
# after a restore; the per-project pin (projects.forge_connection_id) is
# deliberately not exported either, so restored projects fall back to
# keyring-by-host resolution — the documented unpinned behavior (#2778).
# keyring-by-host resolution — the documented unpinned behavior.
"forge_connections",
# Derived, like note_embeddings: the CSS consumer map (milestone 302) is
# Derived, like note_embeddings: the CSS consumer map is
# rebuilt from the repo archive by every coverage sync, and carries no
# judgment — the first refresh after a restore recreates it exactly.
"code_shape_consumers",
@@ -226,20 +202,14 @@ _NOT_INCLUDED = [
# Columns a backed-up table deliberately does NOT export, per table. Paired
# with the column-coverage guard in tests/test_services_backup.py, this is
# _NOT_INCLUDED's shape one level down — and it exists because the table guard
# could not see the failure it was written to stop.
# _NOT_INCLUDED's shape one level down. The table guard cannot see a column
# missing from a table it counts as covered — note_type dropped would restore
# every snippet as a plain note; milestones.body dropped would restore every
# plan as a title. A column added to the model and the migration fails loudly
# if wrong; one never added to the serialiser fails silently.
#
# #2293 was six whole tables missing. #3182 was NINE COLUMNS missing from a
# table that had been "covered" for years: note_type and task_kind, so every
# snippet and process restored as a plain note and every issue and spike as
# `work`; arose_from_id, so every provenance edge vanished; the recurrence
# pair, so recurring tasks stopped recurring; milestones.body, which IS the
# plan; repo_bindings.ref, the branch a ledger follows. Every one arrived the
# same way — a column added to the model and the migration, both of which fail
# loudly, and then never added to the serialiser, which fails silently.
#
# So: a new column on a backed-up table now fails the build unless it is either
# exported or named here with a reason. "I forgot" is no longer expressible.
# So a new column on a backed-up table fails the build unless it is either
# exported or named here with a reason. "I forgot" is not expressible.
_COLUMN_EXCLUSIONS: dict[str, set[str]] = {
# Trash is not exported at all, so neither is the batch id that groups a
# deletion for restore(). Uniform across every soft-deletable table.
@@ -248,14 +218,14 @@ _COLUMN_EXCLUSIONS: dict[str, set[str]] = {
"deleted_at", "deleted_batch_id",
# Credentials-adjacent, same reasoning as api_keys and
# forge_connections: a restored project falls back to keyring-by-host
# resolution, which is the documented unpinned behaviour (#2778).
# resolution, which is the documented unpinned behaviour.
"forge_connection_id",
},
"milestones": {"deleted_at", "deleted_batch_id"},
"notes": {
"deleted_at", "deleted_batch_id",
# A claim is a session's attention, not the work's state (milestone
# 381). Restoring one would assert that a session on another install,
# A claim is a session's attention, not the work's state. Restoring
# one would assert that a session on another install,
# possibly long gone, is working the task right now.
"claimed_by", "claimed_at", "claim_touched_at", "claim_session",
},
@@ -339,7 +309,7 @@ _COLUMN_EXCLUSIONS: dict[str, set[str]] = {
}
# The same accounting for the way BACK IN (#4197). `_COLUMN_EXCLUSIONS` above
# The same accounting for the way BACK IN. `_COLUMN_EXCLUSIONS` above
# makes a dropped column unexpressible on the way out; this one does it for the
# import, where the same mistake is worse — an export gap leaves an obviously
# thin backup, an import gap means holding a complete, correct file and
@@ -553,7 +523,7 @@ def _rule_usage_event_rows(rows) -> list[dict]:
{
"user_id": r.user_id, "rule_id": r.rule_id, "event": r.event,
"source": r.source,
# The reason a rule was departed from (#4212). Exported because
# The reason a rule was departed from. Exported because
# it is the only field on this table that cannot be recomputed:
# counts can be re-derived from a fresh install's own use, a
# stated reason cannot, and a `departed` row that comes back
@@ -566,7 +536,7 @@ def _rule_usage_event_rows(rows) -> list[dict]:
def _moment_mapping_rows(rows) -> list[dict]:
"""An install's action → moment corrections (milestone 458). Not
"""An install's action → moment corrections. Not
`to_dict()`, for _retrieval_tuning_event_rows' reason: the restore needs
`user_id` to remap, and the MCP reader omits it."""
return [
@@ -581,7 +551,7 @@ def _moment_mapping_rows(rows) -> list[dict]:
def _user_profile_rows(rows) -> list[dict]:
"""The Profile page (#5879). Not `to_dict()`: that one fills blanks with
"""The Profile page. Not `to_dict()`: that one fills blanks with
the page's defaults and omits `user_id`, and a restore needs the stored
value and the owner."""
return [
@@ -599,7 +569,7 @@ def _user_profile_rows(rows) -> list[dict]:
def _retrieval_tuning_event_rows(rows) -> list[dict]:
"""The record of why a retrieval dial is where it is (#4102).
"""The record of why a retrieval dial is where it is.
Not `to_dict()`: that method serves the MCP reader, which already knows
whose install it is asking about, so it omits `user_id`. A restore has to
@@ -611,8 +581,8 @@ def _retrieval_tuning_event_rows(rows) -> list[dict]:
"user_id": r.user_id, "surface": r.surface, "dial": r.dial,
"old_value": r.old_value, "new_value": r.new_value,
"actor": r.actor, "reason": r.reason,
# Carried, and NOT defaulted to the current model on the way out
# (#4104): a row that was unstamped when it was written is still
# Carried, and NOT defaulted to the current model on the way
# out: a row that was unstamped when it was written is still
# unstamped after a round trip, and a backup that quietly filled
# the gap would turn "we don't know" into a stated fact.
"embedding_model": r.embedding_model,
@@ -637,9 +607,9 @@ def _code_shape_use_rows(rows) -> list[dict]:
def _repo_binding_rows(rows) -> list[dict]:
return [
# `ref` is the branch this binding's ledger follows (#2873). Without
# it a restored binding silently falls back to the default branch, and
# the shape ledger starts accounting for a different tree (#3182).
# `ref` is the branch this binding's ledger follows. Without it a
# restored binding silently falls back to the default branch, and the
# shape ledger starts accounting for a different tree.
{
"user_id": r.user_id, "project_id": r.project_id,
"repo_key": r.repo_key, "ref": r.ref,
@@ -649,9 +619,8 @@ def _repo_binding_rows(rows) -> list[dict]:
# Row builders for the sections both exporters carry. Pure, like the join-table
# helpers above; the full and per-user exports used to restate every one of
# these comprehensions side by side, and a column added to one and not the
# other is a backup that silently drops it (#2293's shape, one layer down).
# helpers above, and shared so the full and per-user exports cannot drift: a
# column added to one and not the other is a backup that silently drops it.
def _user_rows(rows) -> list[dict]:
return [
@@ -685,10 +654,10 @@ def _milestone_rows(rows) -> list[dict]:
{
"id": m.id, "user_id": m.user_id, "project_id": m.project_id,
"title": m.title, "description": m.description, "status": m.status,
# THE PLAN. A milestone IS the plan (0066) and `body` is its
# design and intent; `description` is only the one-line summary.
# Dropping this restored every plan as a title with no reasoning
# behind it (#3182).
# THE PLAN. A milestone IS the plan and `body` is its design and
# intent; `description` is only the one-line summary. Dropping
# this would restore every plan as a title with no reasoning
# behind it.
"body": m.body,
"order_index": m.order_index,
"created_at": m.created_at.isoformat(),
@@ -709,11 +678,10 @@ def _note_rows(rows) -> list[dict]:
"due_date": n.due_date.isoformat() if n.due_date else None,
"created_at": n.created_at.isoformat(),
"updated_at": n.updated_at.isoformat(),
# WHAT KIND OF RECORD THIS IS — both typing axes (#3182). Missing
# until now, which meant a restore reported success and handed back
# a corpus with every snippet and process flattened into a plain
# note and every issue and spike into `work`. Nothing recomputes
# these; the vocabulary is simply gone.
# WHAT KIND OF RECORD THIS IS — both typing axes. Without them a
# restore reports success and hands back a corpus with every
# snippet and process flattened into a plain note and every issue
# and spike into `work`. Nothing recomputes these.
"note_type": n.note_type,
"task_kind": n.task_kind,
# Provenance — which record caused this one. Re-mapped in the
@@ -735,7 +703,7 @@ def _note_rows(rows) -> list[dict]:
n.recurrence_next_spawn_at.isoformat()
if n.recurrence_next_spawn_at else None
),
# The verification trio (milestone 317, migration 0092). Operator
# The verification trio. Operator
# judgment — "somebody checked this fact, and this is when" —
# which nothing can recompute.
"verify_with": n.verify_with,
@@ -885,7 +853,7 @@ def _rule_relation_rows(rows) -> list[dict]:
def _lesson_rule_link_rows(rows) -> list[dict]:
"""Which rule each lesson is an instance of, with the judgment's state,
reason and evidence (milestone 440). Ids are SOURCE ids, remapped through
reason and evidence. Ids are SOURCE ids, remapped through
the note and rule maps at restore."""
return [
{
@@ -899,7 +867,7 @@ def _lesson_rule_link_rows(rows) -> list[dict]:
def _lesson_no_rule_rows(rows) -> list[dict]:
"""The "no rule fits" answers, by SOURCE lesson id (#4631)."""
"""The "no rule fits" answers, by SOURCE lesson id."""
return [
{
"lesson_id": r.lesson_id, "why": r.why,
@@ -1322,7 +1290,7 @@ async def export_user_backup(user_id: int) -> dict:
select(RuleMomentJudgment).where(RuleMomentJudgment.rule_id.in_(_rule_ids))
)).scalars().all() if _rule_ids else []
# Scoped through the RULE, not the version's user_id. That column is
# the ACTOR (milestone 323), so filtering on it would carry the
# the ACTOR, so filtering on it would carry the
# versions this user wrote on someone ELSE's rule and drop the ones
# someone else wrote on theirs — the opposite of a per-user export.
rule_versions = (await session.execute(
@@ -1478,26 +1446,24 @@ async def export_user_backup(user_id: int) -> dict:
# ---------------------------------------------------------------------------
# Per-table builders — the import side's twin of the row helpers above (#4197)
# Per-table builders — the import side's twin of the row helpers above
# ---------------------------------------------------------------------------
#
# The column guard (#3182) makes a dropped column unexpressible on the way OUT
# because export goes through per-table pure helpers: hand one a stand-in and
# read which KEYS came back. Nothing watched the way back IN, and that is the
# worse half — an export gap leaves an obviously thin backup, an import gap
# means holding a complete, correct file and restoring an incomplete database
# from it, with a success message.
# The column guard makes a dropped column unexpressible on the way OUT because
# export goes through per-table pure helpers: hand one a stand-in and read
# which KEYS came back. The way back IN is the worse half to leave unwatched —
# an export gap leaves an obviously thin backup, an import gap means holding a
# complete, correct file and restoring an incomplete database from it, with a
# success message.
#
# The guard was one-sided because the code was. `_restore_v2` was one long
# procedural function with the model construction inlined in each loop, so
# there was no per-table unit to hand a stand-in to. These builders are that
# unit. Each takes an exported row plus the id maps built so far and returns
# either the model instance to add or None for a row that cannot be restored.
# These builders are the per-table unit the import guard hands a stand-in to.
# Each takes an exported row plus the id maps built so far and returns either
# the model instance to add or None for a row that cannot be restored.
#
# WHAT DELIBERATELY DID NOT MOVE. The loops, the flushes and the id-map
# bookkeeping stay in `_restore_v2`: that is the order-dependent part, where a
# mistake is a restore that half-works, and it gains nothing from being split.
# Only the construction moved, because the construction is the only thing the
# WHAT DELIBERATELY STAYS IN `_restore_v2`: the loops, the flushes and the
# id-map bookkeeping. That is the order-dependent part, where a mistake is a
# restore that half-works, and it gains nothing from being split. Only the
# construction lives here, because the construction is the only thing the
# guard has to see.
#
# RETURNING None IS "SKIP", A None FIELD IS "DEGRADE", and the difference is
@@ -1589,7 +1555,7 @@ def _build_milestone(row: dict, maps: _Maps) -> Milestone | None:
def _build_note(row: dict, maps: _Maps) -> Note | None:
"""`parent_id` and `arose_from_id` land NULL and are patched afterwards.
Both are ids in the SOURCE database (#3182), and the target note may not
Both are ids in the SOURCE database, and the target note may not
have been created yet when this row is built.
"""
uid = maps.users.get(row.get("user_id", 0))
@@ -1733,9 +1699,9 @@ def _build_retrieval_tuning_event(row: dict, maps: _Maps) -> RetrievalTuningEven
new_value=row.get("new_value"),
actor=row.get("actor") or "model",
reason=row.get("reason", ""),
# .get with no default, deliberately (v17): an archive written before
# the stamp existed has no key here, and None is the right answer for
# it — the same "unstamped" a pre-#4104 row carries in place.
# .get with no default, deliberately: an archive older than v17 has
# no key here, and None is the right answer for it — the same
# "unstamped" an unstamped row carries in place.
embedding_model=row.get("embedding_model"),
shape_version=row.get("shape_version"),
created_at=_dt(row.get("created_at")),
@@ -1784,16 +1750,15 @@ def _build_rule(row: dict, maps: _Maps) -> Rule | None:
why=row.get("why") or None,
how_to_apply=row.get("how_to_apply") or None,
when_to_apply=row.get("when_to_apply") or None,
# A file written before milestone 394 carries `tier` and `always_on`;
# neither is read. Dropping a field the schema no longer has is the
# tolerant direction — an archive records what WAS, and refusing it
# because it remembers a deleted column would make every pre-394
# backup unrestorable.
# An older archive may carry `tier` and `always_on`; neither is read.
# Dropping a field the schema does not have is the tolerant direction
# — an archive records what WAS, and refusing it because it remembers
# a deleted column would make every such backup unrestorable.
#
# Same shape, same reason: a file written before 0098 has no kind, and
# every rule in it was a rule. Defaulting the other way would restore
# an old backup with things that had always bound quietly no longer
# binding.
# Same shape, same reason: an archive with no `kind` predates
# preferences, and every rule in it is a rule. Defaulting the other
# way would restore an old backup with things that had always bound
# quietly not binding.
kind=row.get("kind") or "rule",
verify_with=row.get("verify_with") or None,
expires_when=row.get("expires_when") or None,
@@ -1802,9 +1767,8 @@ def _build_rule(row: dict, maps: _Maps) -> Rule | None:
# it would put every constraint at the top of the sweep with nothing
# having actually changed.
verified_at=_dt_or_none(row.get("verified_at")),
# Remapped through the note map like every other note edge. Exported
# since 0088 but dropped on the way back in until milestone 312 — a
# restore silently lost every rule's provenance link. SET NULL
# Remapped through the note map like every other note edge; without
# it a restore silently loses every rule's provenance link. SET NULL
# semantics apply here too: a source note that didn't restore leaves
# the rule intact.
arose_from_id=maps.notes.get(row.get("arose_from_id") or 0),
@@ -2056,7 +2020,7 @@ def _build_system(row: dict, maps: _Maps) -> System | None:
color=row.get("color"),
status=row.get("status", "active"),
order_index=row.get("order_index", 0),
# Absent in a backup taken before milestone 444: no paths, not an error.
# Absent in an archive older than v20: no paths, not an error.
path_patterns=list(row.get("path_patterns") or []),
# An unknown slug restores UNMAPPED rather than failing: the System and
# its records are the payload, the mapping is an aid.
@@ -2142,7 +2106,7 @@ def _build_rule_usage_event(row: dict, maps: _Maps) -> RuleUsageEvent | None:
"""Resolved through the RULE map, not the note map. A rule id run through
the note map would either drop (best case) or land on whatever note took
that number, producing telemetry that is wrong rather than missing and
that nothing downstream could detect (milestone 333)."""
that nothing downstream could detect."""
rid = maps.rules.get(row.get("rule_id", 0))
if rid is None:
return None
@@ -2217,10 +2181,10 @@ def _build_code_shape(row: dict, maps: _Maps) -> CodeShape | None:
status=status,
snippet_id=snippet_id,
reason=row.get("reason"),
# Exported since #2874 and dropped on the way back in until #4197 —
# every judged row restored with its verdict and without the code for
# WHY, which is the column the accounting reads to tell a scoped-css
# exemption from a convention-plumbing one. Kept on a downgrade for
# Without it every judged row restores with its verdict and without
# the code for WHY, which is the column the accounting reads to tell a
# scoped-css exemption from a convention-plumbing one. Kept on a
# downgrade for
# the same reason `reason` is: the free text and its code are one
# statement, and the existing downgrade clears the judgment's author
# and date, not its argument.
@@ -2230,20 +2194,20 @@ def _build_code_shape(row: dict, maps: _Maps) -> CodeShape | None:
first_seen_commit=row.get("first_seen_commit", ""),
last_seen_commit=row.get("last_seen_commit", ""),
vanished_at=_dt(row["vanished_at"]) if row.get("vanished_at") else None,
# Fingerprints restore; proposals (#2792) deliberately do not — they
# Fingerprints restore; proposals deliberately do not — they
# are mechanical, and the next refresh recomputes them against the
# restored snippet ids.
signature=row.get("signature", ""),
body_sha=row.get("body_sha", ""),
classified_sha=row.get("classified_sha", ""),
# Both exported and both dropped until #4197. `recheck_at` is the
# standing "this judgment asks to be confirmed again" flag, so losing
# `recheck_at` is the standing "this judgment asks to be confirmed
# again" flag, so losing
# it restores a tree that looks settled and is not.
recheck_at=_dt(row["recheck_at"]) if row.get("recheck_at") else None,
# RE-MAPPED, not carried: it is a FK to notes.id like `snippet_id`,
# and a raw source id would point at whatever snippet took that
# number in the destination — wrong rather than missing, which is the
# milestone 333 trap one table over.
# number in the destination — wrong rather than missing, the same
# trap `_build_rule_usage_event` avoids.
diverges_from=maps.notes.get(row.get("diverges_from") or 0),
created_at=_dt(row.get("created_at")),
updated_at=_dt(row.get("updated_at")),
@@ -2279,8 +2243,8 @@ def _build_code_shape_event(row: dict, maps: _Maps) -> CodeShapeEvent | None:
def _build_code_shape_use(row: dict, maps: _Maps) -> CodeShapeUse | None:
"""Consumption edges (#2870) ride their shape AND their snippet — both
ends must have survived, or the edge is no longer a fact."""
"""Consumption edges ride their shape AND their snippet — both ends
must have survived, or the edge is no longer a fact."""
shape_id = maps.shapes.get(row.get("shape_id") or 0)
snippet_id = maps.notes.get(row.get("snippet_id") or 0)
if shape_id is None or snippet_id is None:
@@ -2377,7 +2341,7 @@ async def _restore_v1(data: dict) -> dict:
# Patch the two note->note edges now that every note has a new id.
# Both are ids in the SOURCE database, so writing either straight into
# the constructor would point at whatever record happens to hold that
# number here — a restore that succeeds and silently re-parents (#3182).
# number here — a restore that succeeds and silently re-parents.
# An edge whose target did not survive the import is left NULL rather
# than guessed at.
for n_data in data.get("notes", []):
@@ -2417,7 +2381,7 @@ async def _restore_v2(data: dict) -> dict:
Every model is constructed by a `_build_*` helper above, never inline.
That is what gives the import side a per-table unit the column guard can
hand a stand-in to (#4197); what stays here is the ordering, the flushes
hand a stand-in to; what stays here is the ordering, the flushes
and the id-map bookkeeping, which is the part a split would only make
harder to read.
"""
@@ -2488,8 +2452,8 @@ async def _restore_v2(data: dict) -> dict:
stats["notes"] += 1
# 4b. Patch the note->note edges. Deferred for the same reason
# parent_id always has been: these are ids in the SOURCE database
# (#3182). An edge whose target did not survive stays NULL.
# parent_id is: these are ids in the SOURCE database. An edge whose
# target did not survive stays NULL.
for new_note_id, old_parent_id in notes_with_parents:
new_parent_id = maps.notes.get(old_parent_id)
if new_parent_id:
@@ -2537,8 +2501,8 @@ async def _restore_v2(data: dict) -> dict:
# 8b. Retrieval tuning history (v16) — restored beside the settings it
# explains, and for the same reason: the numbers above are the state,
# these rows are the argument for it. From milestone 416 those dials
# move on the operator's behalf, so an install restored with the values
# these rows are the argument for it. Those dials move on the
# operator's behalf, so an install restored with the values
# and without the reasons is one tuned by nobody it can name.
for t_data in data.get("retrieval_tuning_events", []):
event = _build_retrieval_tuning_event(t_data, maps)
@@ -2596,15 +2560,14 @@ async def _restore_v2(data: dict) -> dict:
# 12-14. Rulebook subscriptions, rule and topic suppressions (v3-v14)
# `rulebook_subscriptions`, `rule_suppressions` and `topic_suppressions`
# are READ BY NOBODY since milestone 414 dropped their tables. An
# archive carrying them still imports, for the reason 14b gives.
# are READ BY NOBODY: their tables do not exist. An archive carrying
# them still imports, for the reason 14b gives.
# 14b. Always-on rulebook exclusions (v10, milestone 297)
# `rulebook_exclusions` was a v10 section and is READ BY NOBODY since
# milestone 394 removed the table. An archive carrying it still
# imports — the key is simply not looked at — because refusing a
# backup for remembering something we deleted would make every v10-v13
# archive unrestorable.
# 14b. Always-on rulebook exclusions (v10)
# `rulebook_exclusions` is a v10 section READ BY NOBODY: its table
# does not exist. An archive carrying it still imports — the key is
# simply not looked at — because refusing a backup for remembering
# something deleted would make every such archive unrestorable.
# --- v5 sections. Every one is data.get()-guarded, so a v2/v3/v4
# payload restores without them rather than failing on an absent key.
@@ -2669,7 +2632,7 @@ async def _restore_v2(data: dict) -> dict:
session.add(relation)
stats["rule_relations"] += 1
# Lesson → rule links (milestone 440): after both notes and rules are
# Lesson → rule links: after both notes and rules are
# mapped, which is why they sit beside the rule edges. Archives before
# v18 carry no section and restore with none.
for lr in data.get("lesson_rule_links", []):
@@ -2678,7 +2641,7 @@ async def _restore_v2(data: dict) -> dict:
continue
session.add(link)
stats["lesson_rule_links"] += 1
# "No rule fits" answers (#4631); archives before v19 carry none.
# "No rule fits" answers; archives before v19 carry none.
for nr in data.get("lesson_no_rule", []):
answer = _build_lesson_no_rule(nr, maps)
if answer is None:
@@ -2746,9 +2709,9 @@ async def _restore_v2(data: dict) -> dict:
maps.decisions[int(fd["id"])] = decision.id
stats["family_decisions"] += 1
# A rule's edit history (milestone 323). Must come after the rules
# themselves — the rule map is only populated above — and both ids are
# ids in the SOURCE database, which is #3182's arose_from_id trap.
# A rule's edit history. Must come after the rules themselves — the
# rule map is only populated above — and both ids are ids in the
# SOURCE database, the same trap as a note's arose_from_id.
for rv in data.get("rule_versions", []):
version = _build_rule_version(rv, maps)
if version is None:
@@ -2834,7 +2797,7 @@ async def _restore_v2(data: dict) -> dict:
session.add(binding)
stats["repo_bindings"] += 1
# 21. Code shapes (v7, #2787)
# 21. Code shapes (v7)
for cs_data in data.get("code_shapes", []):
shape = _build_code_shape(cs_data, maps)
if shape is None:
@@ -2853,7 +2816,7 @@ async def _restore_v2(data: dict) -> dict:
session.add(event)
stats["code_shape_events"] += 1
# v9: consumption edges (#2870).
# v9: consumption edges.
for use in data.get("code_shape_uses", []):
edge = _build_code_shape_use(use, maps)
if edge is None:
@@ -2882,13 +2845,13 @@ async def _restore_v2(data: dict) -> dict:
inception = p_data.get("inception")
if isinstance(inception, dict):
choices = dict(inception.get("choices") or {})
# DROPPED, not remapped (milestone 394). A pre-394 archive
# carries the retired exclusion choice; restoring it would put
# DROPPED, not remapped. An older archive may carry the
# retired exclusion choice; restoring it would put
# a key back that `validate_inception` now rejects as unknown,
# so the next edit to that project would fail on data this
# importer wrote.
choices.pop("exclude_always_on_rulebooks", None)
# Same for subscribe_rulebooks since milestone 414.
# Same for the retired subscribe_rulebooks choice.
choices.pop("subscribe_rulebooks", None)
ds = choices.get("design_system_id")
choices["design_system_id"] = maps.design_systems.get(ds) if ds else None
src/scribe/services/embeddings.py
+2 -2
View File
@@ -224,8 +224,8 @@ async def _apply_supersession_penalty(
stale = await superseded_ids([int(note.id) for _score, note in scored])
except Exception:
# Fail OPEN, and the direction matters: ranking without the penalty is
# the behaviour that shipped for months. Returning nothing, or raising,
# would turn a supersession-lookup hiccup into a broken search.
# a slightly worse search. Returning nothing, or raising, would turn a
# supersession-lookup hiccup into a broken one.
logger.warning("Supersession lookup failed — ranking unpenalised", exc_info=True)
return scored[:limit]
src/scribe/services/shape_ledger.py
+6 -7
View File
@@ -985,13 +985,12 @@ PULL_WINDOW = timedelta(hours=6)
# THE PROBLEM THIS EXISTS FOR. `kind` has exactly two values, so a frozen
# dataclass, a module constant, a sync predicate, a class and an async service
# function are all siblings of one another. Nothing downstream could possibly
# discriminate on a distinction the column does not carry — which is how a
# directory base rate ended up standing alone as the entire divergence
# argument. It was not chosen over a better signal; it was the only signal
# there was. Measured consequence: writing a registry module of pure helpers
# produced five prompts to build them from the `async_session` service canon,
# and on another project one snippet had been recorded as the canon for a
# `class`, a plain `def`, an `async def` and a dozen test functions at once.
# discriminate on a distinction the column does not carry, which leaves a
# directory base rate standing alone as the entire divergence argument. The
# consequence: a registry module of pure helpers gets prompted to build from
# an `async_session` service canon, and one snippet can be recorded as the
# canon for a `class`, a plain `def`, an `async def` and a dozen test
# functions at once.
#
# DERIVED FROM THE SIGNATURE ON READ, not stored. `kind` is part of the row
# identity — ("project_id", "repo_key", "path", "symbol", "kind") — so