rename(547): the server is grimoire — package, image, DB defaults, backups

Step 2 of the Scribe -> Grimoire rename.

- src/scribe -> src/grimoire, and every import, patch target, module path,
  hypercorn entry point and `python -m grimoire.restore` with it; pyproject,
  uv.lock and the frontend package are named grimoire.
- Image git.fabledsword.com/bvandeusen/fabledgrimoire; CI's Postgres service
  runs as grimoire / grimoire_test.
- Compose files default the DB user, database and (prod) volume to grimoire,
  each overridable from .env, so an install created as Scribe keeps its data
  by setting POSTGRES_USER, POSTGRES_DB and PGDATA_VOLUME.
- Backups are written as grimoire-<ts>.tar with format grimoire-backup-tar.
  Archives named scribe-* with the scribe-backup-tar manifest still restore
  and are still counted at the destination (folder and S3), with tests.
- Exports, the MCP auth realm, the ASGI scope key and the persistent logger
  carry the new name.

Kept on purpose: the secret-sealing key label stays "scribe.secrets.v1",
because every stored secret was derived under it.

Plugin identity (plugin name, hooks, env vars, .scribe marker, skills) is
step 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/grimoire/services/record_refs.py

@@ -0,0 +1,255 @@
+"""Record references written into prose: refusing guessed ids, resolving placeholders, reading the ids a message names.
+
+THE DEFECT THIS EXISTS FOR
+
+A session about to create several records guesses the ids they will get — the
+last id it saw, plus one, plus two — and writes those guesses into a plan body
+or a reference note before the records exist. The database never collides:
+ids come from one Postgres sequence per table, handed out once each. But that
+sequence is shared by every session and every user, so any concurrent create
+takes a guessed number, and the reference silently points at someone else's
+record. More users make it worse, never better.
+
+Two halves, and they are not alternatives:
+
+- `{{ref:N}}` PLACEHOLDERS (services/record_batch.py) remove the reason to
+  guess. A batch create inserts its records, receives their real ids and
+  rewrites the placeholders inside ONE transaction, so there is no moment at
+  which an id has to be predicted.
+- REFUSING A GUESSED ID is the backstop for a session that guesses anyway. It
+  is a refusal, not a warning, because a warning is an instruction and
+  instructions are not always followed.
+
+HOW A GUESS IS RECOGNISED
+
+A guess is always just ABOVE the highest id that exists — that is the only
+place "the next id" can be. So a `#N` is refused when it sits in
+(max_id, max_id + GUESS_WINDOW]. Numbers below the max are real records (or
+were), and numbers far above it are not Grimoire ids at all — a PR, a forge
+issue — so both pass. What this cannot catch, by construction: a guess that
+another session has ALREADY used. That reference points at a real record, and
+nothing in the text distinguishes it from a deliberate one. The placeholders
+are the fix for that case; this is not.
+
+WHERE IT IS ENFORCED
+
+At the MCP door (the create/update tools), not in the services. The defect is
+an agent predicting ids; a person typing `#123` in the web editor has read that
+number off a record. Enforcing it in `notes.create_note` would also reach
+restores and internal writers that legitimately carry numbers from elsewhere.
+"""
+from __future__ import annotations
+
+import re
+
+from sqlalchemy import func, select
+
+from grimoire.models import async_session
+from grimoire.models.note import Note
+
+# `#123` as a standalone token. The lookbehind keeps out `{` entities,
+# URL fragments (`page#12`) and doubled `##`; `\b` after the digits keeps out a
+# hex colour like `#123abc`.
+REF_RE = re.compile(r"(?<![\w&#/])#(\d+)\b")
+
+# How far above the highest existing id a number still reads as a guess. It
+# only has to cover how far ahead a session predicts, which is the size of the
+# batch it is planning — a plan's steps, a handful of cross-linked notes. 50 is
+# generous for that and still narrow enough that an unrelated number (a PR, an
+# issue on a forge) landing inside it is rare. Widening it trades exactly that.
+GUESS_WINDOW = 50
+
+# `{{ref:3}}` names the third record of the same batch; `{{ref:milestone}}` names
+# the milestone a start_planning call creates alongside its steps.
+PLACEHOLDER_RE = re.compile(r"\{\{\s*ref:\s*(\d+|milestone)\s*\}\}")
+
+
+def cited_ids(*texts: str | None) -> set[int]:
+    """Every `#N` written in these texts."""
+    found: set[int] = set()
+    for text in texts:
+        if text:
+            found.update(int(m) for m in REF_RE.findall(text))
+    return found
+
+
+async def _max_note_id() -> int:
+    # Across ALL users and including trashed rows: the sequence is global, so
+    # "the next id" is global too. Only the number is read, never a row.
+    async with async_session() as session:
+        return (await session.execute(select(func.max(Note.id)))).scalar() or 0
+
+
+async def guessed_ids(*texts: str | None) -> list[int]:
+    """The `#N` references in these texts that can only be predictions."""
+    cited = cited_ids(*texts)
+    if not cited:
+        return []
+    ceiling = await _max_note_id()
+    return sorted(n for n in cited if ceiling < n <= ceiling + GUESS_WINDOW)
+
+
+async def refuse_guessed_ids(*texts: str | None) -> None:
+    """Raise if any text cites a note/task id that has not been assigned yet.
+
+    Raised before anything is written, so the caller's create or update does
+    not happen. The message names the fix, because the agent reading it is
+    mid-turn and can apply it immediately.
+    """
+    guessed = await guessed_ids(*texts)
+    if not guessed:
+        return
+    listed = ", ".join(f"#{n}" for n in guessed)
+    raise ValueError(
+        f"{listed} does not exist yet — it reads as a guessed id, and a guess "
+        "is taken by whatever any other session creates next. Nothing was "
+        "written. An id exists only once a create call RETURNS it: create the "
+        "records first and cite the ids they come back with, or create them "
+        "together with create_records / start_planning(steps=...) and write "
+        "{{ref:N}} where the Nth record's id belongs. If this number is not a "
+        "Grimoire record (a PR, an issue elsewhere), write it without the '#'."
+    )
+
+
+def placeholder_keys(*texts: str | None) -> set[str]:
+    """Every `{{ref:...}}` key in these texts: "1", "2", …, "milestone"."""
+    keys: set[str] = set()
+    for text in texts:
+        if text:
+            keys.update(PLACEHOLDER_RE.findall(text))
+    return keys
+
+
+def resolve_placeholders(text: str | None, refs: dict[str, str]) -> str | None:
+    """Rewrite `{{ref:...}}` using `refs` (key -> rendered reference).
+
+    Callers validate the keys first; an unknown key here is a bug in the
+    caller, so it raises rather than leaving a placeholder in a stored record.
+    """
+    if not text:
+        return text
+    return PLACEHOLDER_RE.sub(lambda m: refs[m.group(1)], text)
+
+
+# ── Records an operator names in a message ─────────────────────────────────
+#
+# "yes go ahead with 4448" says exactly which record is meant, and a ranked
+# menu cannot use it: a number carries no meaning to an embedding, so the menu
+# fills with whatever resembles the words AROUND it. These ids are looked up
+# directly instead (plugin_context.build_autoinject_hint).
+#
+# A guard against a false READ, not a false lookup. A number that is not a
+# record id still has to resolve to a record this user can open before anything
+# is shown, and the line says it was found by number — but every wrong id is a
+# record dropped in front of the reader for no reason, so the parser takes a
+# number only where the sentence makes it a reference.
+
+# How many named records one message can bring in. A real message names one or
+# two; past a handful the numbers are a pasted list or a log, not a reference.
+NAMED_LIMIT = 5
+
+# A number, with its `#` when it has one. The lookbehind keeps out a number
+# inside a word, a path or URL (`pulls/198`), a version or date (`1.2`,
+# `2026-10-03`), money and entities; the lookahead keeps out the same on the
+# other side, a unit glued on (`300ms`, `40%`) and a thousands separator.
+_NAMED_RE = re.compile(
+    r"(?<![\w#&$€£.:/\\=@~-])(#?)(\d{1,7})(?![\w%]|[.:/-]\d|,\d{3}(?!\d))"
+)
+
+# Words after which `#N` belongs to a different numbering: a forge's (a PR, a
+# CI run, a job) or one of Grimoire's own id spaces that is not a note's. Rules,
+# milestones, Systems, projects and work-logs each draw from their own
+# sequence, so "rule #153" is not note 153 — and note 153 is somebody's record.
+_OTHER_SPACE = frozenset({
+    "pr", "prs", "pull", "pulls", "request", "requests", "mr", "mrs",
+    "run", "runs", "ci", "build", "builds", "job", "jobs", "pipeline",
+    "action", "actions", "commit", "github", "gitlab", "gitea", "gh",
+    "rule", "rules", "preference", "preferences", "milestone", "milestones",
+    "system", "systems", "project", "projects", "log", "logs", "topic",
+    "topics", "rulebook", "rulebooks", "step", "steps", "rank", "line",
+    "lines", "page", "item", "option", "version", "port",
+})
+
+# Words after which a BARE number is a reference. Bare numbers are most of what
+# a message holds — sizes, counts, durations — so a bare one is taken only
+# after a word that points at something, at the very start of the message, or
+# continuing a list that started as a reference.
+_REF_WORDS = frozenset({
+    "with", "for", "on", "at", "about", "re", "regarding", "into", "up",
+    "task", "tasks", "issue", "issues", "note", "notes", "record", "records",
+    "snippet", "snippets", "lesson", "lessons", "spike", "spikes", "process",
+    "continue", "resume", "start", "reopen", "open", "close", "finish",
+    "see", "check", "read", "do", "id", "ids",
+})
+
+# A word after a bare number that makes it a quantity: "for 300 seconds",
+# "about 500 records". Not applied after `#`, which is never a quantity.
+_UNITS = frozenset({
+    "s", "sec", "secs", "second", "seconds", "ms", "min", "mins", "minute",
+    "minutes", "h", "hr", "hrs", "hour", "hours", "day", "days", "week",
+    "weeks", "month", "months", "year", "years", "char", "chars", "character",
+    "characters", "token", "tokens", "line", "lines", "word", "words", "byte",
+    "bytes", "kb", "mb", "gb", "px", "row", "rows", "record", "records",
+    "note", "notes", "task", "tasks", "item", "items", "file", "files",
+    "test", "tests", "call", "calls", "time", "times", "x", "percent",
+    "result", "results", "hit", "hits", "query", "queries", "message",
+    "messages", "prompt", "prompts", "step", "steps", "point", "points",
+})
+
+_FENCE_RE = re.compile(r"```.*?(?:```|\Z)", re.S)
+_PREV_WORD_RE = re.compile(r"([A-Za-z]+)[\s\"'(\[*_:]*\Z")
+_NEXT_WORD_RE = re.compile(r"\s*([A-Za-z]+)")
+_OPENS_RE = re.compile(r"[\s\"'(\[*_>-]*")
+_LIST_JOIN_RE = re.compile(r"\s*(?:,|,?\s*(?:and|or|&|\+|/))\s*", re.I)
+
+
+def _prev_word(before: str) -> str:
+    """The word right before a number, past a space, a quote or a bracket."""
+    m = _PREV_WORD_RE.search(before)
+    return m.group(1).lower() if m else ""
+
+
+def _next_word(after: str) -> str:
+    m = _NEXT_WORD_RE.match(after)
+    return m.group(1).lower() if m else ""
+
+
+def named_record_ids(prompt: str | None, limit: int = NAMED_LIMIT) -> list[int]:
+    """The record ids a message names, in the order it names them.
+
+    `#4448` unless the word before it is another numbering ("PR #198"); a bare
+    number of three or more digits when it opens the message ("4417 sounds
+    right"), follows a word that points at something ("go ahead with 4448") or
+    continues a list one of those started ("4448, 4449 and 4450"), and is not
+    a quantity ("for 300 seconds"). Fenced code is skipped: a pasted log is
+    full of numbers and names nothing.
+
+    Read the operator's OWN words, never a query enriched with the assistant's
+    reply — a number the agent wrote is not one the operator named.
+    """
+    text = _FENCE_RE.sub("\n", prompt or "")
+    found: list[int] = []
+    last_end, last_ok = -1, False
+    for m in _NAMED_RE.finditer(text):
+        hashed, digits = bool(m.group(1)), m.group(2)
+        before, after = text[:m.start()], text[m.end():]
+        # A list inherits its head's reading, either way: "PR #198 and #199"
+        # is two PRs, "with 4448, 4449" is two records.
+        if last_end >= 0 and _LIST_JOIN_RE.fullmatch(text[last_end:m.start()]):
+            ok = last_ok
+        elif hashed:
+            ok = _prev_word(before) not in _OTHER_SPACE
+        else:
+            ok = (bool(_OPENS_RE.fullmatch(before))
+                  or _prev_word(before) in _REF_WORDS)
+        last_end, last_ok = m.end(), ok
+        if not ok or digits[0] == "0" or len(digits) < (2 if hashed else 3):
+            continue
+        if not hashed and _next_word(after) in _UNITS:
+            continue
+        nid = int(digits)
+        if nid not in found:
+            found.append(nid)
+            if len(found) >= limit:
+                break
+    return found