Files
FabledGrimoire/src/grimoire/services/record_refs.py
T
bvandeusenandClaude Opus 5.5 38bc25a2b7
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / Python tests (push) Failing after 1m37s
CI & Build / integration (push) Successful in 2m11s
CI & Build / Build & push image (push) Skipped
fix(7736): #N in a body links to the record it names, not to a tag
A purely numeric #N is a record reference everywhere now. The tag
patterns (frontend utils/tags.ts, the editor decoration, the first-line
tag strip, tag autocomplete, backend utils/tags.py) leave digits alone,
so #3127 is no longer drawn as a tag chip linking to /notes?tag=3127.

- utils/recordRefs.ts links #N, milestone N and rule N in rendered
  markdown, with the same patterns and other-numbering word list as
  services/record_refs.py. A drift test reads the TS source and fails
  when the two differ.
- v-record-refs (directives/recordRefs.ts) asks POST
  /api/notes/resolve-refs once per body for what each id is. A link
  gets the route for its kind (task, lesson, snippet, note, milestone,
  rule) and its kind and title on hover. One the reader cannot open is
  drawn as a dead reference, and missing vs not shared read the same.
  Applied on every surface that renders markdown.
- /milestones/:id finds the project and opens its board at that
  milestone (?milestone= expands it, scrolls to it and focuses it).
- Backlinks include records that cite #N in a body or a work log,
  through readable_notes_clause instead of an owner-only filter, and
  route by kind. The graph draws #N citations as dashed reference edges.
- create_note, create_task and add_task_log say which forms link.

There were no numeric tags to clean up: tags are stored only from the
explicit field, and none of the 200 most used is numeric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 14:15:15 -05:00

379 lines
17 KiB
Python

"""Record references written into prose: refusing guessed ids, resolving placeholders, reading the ids a message names, and saying what a body links to.
THE DEFECT THIS EXISTS FOR
A session about to create several records guesses the ids they will get — the
last id it saw, plus one, plus two — and writes those guesses into a plan body
or a reference note before the records exist. The database never collides:
ids come from one Postgres sequence per table, handed out once each. But that
sequence is shared by every session and every user, so any concurrent create
takes a guessed number, and the reference silently points at someone else's
record. More users make it worse, never better.
Two halves, and they are not alternatives:
- `{{ref:N}}` PLACEHOLDERS (services/record_batch.py) remove the reason to
guess. A batch create inserts its records, receives their real ids and
rewrites the placeholders inside ONE transaction, so there is no moment at
which an id has to be predicted.
- REFUSING A GUESSED ID is the backstop for a session that guesses anyway. It
is a refusal, not a warning, because a warning is an instruction and
instructions are not always followed.
HOW A GUESS IS RECOGNISED
A guess is always just ABOVE the highest id that exists — that is the only
place "the next id" can be. So a `#N` is refused when it sits in
(max_id, max_id + GUESS_WINDOW]. Numbers below the max are real records (or
were), and numbers far above it are not Grimoire ids at all — a PR, a forge
issue — so both pass. What this cannot catch, by construction: a guess that
another session has ALREADY used. That reference points at a real record, and
nothing in the text distinguishes it from a deliberate one. The placeholders
are the fix for that case; this is not.
WHERE IT IS ENFORCED
At the MCP door (the create/update tools), not in the services. The defect is
an agent predicting ids; a person typing `#123` in the web editor has read that
number off a record. Enforcing it in `notes.create_note` would also reach
restores and internal writers that legitimately carry numbers from elsewhere.
"""
from __future__ import annotations
import re
from collections.abc import Iterable
from sqlalchemy import func, select
from grimoire.models import async_session
from grimoire.models.note import Note
# `#123` as a standalone token. The lookbehind keeps out `&#123;` entities,
# URL fragments (`page#12`) and doubled `##`; `\b` after the digits keeps out a
# hex colour like `#123abc`.
REF_RE = re.compile(r"(?<![\w&#/])#(\d+)\b")
# How far above the highest existing id a number still reads as a guess. It
# only has to cover how far ahead a session predicts, which is the size of the
# batch it is planning — a plan's steps, a handful of cross-linked notes. 50 is
# generous for that and still narrow enough that an unrelated number (a PR, an
# issue on a forge) landing inside it is rare. Widening it trades exactly that.
GUESS_WINDOW = 50
# `{{ref:3}}` names the third record of the same batch; `{{ref:milestone}}` names
# the milestone a start_planning call creates alongside its steps.
PLACEHOLDER_RE = re.compile(r"\{\{\s*ref:\s*(\d+|milestone)\s*\}\}")
def cited_ids(*texts: str | None) -> set[int]:
"""Every `#N` written in these texts."""
found: set[int] = set()
for text in texts:
if text:
found.update(int(m) for m in REF_RE.findall(text))
return found
async def _max_note_id() -> int:
# Across ALL users and including trashed rows: the sequence is global, so
# "the next id" is global too. Only the number is read, never a row.
async with async_session() as session:
return (await session.execute(select(func.max(Note.id)))).scalar() or 0
async def guessed_ids(*texts: str | None) -> list[int]:
"""The `#N` references in these texts that can only be predictions."""
cited = cited_ids(*texts)
if not cited:
return []
ceiling = await _max_note_id()
return sorted(n for n in cited if ceiling < n <= ceiling + GUESS_WINDOW)
async def refuse_guessed_ids(*texts: str | None) -> None:
"""Raise if any text cites a note/task id that has not been assigned yet.
Raised before anything is written, so the caller's create or update does
not happen. The message names the fix, because the agent reading it is
mid-turn and can apply it immediately.
"""
guessed = await guessed_ids(*texts)
if not guessed:
return
listed = ", ".join(f"#{n}" for n in guessed)
raise ValueError(
f"{listed} does not exist yet — it reads as a guessed id, and a guess "
"is taken by whatever any other session creates next. Nothing was "
"written. An id exists only once a create call RETURNS it: create the "
"records first and cite the ids they come back with, or create them "
"together with create_records / start_planning(steps=...) and write "
"{{ref:N}} where the Nth record's id belongs. If this number is not a "
"Grimoire record (a PR, an issue elsewhere), write it without the '#'."
)
def placeholder_keys(*texts: str | None) -> set[str]:
"""Every `{{ref:...}}` key in these texts: "1", "2", …, "milestone"."""
keys: set[str] = set()
for text in texts:
if text:
keys.update(PLACEHOLDER_RE.findall(text))
return keys
def resolve_placeholders(text: str | None, refs: dict[str, str]) -> str | None:
"""Rewrite `{{ref:...}}` using `refs` (key -> rendered reference).
Callers validate the keys first; an unknown key here is a bug in the
caller, so it raises rather than leaving a placeholder in a stored record.
"""
if not text:
return text
return PLACEHOLDER_RE.sub(lambda m: refs[m.group(1)], text)
# ── Records an operator names in a message ─────────────────────────────────
#
# "yes go ahead with 4448" says exactly which record is meant, and a ranked
# menu cannot use it: a number carries no meaning to an embedding, so the menu
# fills with whatever resembles the words AROUND it. These ids are looked up
# directly instead (plugin_context.build_autoinject_hint).
#
# A guard against a false READ, not a false lookup. A number that is not a
# record id still has to resolve to a record this user can open before anything
# is shown, and the line says it was found by number — but every wrong id is a
# record dropped in front of the reader for no reason, so the parser takes a
# number only where the sentence makes it a reference.
# How many named records one message can bring in. A real message names one or
# two; past a handful the numbers are a pasted list or a log, not a reference.
NAMED_LIMIT = 5
# A number, with its `#` when it has one. The lookbehind keeps out a number
# inside a word, a path or URL (`pulls/198`), a version or date (`1.2`,
# `2026-10-03`), money and entities; the lookahead keeps out the same on the
# other side, a unit glued on (`300ms`, `40%`) and a thousands separator.
_NAMED_RE = re.compile(
r"(?<![\w#&$€£.:/\\=@~-])(#?)(\d{1,7})(?![\w%]|[.:/-]\d|,\d{3}(?!\d))"
)
# Words after which `#N` belongs to a different numbering: a forge's (a PR, a
# CI run, a job) or one of Grimoire's own id spaces that is not a note's. Rules,
# milestones, Systems, projects and work-logs each draw from their own
# sequence, so "rule #153" is not note 153 — and note 153 is somebody's record.
_OTHER_SPACE = frozenset({
"pr", "prs", "pull", "pulls", "request", "requests", "mr", "mrs",
"run", "runs", "ci", "build", "builds", "job", "jobs", "pipeline",
"action", "actions", "commit", "github", "gitlab", "gitea", "gh",
"rule", "rules", "preference", "preferences", "milestone", "milestones",
"system", "systems", "project", "projects", "log", "logs", "topic",
"topics", "rulebook", "rulebooks", "step", "steps", "rank", "line",
"lines", "page", "item", "option", "version", "port",
})
# Words after which a BARE number is a reference. Bare numbers are most of what
# a message holds — sizes, counts, durations — so a bare one is taken only
# after a word that points at something, at the very start of the message, or
# continuing a list that started as a reference.
_REF_WORDS = frozenset({
"with", "for", "on", "at", "about", "re", "regarding", "into", "up",
"task", "tasks", "issue", "issues", "note", "notes", "record", "records",
"snippet", "snippets", "lesson", "lessons", "spike", "spikes", "process",
"continue", "resume", "start", "reopen", "open", "close", "finish",
"see", "check", "read", "do", "id", "ids",
})
# A word after a bare number that makes it a quantity: "for 300 seconds",
# "about 500 records". Not applied after `#`, which is never a quantity.
_UNITS = frozenset({
"s", "sec", "secs", "second", "seconds", "ms", "min", "mins", "minute",
"minutes", "h", "hr", "hrs", "hour", "hours", "day", "days", "week",
"weeks", "month", "months", "year", "years", "char", "chars", "character",
"characters", "token", "tokens", "line", "lines", "word", "words", "byte",
"bytes", "kb", "mb", "gb", "px", "row", "rows", "record", "records",
"note", "notes", "task", "tasks", "item", "items", "file", "files",
"test", "tests", "call", "calls", "time", "times", "x", "percent",
"result", "results", "hit", "hits", "query", "queries", "message",
"messages", "prompt", "prompts", "step", "steps", "point", "points",
})
_FENCE_RE = re.compile(r"```.*?(?:```|\Z)", re.S)
_PREV_WORD_RE = re.compile(r"([A-Za-z]+)[\s\"'(\[*_:]*\Z")
_NEXT_WORD_RE = re.compile(r"\s*([A-Za-z]+)")
_OPENS_RE = re.compile(r"[\s\"'(\[*_>-]*")
_LIST_JOIN_RE = re.compile(r"\s*(?:,|,?\s*(?:and|or|&|\+|/))\s*", re.I)
def _prev_word(before: str) -> str:
"""The word right before a number, past a space, a quote or a bracket."""
m = _PREV_WORD_RE.search(before)
return m.group(1).lower() if m else ""
# ── References written into a record's prose ───────────────────────────────
#
# What a body, a work log or a plan names, read the way the web UI links it.
# `frontend/src/utils/recordRefs.ts` holds the same patterns and the same word
# list, and tests/test_record_ref_links.py reads that file and fails when its
# patterns or words differ from these.
#
# Three id spaces, three forms:
# `#N` a note, task, lesson or snippet (one sequence, the notes table)
# `milestone N` a milestone — what `{{ref:milestone}}` renders as
# `rule N` a rule; `preference N` too, since preferences are rules
# A `#N` after a word that names another numbering ("PR #198", "rule #34",
# "step #3") is not a note: `_OTHER_SPACE` above decides, exactly as it does for
# the numbers an operator names in a message.
PROSE_REF_RE = re.compile(r"(?<![\w/])\b(milestone|rule|preference)\s+#?(\d+)\b", re.IGNORECASE)
# Fenced and inline code: a pasted log or a snippet of source names nothing.
_CODE_RE = re.compile(r"```[\s\S]*?```|`[^`\n]+`")
def referenced(*texts: str | None) -> dict[str, set[int]]:
"""Every record these texts link to, by id space: notes, milestones, rules."""
out: dict[str, set[int]] = {"notes": set(), "milestones": set(), "rules": set()}
for text in texts:
if not text:
continue
text = _CODE_RE.sub(" ", text)
for m in PROSE_REF_RE.finditer(text):
space = "milestones" if m.group(1).lower() == "milestone" else "rules"
out[space].add(int(m.group(2)))
for m in REF_RE.finditer(text):
if _prev_word(text[:m.start()]) not in _OTHER_SPACE:
out["notes"].add(int(m.group(1)))
return out
# How many references one resolve call answers. A long plan cites a few dozen;
# past this the caller is listing, not rendering, and gets a 400 at the door.
RESOLVE_LIMIT = 200
async def resolve_refs(
user_id: int,
*,
notes: Iterable[int] = (),
milestones: Iterable[int] = (),
rules: Iterable[int] = (),
) -> dict[str, dict[int, dict]]:
"""What each referenced id IS, for the records this user may read.
An id that is missing, trashed or not readable is simply absent, and the
caller shows it as a reference that leads nowhere. The answer never says
which of the three it was: "not shared with you" and "does not exist" must
read the same, or the resolver becomes a way to probe other people's ids.
No pull is recorded. Hovering a link is not choosing to read the record;
the view it opens records that, the same as any other way of arriving.
"""
from grimoire.models.milestone import Milestone
from grimoire.services import rulebooks as rulebooks_svc
from grimoire.services.access import can_read_project, readable_notes_clause
out: dict[str, dict[int, dict]] = {"notes": {}, "milestones": {}, "rules": {}}
note_ids = sorted({int(i) for i in notes})
milestone_ids = sorted({int(i) for i in milestones})
rule_ids = sorted({int(i) for i in rules})
async with async_session() as session:
if note_ids:
rows = (await session.execute(
select(Note).where(
Note.id.in_(note_ids),
Note.deleted_at.is_(None),
readable_notes_clause(user_id),
)
)).scalars().all()
for n in rows:
out["notes"][n.id] = {
"id": n.id,
"title": n.title or "",
"is_task": n.is_task,
"note_type": n.note_type or "note",
"status": n.status,
"task_kind": n.task_kind if n.is_task else None,
}
if milestone_ids:
rows = (await session.execute(
select(Milestone).where(
Milestone.id.in_(milestone_ids), Milestone.deleted_at.is_(None),
)
)).scalars().all()
readable_projects: dict[int, bool] = {}
for ms in rows:
if ms.user_id != user_id:
pid = ms.project_id
if pid is None:
continue
if pid not in readable_projects:
readable_projects[pid] = await can_read_project(user_id, pid)
if not readable_projects[pid]:
continue
out["milestones"][ms.id] = {
"id": ms.id,
"title": ms.title or "",
"project_id": ms.project_id,
"status": ms.status,
}
# One lookup per rule: readability runs through rulebook ownership and
# project ownership, which `get_rule` already resolves. A body names a few.
for rid in rule_ids:
rule = await rulebooks_svc.get_rule(rid, user_id)
if rule is not None:
out["rules"][rule.id] = {
"id": rule.id,
"title": rule.title or "",
"kind": getattr(rule, "kind", None) or "rule",
}
return out
def _next_word(after: str) -> str:
m = _NEXT_WORD_RE.match(after)
return m.group(1).lower() if m else ""
def named_record_ids(prompt: str | None, limit: int = NAMED_LIMIT) -> list[int]:
"""The record ids a message names, in the order it names them.
`#4448` unless the word before it is another numbering ("PR #198"); a bare
number of three or more digits when it opens the message ("4417 sounds
right"), follows a word that points at something ("go ahead with 4448") or
continues a list one of those started ("4448, 4449 and 4450"), and is not
a quantity ("for 300 seconds"). Fenced code is skipped: a pasted log is
full of numbers and names nothing.
Read the operator's OWN words, never a query enriched with the assistant's
reply — a number the agent wrote is not one the operator named.
"""
text = _FENCE_RE.sub("\n", prompt or "")
found: list[int] = []
last_end, last_ok = -1, False
for m in _NAMED_RE.finditer(text):
hashed, digits = bool(m.group(1)), m.group(2)
before, after = text[:m.start()], text[m.end():]
# A list inherits its head's reading, either way: "PR #198 and #199"
# is two PRs, "with 4448, 4449" is two records.
if last_end >= 0 and _LIST_JOIN_RE.fullmatch(text[last_end:m.start()]):
ok = last_ok
elif hashed:
ok = _prev_word(before) not in _OTHER_SPACE
else:
ok = (bool(_OPENS_RE.fullmatch(before))
or _prev_word(before) in _REF_WORDS)
last_end, last_ok = m.end(), ok
if not ok or digits[0] == "0" or len(digits) < (2 if hashed else 3):
continue
if not hashed and _next_word(after) in _UNITS:
continue
nid = int(digits)
if nid not in found:
found.append(nid)
if len(found) >= limit:
break
return found