Restore rolls back to a backup; a session can only file issues in another project #220

Merged

bvandeusen merged 15 commits from dev into main 2026-11-05 09:29:55 -05:00

Conversation 0 Commits 15 Files Changed 52

+3853 -729

bvandeusen commented 2026-11-05 09:29:49 -05:00

Owner

Copy Link

Copy Source

Merges dev into main. Two milestones.

Milestone 582, restore rolls the instance back to the backup:

• Settings restore, POST /api/admin/restore and python -m grimoire.restore all roll a live instance back to one of its own backups, after saving the current state first.
• The confirm says what will be lost.

Milestone 583, a session can only file issues in a project other than its own:

• A new plugin hook (grimoire_project_fence.sh), backed by GET /api/plugin/write-scope, asks the operator before a session writes to another project. That covers editing its files, committing or pushing in its repo, and changing its Grimoire records.
• Reading, and filing an issue there, pass without a prompt.
• Plugin version: 2026.11.05.0301.

Also included: the ScrollSentinel adoption (#6839).

CI is green on dev head 49b5cee (run 10238).

🤖 Generated with Claude Code

Merges dev into main. Two milestones. **Milestone 582, restore rolls the instance back to the backup:** - Settings restore, `POST /api/admin/restore` and `python -m grimoire.restore` all roll a live instance back to one of its own backups, after saving the current state first. - The confirm says what will be lost. **Milestone 583, a session can only file issues in a project other than its own:** - A new plugin hook (`grimoire_project_fence.sh`), backed by `GET /api/plugin/write-scope`, asks the operator before a session writes to another project. That covers editing its files, committing or pushing in its repo, and changing its Grimoire records. - Reading, and filing an issue there, pass without a prompt. - Plugin version: 2026.11.05.0301. Also included: the ScrollSentinel adoption (#6839). CI is green on dev head 49b5cee (run 10238). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 15 commits 2026-11-05 09:29:49 -05:00

feat(6838): read the client address and HTTPS from the trusted proxy hop ... dbd6d5b8b5

grimoire/proxy.py is the one reader of X-Forwarded-For and X-Forwarded-Proto.
It takes the entry TRUSTED_PROXY_HOPS in from the right (default 1; a shorter
header reads its leftmost entry; 0 trusts no proxy), so an entry a client puts
left of the proxy's is never believed. Before this, with TRUST_PROXY_HEADERS
on, the leftmost X-Forwarded-For entry keyed the login, register, forgot and
reset rate limits and the audit log, and a client could choose it.

The session cookie is Secure, and HSTS (max-age=31536000) is sent, exactly
when the request is HTTPS by the trusted proxy's word. Plain HTTP is
untouched and nothing redirects. SECURE_COOKIES is gone (a startup warning says
so when it is still set); TRUST_PROXY_HEADERS=false still maps to 0 hops, with
a warning to move to TRUSTED_PROXY_HOPS. Admin audit lines now use the same
client address instead of the socket peer.

The placeholder-key refusal keyed on SECURE_COOKIES, so it now refuses either
placeholder key outright; the instance makes its own key when none is set.

Family idea #5004 (IsHTTPS, trusted-hop X-Forwarded-Proto), canon v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(6838): give the test request its socket peer the way Quart reads it ... 3c01e17559

Quart takes remote_addr from a Remote-Addr header its ASGI layer writes from
the scope (replacing any the client sent). A test request context does not
read scope_base for it, so the two socket-peer tests saw an empty address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6845): every instance has an id lineage, and every full backup carries it ... 6d8db0bb65

instance_identity is one row made by migration 0127 with a random id, so an
instance has it before its first account. A full backup carries it in its
header as instance_id (the tar manifest header and the JSON top level);
BACKUP_VERSION is 28. The table rides in _NOT_INCLUDED: a restore decides from
the header value rather than copying a row.

Two stores with the same id number their rows the same way. That is what lets
a later step roll a live instance back by original id, refuse a backup whose
ids would land on unrelated rows, and have a fresh-instance restore adopt the
backup's id (adopt_instance_id).

Milestone 582, step 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6846): replace_from_backup rolls the store back to a full backup by id ... 8806ab85b4

One transaction:
- Refuse unless the backup is a full one and, on a live instance, carries
  this instance's id. A fresh instance (no records, at most the restoring
  account) takes any full backup and adopts its id.
- Clear first: search vectors and the CSS consumer map (derived); every
  backed-up table nothing outside the backup points at, whole; the settings
  the backup carries; and the identity-table rows the backup lacks. A share
  granted by a removed user goes before the user.
- Write the backup through the same _build_* helpers as the copy-in restore,
  now _load_v2(keep_ids=...). Identity tables (users, projects, milestones,
  notes, rulebooks, topics, rules, systems, design systems, code shapes,
  family decisions) go through _place. That keeps the backup's id and
  merges, so API keys, shares and forge connections of a surviving row stay.
- Reset id sequences past the restored max, and sign every session out.

reindex_after_restore runs the four embedding backfills afterwards. A unit
guard checks that no row kept for outside references cascades from a table
replaced whole. Integration tests cover edits, deletions and new rows put
back, a surviving API key, sessions ending, fresh ids afterwards, foreign or
id-less backups refused, a user export refused, a failure part-way changing
nothing, and a fresh instance adopting the id.

Milestone 582, step 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6847): a rollback first backs up the state it replaces ... 34f8d7060b

backup_job.rollback_to_backup:
1. Checks the backup can apply here (backup.check_replace, which shares
   _replace_mode with replace_from_backup), refusing before anything is written.
2. Takes the backup lock, so no scheduled backup runs during the rollback.
3. Saves the current store with save_state_before_restore.
4. Replaces the store.

The safety file goes to the backup destination when backups are on and it
takes it. Otherwise it goes to DATA_DIR/pre-restore, with a note when the
destination refused. If neither takes it the rollback does not run. The
result names where the file went.

Milestone 582, step 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6848): Settings restore rolls back, from the destination or a file, after a preview ... 1f5e963c35

The web restore no longer copies a backup in beside the live data.

- GET /api/admin/restore/sources lists the backups at the backup destination,
  newest first.
- POST /api/admin/restore/preview says what a restore would do, changing
  nothing: fresh or rollback, when the backup was taken, and counts now and
  after per kind. A backup that cannot apply here gets a 409 with the reason.
- POST /api/admin/restore?confirm=rollback runs backup_job.rollback_to_backup:
  safety backup, replace, then search rebuilt in the background.

Each takes ?source=<name> (a backup at the destination) or an upload. The
destination path exists because uploads are held to the request-size limit,
and a real instance's backup is far over it. backup_s3.get_file downloads
from a bucket. backup_destination.list_backups/open_backup only accept names
the scheduled backup writes, so a request cannot reach any other file.

Settings -> Data -> Restore from Backup opens RestoreBackupDialog. The dialog:
1. Pick a backup at the destination or a file.
2. Read the preview.
3. Type "roll back".
4. Run it. The dialog then says where the replaced state was saved, and sends
   the user to sign in again.

Closes #6840. Milestone 582, step 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6849): the command-line restore uses the same rollback, and the docs say how restore works now ... 37be3b0bff

python -m grimoire.restore <file> prints what the restore would change (the
same preview as Settings) and stops. With --yes it runs
backup_job.rollback_to_backup: the current state is saved first, then the
whole instance goes back to the backup. An empty instance takes any full
backup. A live one only takes a backup of its own, and otherwise exits 3 with
the reason. A v1 backup keeps the copy-in path into an empty instance.

docs/backups.md Restoring is rewritten:
- what a restore puts back and removes;
- the same-instance rule;
- the safety backup;
- what survives;
- the web and command-line ways in.

Its S3 key advice now asks for read as well as write when restoring from the
bucket. README, features, architecture and the API reference follow.

Milestone 582, step 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(6849): the restore command previews without --yes and rolls back from the tar with it ... 846c87629f

Two integration tests in test_integration_backup_rollback.py drive
grimoire.restore.run on the scheduled backup's tar, written from a full
export of this instance:
- without --yes it exits 4 and nothing changes;
- with --yes it takes the safety backup, then puts back the edited and the
  deleted task and removes the new one.

The file-restore test that expected a refusal "because the database has
accounts" now names the reason it actually refuses: on a live instance, a
backup that records no instance id can't be rolled back to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6839): the decision log and the knowledge feed load as you scroll, through one ScrollSentinel ... 373f5bd454

ScrollSentinel sits at the foot of a server-paged list and asks for the
next page a screen before the reader reaches it. It asks once per cursor,
forgets the cursor when its element goes, and shows Try again on a failed
page instead of retrying in a loop.

FamilyView's decision log loses its "Older decisions" button. A failed
page shows in place rather than as a toast, and rows already shown are
skipped when a new decision shifts the offset.

KnowledgeView drops its inline IntersectionObserver for the component.
The cursor is the ids consumed, not the items rendered, so a batch that
returns fewer records than it asked for still moves the feed on. A failed
batch puts its ids back and offers Try again, where it used to fail
silently and stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6854): the server says which project a pending act would change, and whether this session may do it ... 55f0f96016

GET /api/plugin/write-scope answers the plugin's fence. The session's project
comes from repo / project_id, as every plugin endpoint reads it. The act is
either:
- a directory (target_repo / target_project_id), for a file edit or a git
  command; or
- an MCP call (tool, plus input: the call's short arguments).

It returns allowed, the other projects the act would change, and the reason
the operator is asked. It is a GET so a read-scoped key still fences file
edits and git.

services/write_scope.WRITE_TARGETS lists every write tool with the arguments
that locate its write: a project id, or a task, note, milestone, System, rule,
relation, trash batch, repo or family decision, each traced to its project. A
write that lands in no project (install settings, user-level records, the
judgments a session is asked for anywhere) has an empty entry. The one
cross-project write allowed is create_task(kind="issue") as a new, unplanned,
open issue. A project the caller cannot read is never named.

tests/test_write_scope.py:
- requires an entry for every write tool;
- requires every declared argument to exist on the tool;
- refuses a locating argument (project_id, task_id, milestone_id and the
  rest) that a write tool takes without declaring it.

The integration test traces each locator from real rows.

Milestone 583, step 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6854): the locating arguments that only point are declared as references, and the plugin version is minted ... 16b1d68ebd

The guard caught seven write-tool arguments with a locating name that are not
where the write lands:
- snippet_id on classify_shapes_by_rule and confirm_shape_proposals names the
  canon, not the rows written;
- rule_outcome, rule_misfired and judge_lesson_link record judgments.

REFERENCES lists each one with its reason, and the guard reads it.
READ_PREFIXES names the tool prefixes the fence answers without the server,
and a test holds every write tool to a name outside them.

plugin.json is minted to 2026.11.05.0242 for the PACKAGING.md row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(6855): the plugin asks before a session writes to a project other than its own ... 805fda5cb1

plugin/hooks/grimoire_project_fence.sh runs as PreToolUse on editor writes,
Bash and Grimoire's tools. A session belongs to the project its starting
directory resolves to (CLAUDE_PROJECT_DIR: a .grimoire marker, else the git
remote's binding). A session that starts outside any project belongs to the
first one it enters, which PostToolUse on enter_project records.

These acts stop on permissionDecision "ask", with the server's reason naming
the other project:
- an edit in another project's repo or a worktree of it;
- a git commit, push, merge, rebase, pull, cherry-pick, revert, tag, am, reset
  or worktree add there, found through git -C, a cd before it, or the shell's
  current directory;
- a Grimoire write that lands in another project.

A live probe showed "ask" holds in bypassPermissions. Filing an issue and
every read go ahead. An edit or git command in the session's own repo, and any
tool named like a read, never calls the server. Grimoire writes send only
their short arguments, so the URL stays small.

When the instance doesn't answer, the act goes ahead and the hook says once
per outage that it couldn't check. check_plugin.py now holds one outage line
per hook and has a smoke event for the fence.

GET /api/plugin/write-scope takes fallback_project_id for the entered project.

tests/test_project_fence.py drives the hook against real git repos and a
stand-in server. tests/test_write_scope.py holds the hook's
FENCE_READ_PREFIXES equal to services/write_scope.READ_PREFIXES. The plugin
README and PACKAGING describe the fence, and the plugin is minted to
2026.11.05.0245.

Milestone 583, step 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(6855): the fence keeps its files where no compaction sweeps them, and builds id lists under mawk ... ddcac96542

The project a session entered now lives in grimoire-fence, which the
compaction clear never visits, and session end removes it. Arrays of ids
in a tool's input were sent as [,"3","4"] under mawk, which creates an
array element as soon as it is assigned to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(6856): the plugin's guidance and the docs say what a session may do in another project ... 8f8263b347

using-grimoire, which owns cross-project conduct, now says the one change
to make toward another project is filing an issue with how it was found
and the evidence, and that the plugin asks before anything more. The
features page describes the fence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(6856): the scope section says the cross-project rule in fewer lines, so using-grimoire fits its 350-line budget ... 49b5ceee41

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bvandeusen merged commit 85093c859b into main 2026-11-05 09:29:55 -05:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledGrimoire#220