Restore rolls back to a backup; a session can only file issues in another project #220

Merged
bvandeusen merged 15 commits from dev into main 2026-11-05 09:29:55 -05:00
6 changed files with 139 additions and 56 deletions
Showing only changes of commit 37be3b0bff - Show all commits
+1 -1
View File
@@ -96,7 +96,7 @@ docker compose up -d
Database migrations run when the app starts, and the key stays in its volume. Claude Code updates the plugin from its marketplace; run `/reload-plugins` after an update.
**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](docs/backups.md).
**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and Settings → Data → Restore from Backup rolls the instance back to one, or brings a new instance back from it. See [Backups](docs/backups.md).
## Features
+3 -1
View File
@@ -272,7 +272,9 @@ These serve the plugin's hooks.
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/admin/backup` | Export backup (format v4; `?scope=user` for own data) |
| POST | `/api/admin/restore` | Restore from a backup |
| GET | `/api/admin/restore/sources` | The backups at the backup destination, newest first |
| POST | `/api/admin/restore/preview` | What restoring a backup (`?source=<name>` or an uploaded file) would change; 409 with the reason when it can't apply here |
| POST | `/api/admin/restore?confirm=rollback` | Put the whole instance back to that backup, after saving the current state |
| GET / DELETE | `/api/admin/users` · `/api/admin/users/:id` | List / delete users |
| GET / PUT | `/api/admin/registration` | Get / toggle registration |
| GET / PUT | `/api/admin/smtp` · POST `/api/admin/smtp/test` | SMTP config + test email |
+1 -1
View File
@@ -87,7 +87,7 @@ Credentials Grimoire stores for outside services (forge tokens, the backup desti
│ ├── routes/ REST blueprints
│ ├── mcp/ MCP server and its tool modules
│ ├── services/ shared logic
│ └── restore.py `python -m grimoire.restore` for a fresh instance
│ └── restore.py `python -m grimoire.restore`: the restore, from the command line
├── frontend/src/ views, components, stores, composables, api client
├── plugin/ Claude Code hooks, skills and commands
├── scripts/ CI checks and the plugin version minter
+52 -21
View File
@@ -88,9 +88,12 @@ You need four things:
- **Access key.**
- **Secret key.**
Grimoire uploads and lists; it never deletes. A key that may **only add
objects** is therefore enough, and it is the safer key to hand out: a stolen
one cannot wipe the history.
Grimoire uploads and lists, and reads a backup back when you restore from the
bucket; it never deletes. A key that may **add and read objects but not
delete them** is therefore enough, and it is the safer key to hand out: a
stolen one cannot wipe the history. A key that may only add objects still
takes backups; restore then works by uploading a file or from the command
line.
The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
never shown again, and never included in a backup. If an instance's
@@ -122,8 +125,10 @@ keeps service backups as "backup targets". Either destination works with it.
1. On Archives' **Backups** page, make a target for Grimoire, named `grimoire`
for example.
2. Choose how Grimoire's files arrive:
- **S3**: turn on the S3 way in for the target and make an **append-only**
S3 key. In Grimoire, choose **An S3 bucket**:
- **S3**: turn on the S3 way in for the target and make an S3 key that
may add and read objects but not delete them (an **append-only** key
takes backups too, but then Grimoire can't restore from the bucket). In
Grimoire, choose **An S3 bucket**:
- address: Archives' backup address (`http://<archives>:8081`);
- bucket: the target's name;
- access key and secret key: the key you just made.
@@ -139,26 +144,52 @@ keeps service backups as "backup targets". Either destination works with it.
## Restoring
### Into a fresh instance
A restore puts the **whole instance** back to the moment a backup was taken,
for every user:
Start Grimoire on an empty database, copy the backup file into the container,
and run:
- records edited since go back to how they were;
- records deleted since come back;
- anything made since is removed.
It matches records by their ids, so a live instance only restores a backup **of
its own**. Each instance has an id, and every backup records the one it came
from. A backup from another instance, or one made before backups recorded it,
only goes into a new, empty instance, which takes it as it is.
Before changing anything, the restore saves the current state as one more
backup. It goes to the backup destination when scheduled backups are on, and
otherwise to `/data/pre-restore`. The result says which file it is, so a
restore can itself be undone.
What a backup doesn't carry stays as it is wherever its owner survives the
restore: API keys, forge connections, sharing, logs, and the destination's S3
secret. The trash isn't in backups, so anything in the trash when the backup
was taken is gone afterwards. Everyone is signed out, and search is rebuilt.
### From the web
**Settings → Data → Restore from Backup** lists the backups at the backup
destination. You can pick one of those, or upload a `.tar` or the JSON
**Full Backup** download (uploads are held to the server's request-size
limit). The restore then shows what would change: when the backup was taken,
and how many accounts, projects, plans, notes and rules there are now and will
be afterwards. Type `roll back` to run it. Then sign in again.
To read backups back from an S3 bucket, the key needs read access as well as
write. Grimoire never deletes from the bucket.
### From the command line
For a file with no size limit, or a new instance with no backup destination
yet, copy the file into the container and run:
```sh
docker exec -it grimoire python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
```
- The command refuses a database that already has an account. A restore
re-creates everything with new ids, so on a live instance it would add a
second copy instead of replacing anything.
- Nothing limits the file's size.
- It also reads the JSON from the admin **Full Backup** download.
It prints what would change and stops. Add `--yes` to restore. It reads the
JSON download too. Afterwards, **restart Grimoire** so the search indexes are
rebuilt.
Afterwards, **restart Grimoire** so the search indexes are rebuilt. Then issue
new API keys, add forge connections again, and set up any sharing.
### From the web
**Settings → Data → Restore from Backup** takes the `.tar` as well as the
JSON download. The upload is held to the server's request-size limit. For a
bigger file, use the command above.
After restoring into a new instance, issue new API keys, add forge connections
again, and set up any sharing: none of them travel in a backup.
+2 -2
View File
@@ -85,8 +85,8 @@ Share a project, note or task with users or groups, as viewer, editor or admin.
## Export and backup
- **Personal export**: your notes and tasks as a Markdown ZIP (with YAML frontmatter) or JSON.
- **Admin backup and restore**, with ids remapped so a backup restores onto another instance.
- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](backups.md).
- **Admin backup and restore**: a restore rolls the whole instance back to one of its own backups, after saving the current state, or brings a new instance back from any backup.
- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. Restoring is in Settings, or `python -m grimoire.restore`. See [Backups](backups.md).
## Sign-in
+80 -30
View File
@@ -1,20 +1,26 @@
"""Restore a Grimoire backup into an empty instance, from the command line.
"""Restore a Grimoire backup from the command line.
python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar --yes
This is how a fresh instance is brought back after losing the old one. Run it
inside the Grimoire container, which already has the database settings and has
already migrated the schema on start. It reads either the scheduled backup's
tar (services/backup_file.py) or the JSON the admin download produces.
Run it inside the Grimoire container, which already has the database settings
and has migrated the schema on start. It reads the scheduled backup's tar
(services/backup_file.py) or the JSON the admin download produces, and nothing
limits the file's size.
It refuses a database that already holds an account. A restore re-creates
every user, project and record from the file with new ids; into a live
instance it would add a second copy of everything rather than replace it.
(Settings → Backup & Restore still offers the upload for that case, on
purpose and with its own warning.)
It does what Settings → Restore does (services/backup.py replace_from_backup):
- **A new, empty instance** takes the backup as it is.
- **A live instance** rolls back to a backup of its own: everything goes back
to the moment the backup was taken, for every user.
- A backup from another instance, or one made before backups recorded their
instance, only goes into an empty instance.
Unlike the upload, nothing limits the file's size here, so this is also the
way in for a backup too big for the web upload.
Without --yes it only says what would change. With --yes it first saves the
current state as a backup (to the backup destination, or DATA_DIR/pre-restore),
then restores. Everyone is signed out afterwards.
A backup in the oldest (v1) format is copied into an empty instance with new
ids instead, as before.
Afterwards, restart Grimoire: search indexes are not part of a backup and are
rebuilt from the restored records at startup.
@@ -48,9 +54,39 @@ async def _account_count() -> int:
return (await session.execute(select(func.count()).select_from(User))).scalar_one()
async def run(path: Path) -> int:
from grimoire.models import engine
async def _restore_v1(payload: dict) -> int:
from grimoire.services.backup import restore_full_backup
accounts = await _account_count()
if accounts:
print(
f"This database already holds {accounts} account(s). A backup in the oldest "
"format is copied in with new ids rather than replacing anything, so it only "
"goes into an empty instance. Start Grimoire on an empty database and run "
"this again.",
file=sys.stderr,
)
return 3
stats = await restore_full_backup(payload)
print(json.dumps({"restored": stats}, indent=2, default=str))
return 0
def _print_preview(preview: dict) -> None:
taken = preview.get("exported_at") or "an unknown date"
if preview["mode"] == "rollback":
print(f"This rolls the whole instance back to {taken}, for every user. Anything "
"made since is removed, edits since are undone, records deleted since come back.")
else:
print(f"This instance has no records yet; it takes the backup from {taken} as it is.")
for count in preview["counts"]:
print(f" {count['kind']:<24} now {count['now']:>7} after {count['after']:>7}")
async def run(path: Path, *, yes: bool = False) -> int:
from grimoire.models import engine
from grimoire.services import backup_job
from grimoire.services.backup import RestoreRefused, replace_preview
from grimoire.services.backup_file import ArchiveError
try:
@@ -63,20 +99,29 @@ async def run(path: Path) -> int:
print(f"{path} isn't a backup this Grimoire can read: {exc}", file=sys.stderr)
return 2
accounts = await _account_count()
if accounts:
print(
f"This database already holds {accounts} account(s). A restore adds a second "
"copy of everything rather than replacing it, so it only runs into an empty "
"instance. Start Grimoire on an empty database and run this again.",
file=sys.stderr,
)
return 3
stats = await restore_full_backup(payload)
print(json.dumps({"restored": stats}, indent=2, default=str))
print("Restored. Restart Grimoire so the search indexes are rebuilt.", file=sys.stderr)
return 0
if (payload.get("version") or 1) < 2:
code = await _restore_v1(payload)
else:
try:
_print_preview(await replace_preview(payload))
if not yes:
print("Nothing was changed. Run it again with --yes to restore.",
file=sys.stderr)
return 4
result = await backup_job.rollback_to_backup(payload)
except RestoreRefused as exc:
print(str(exc), file=sys.stderr)
return 3
safety = result["safety_backup"]
print(json.dumps({"restored": result["restored"], "safety_backup": safety},
indent=2, default=str))
print(f"The state it replaced is saved as {safety['file']} in {safety['where']}.",
file=sys.stderr)
code = 0
if code == 0:
print("Restored. Restart Grimoire so the search indexes are rebuilt; "
"everyone signs in again.", file=sys.stderr)
return code
finally:
await engine.dispose()
@@ -84,11 +129,16 @@ async def run(path: Path) -> int:
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m grimoire.restore",
description="Restore a Grimoire backup (.tar or .json) into an empty instance.",
description=(
"Restore a Grimoire backup (.tar or .json): into an empty instance, or "
"roll this instance back to a backup of its own."
),
)
parser.add_argument("file", type=Path, help="the backup file")
parser.add_argument("--yes", action="store_true",
help="restore; without it, only say what would change")
args = parser.parse_args(argv)
return asyncio.run(run(args.file))
return asyncio.run(run(args.file, yes=args.yes))
if __name__ == "__main__":