diff --git a/README.md b/README.md index 6d449355..b90b4979 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ docker compose up -d Database migrations run when the app starts, and the key stays in its volume. Claude Code updates the plugin from its marketplace; run `/reload-plugins` after an update. -**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](docs/backups.md). +**Settings → Data → Scheduled backups** writes a daily backup of the database to a folder or an S3-compatible bucket, and Settings → Data → Restore from Backup rolls the instance back to one, or brings a new instance back from it. See [Backups](docs/backups.md). ## Features diff --git a/docs/api-reference.md b/docs/api-reference.md index c953cba0..ee926247 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -272,7 +272,9 @@ These serve the plugin's hooks. | Method | Path | Description | |--------|------|-------------| | GET | `/api/admin/backup` | Export backup (format v4; `?scope=user` for own data) | -| POST | `/api/admin/restore` | Restore from a backup | +| GET | `/api/admin/restore/sources` | The backups at the backup destination, newest first | +| POST | `/api/admin/restore/preview` | What restoring a backup (`?source=` or an uploaded file) would change; 409 with the reason when it can't apply here | +| POST | `/api/admin/restore?confirm=rollback` | Put the whole instance back to that backup, after saving the current state | | GET / DELETE | `/api/admin/users` · `/api/admin/users/:id` | List / delete users | | GET / PUT | `/api/admin/registration` | Get / toggle registration | | GET / PUT | `/api/admin/smtp` · POST `/api/admin/smtp/test` | SMTP config + test email | diff --git a/docs/architecture.md b/docs/architecture.md index 4611f3bc..a6c8b672 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -87,7 +87,7 @@ Credentials Grimoire stores for outside services (forge tokens, the backup desti │ ├── routes/ REST blueprints │ ├── mcp/ MCP server and its tool modules │ ├── services/ shared logic -│ └── restore.py `python -m grimoire.restore` for a fresh instance +│ └── restore.py `python -m grimoire.restore`: the restore, from the command line ├── frontend/src/ views, components, stores, composables, api client ├── plugin/ Claude Code hooks, skills and commands ├── scripts/ CI checks and the plugin version minter diff --git a/docs/backups.md b/docs/backups.md index e0c630cb..6df13ea2 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -88,9 +88,12 @@ You need four things: - **Access key.** - **Secret key.** -Grimoire uploads and lists; it never deletes. A key that may **only add -objects** is therefore enough, and it is the safer key to hand out: a stolen -one cannot wipe the history. +Grimoire uploads and lists, and reads a backup back when you restore from the +bucket; it never deletes. A key that may **add and read objects but not +delete them** is therefore enough, and it is the safer key to hand out: a +stolen one cannot wipe the history. A key that may only add objects still +takes backups; restore then works by uploading a file or from the command +line. The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is never shown again, and never included in a backup. If an instance's @@ -122,8 +125,10 @@ keeps service backups as "backup targets". Either destination works with it. 1. On Archives' **Backups** page, make a target for Grimoire, named `grimoire` for example. 2. Choose how Grimoire's files arrive: - - **S3**: turn on the S3 way in for the target and make an **append-only** - S3 key. In Grimoire, choose **An S3 bucket**: + - **S3**: turn on the S3 way in for the target and make an S3 key that + may add and read objects but not delete them (an **append-only** key + takes backups too, but then Grimoire can't restore from the bucket). In + Grimoire, choose **An S3 bucket**: - address: Archives' backup address (`http://:8081`); - bucket: the target's name; - access key and secret key: the key you just made. @@ -139,26 +144,52 @@ keeps service backups as "backup targets". Either destination works with it. ## Restoring -### Into a fresh instance +A restore puts the **whole instance** back to the moment a backup was taken, +for every user: -Start Grimoire on an empty database, copy the backup file into the container, -and run: +- records edited since go back to how they were; +- records deleted since come back; +- anything made since is removed. + +It matches records by their ids, so a live instance only restores a backup **of +its own**. Each instance has an id, and every backup records the one it came +from. A backup from another instance, or one made before backups recorded it, +only goes into a new, empty instance, which takes it as it is. + +Before changing anything, the restore saves the current state as one more +backup. It goes to the backup destination when scheduled backups are on, and +otherwise to `/data/pre-restore`. The result says which file it is, so a +restore can itself be undone. + +What a backup doesn't carry stays as it is wherever its owner survives the +restore: API keys, forge connections, sharing, logs, and the destination's S3 +secret. The trash isn't in backups, so anything in the trash when the backup +was taken is gone afterwards. Everyone is signed out, and search is rebuilt. + +### From the web + +**Settings → Data → Restore from Backup** lists the backups at the backup +destination. You can pick one of those, or upload a `.tar` or the JSON +**Full Backup** download (uploads are held to the server's request-size +limit). The restore then shows what would change: when the backup was taken, +and how many accounts, projects, plans, notes and rules there are now and will +be afterwards. Type `roll back` to run it. Then sign in again. + +To read backups back from an S3 bucket, the key needs read access as well as +write. Grimoire never deletes from the bucket. + +### From the command line + +For a file with no size limit, or a new instance with no backup destination +yet, copy the file into the container and run: ```sh docker exec -it grimoire python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar ``` -- The command refuses a database that already has an account. A restore - re-creates everything with new ids, so on a live instance it would add a - second copy instead of replacing anything. -- Nothing limits the file's size. -- It also reads the JSON from the admin **Full Backup** download. +It prints what would change and stops. Add `--yes` to restore. It reads the +JSON download too. Afterwards, **restart Grimoire** so the search indexes are +rebuilt. -Afterwards, **restart Grimoire** so the search indexes are rebuilt. Then issue -new API keys, add forge connections again, and set up any sharing. - -### From the web - -**Settings → Data → Restore from Backup** takes the `.tar` as well as the -JSON download. The upload is held to the server's request-size limit. For a -bigger file, use the command above. +After restoring into a new instance, issue new API keys, add forge connections +again, and set up any sharing: none of them travel in a backup. diff --git a/docs/features.md b/docs/features.md index 46734f89..26e37e83 100644 --- a/docs/features.md +++ b/docs/features.md @@ -85,8 +85,8 @@ Share a project, note or task with users or groups, as viewer, editor or admin. ## Export and backup - **Personal export**: your notes and tasks as a Markdown ZIP (with YAML frontmatter) or JSON. -- **Admin backup and restore**, with ids remapped so a backup restores onto another instance. -- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. `python -m grimoire.restore` brings a fresh instance back from one. See [Backups](backups.md). +- **Admin backup and restore**: a restore rolls the whole instance back to one of its own backups, after saving the current state, or brings a new instance back from any backup. +- **Scheduled backups**: a daily, dated, uncompressed tar to a folder or an S3-compatible bucket, laid out so a deduplicating tool such as restic stores only what changed. Restoring is in Settings, or `python -m grimoire.restore`. See [Backups](backups.md). ## Sign-in diff --git a/src/grimoire/restore.py b/src/grimoire/restore.py index 2a8549c9..a2ed4ace 100644 --- a/src/grimoire/restore.py +++ b/src/grimoire/restore.py @@ -1,20 +1,26 @@ -"""Restore a Grimoire backup into an empty instance, from the command line. +"""Restore a Grimoire backup from the command line. python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar + python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar --yes -This is how a fresh instance is brought back after losing the old one. Run it -inside the Grimoire container, which already has the database settings and has -already migrated the schema on start. It reads either the scheduled backup's -tar (services/backup_file.py) or the JSON the admin download produces. +Run it inside the Grimoire container, which already has the database settings +and has migrated the schema on start. It reads the scheduled backup's tar +(services/backup_file.py) or the JSON the admin download produces, and nothing +limits the file's size. -It refuses a database that already holds an account. A restore re-creates -every user, project and record from the file with new ids; into a live -instance it would add a second copy of everything rather than replace it. -(Settings → Backup & Restore still offers the upload for that case, on -purpose and with its own warning.) +It does what Settings → Restore does (services/backup.py replace_from_backup): +- **A new, empty instance** takes the backup as it is. +- **A live instance** rolls back to a backup of its own: everything goes back + to the moment the backup was taken, for every user. +- A backup from another instance, or one made before backups recorded their + instance, only goes into an empty instance. -Unlike the upload, nothing limits the file's size here, so this is also the -way in for a backup too big for the web upload. +Without --yes it only says what would change. With --yes it first saves the +current state as a backup (to the backup destination, or DATA_DIR/pre-restore), +then restores. Everyone is signed out afterwards. + +A backup in the oldest (v1) format is copied into an empty instance with new +ids instead, as before. Afterwards, restart Grimoire: search indexes are not part of a backup and are rebuilt from the restored records at startup. @@ -48,9 +54,39 @@ async def _account_count() -> int: return (await session.execute(select(func.count()).select_from(User))).scalar_one() -async def run(path: Path) -> int: - from grimoire.models import engine +async def _restore_v1(payload: dict) -> int: from grimoire.services.backup import restore_full_backup + + accounts = await _account_count() + if accounts: + print( + f"This database already holds {accounts} account(s). A backup in the oldest " + "format is copied in with new ids rather than replacing anything, so it only " + "goes into an empty instance. Start Grimoire on an empty database and run " + "this again.", + file=sys.stderr, + ) + return 3 + stats = await restore_full_backup(payload) + print(json.dumps({"restored": stats}, indent=2, default=str)) + return 0 + + +def _print_preview(preview: dict) -> None: + taken = preview.get("exported_at") or "an unknown date" + if preview["mode"] == "rollback": + print(f"This rolls the whole instance back to {taken}, for every user. Anything " + "made since is removed, edits since are undone, records deleted since come back.") + else: + print(f"This instance has no records yet; it takes the backup from {taken} as it is.") + for count in preview["counts"]: + print(f" {count['kind']:<24} now {count['now']:>7} after {count['after']:>7}") + + +async def run(path: Path, *, yes: bool = False) -> int: + from grimoire.models import engine + from grimoire.services import backup_job + from grimoire.services.backup import RestoreRefused, replace_preview from grimoire.services.backup_file import ArchiveError try: @@ -63,20 +99,29 @@ async def run(path: Path) -> int: print(f"{path} isn't a backup this Grimoire can read: {exc}", file=sys.stderr) return 2 - accounts = await _account_count() - if accounts: - print( - f"This database already holds {accounts} account(s). A restore adds a second " - "copy of everything rather than replacing it, so it only runs into an empty " - "instance. Start Grimoire on an empty database and run this again.", - file=sys.stderr, - ) - return 3 - - stats = await restore_full_backup(payload) - print(json.dumps({"restored": stats}, indent=2, default=str)) - print("Restored. Restart Grimoire so the search indexes are rebuilt.", file=sys.stderr) - return 0 + if (payload.get("version") or 1) < 2: + code = await _restore_v1(payload) + else: + try: + _print_preview(await replace_preview(payload)) + if not yes: + print("Nothing was changed. Run it again with --yes to restore.", + file=sys.stderr) + return 4 + result = await backup_job.rollback_to_backup(payload) + except RestoreRefused as exc: + print(str(exc), file=sys.stderr) + return 3 + safety = result["safety_backup"] + print(json.dumps({"restored": result["restored"], "safety_backup": safety}, + indent=2, default=str)) + print(f"The state it replaced is saved as {safety['file']} in {safety['where']}.", + file=sys.stderr) + code = 0 + if code == 0: + print("Restored. Restart Grimoire so the search indexes are rebuilt; " + "everyone signs in again.", file=sys.stderr) + return code finally: await engine.dispose() @@ -84,11 +129,16 @@ async def run(path: Path) -> int: def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser( prog="python -m grimoire.restore", - description="Restore a Grimoire backup (.tar or .json) into an empty instance.", + description=( + "Restore a Grimoire backup (.tar or .json): into an empty instance, or " + "roll this instance back to a backup of its own." + ), ) parser.add_argument("file", type=Path, help="the backup file") + parser.add_argument("--yes", action="store_true", + help="restore; without it, only say what would change") args = parser.parse_args(argv) - return asyncio.run(run(args.file)) + return asyncio.run(run(args.file, yes=args.yes)) if __name__ == "__main__":