Backups: a daily dedupe-friendly tar to a folder or S3, with restore (#5918-#5921)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Build now, or wait for Android? (push) Successful in 3s
Android / Build, or is the channel already serving this? (push) Successful in 4s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Android / Build the server image (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 3s
CI & Build / Web typecheck and unit tests (push) Successful in 10s
CI & Build / Python tests (push) Successful in 14s
CI & Build / integration (push) Successful in 1m38s
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Successful in 1m41s
CI & Build / Build & push image (push) Successful in 41s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m18s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m5s
Desktop (Tauri) / Update manifest (push) Successful in 3s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Build now, or wait for Android? (push) Successful in 3s
Android / Build, or is the channel already serving this? (push) Successful in 4s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Android / Build the server image (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 3s
CI & Build / Web typecheck and unit tests (push) Successful in 10s
CI & Build / Python tests (push) Successful in 14s
CI & Build / integration (push) Successful in 1m38s
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Successful in 1m41s
CI & Build / Build & push image (push) Successful in 41s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m18s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m5s
Desktop (Tauri) / Update manifest (push) Successful in 3s
#5918 "Backup file": an uncompressed tar of per-table JSON Lines (primary-key order, sorted keys) plus every attachment file, with fixed tar headers, a dated name, and an atomic write through a dot-named .part file. Unchanged rows are the same bytes every day, so restic off-site stores only the edits. No credential ever enters a backup. #5919 "Backup destinations": a folder, or an S3-compatible bucket through a stdlib SigV4 put/list client (checked against AWS's published signatures), with an upload limit. Test destination only lists, so an append-only key works. #5920 "Backup job": daily at a chosen hour, Back up now, a 1/2/4 h retry ladder, a one-hour deadline, an advisory lock, the last 30 runs, and an overdue verdict on Settings -> Backups. #5921 "Restore": python -m inkwell.restore FILE into an empty instance, an integration test that backs up and restores, and docs/backups.md with Fabled Archives as the worked example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+125
@@ -0,0 +1,125 @@
|
||||
# Backups
|
||||
|
||||
Inkwell can write a backup of everything once a day: every account, note, tag, share,
|
||||
group, invite and setting, and every attachment file. Each backup goes to a folder or
|
||||
to an S3-compatible bucket, and is shaped so that a deduplicating off-site copy
|
||||
(restic) stores only what changed.
|
||||
|
||||
Set it up in **Settings → Backups**.
|
||||
|
||||
## What a backup is
|
||||
|
||||
One file per run, named for the moment it was taken: `inkwell-20261016T020000Z.tar`.
|
||||
It is an ordinary, **uncompressed** tar:
|
||||
|
||||
```
|
||||
manifest.json when, which version, how many rows and files
|
||||
tables/<table>.jsonl one row per line, in primary-key order
|
||||
files/<path> every attachment, byte for byte
|
||||
```
|
||||
|
||||
The layout is chosen for restic. restic splits a file into chunks at points chosen by
|
||||
its content, so a chunk it has already stored costs nothing to store again. To make
|
||||
the most of that:
|
||||
|
||||
- **Nothing is compressed.** Compression changes every byte after the first change,
|
||||
and nothing would match from one day to the next. restic compresses its own
|
||||
repository after it deduplicates.
|
||||
- **An unchanged row is the same bytes every day.** Rows are written in primary-key
|
||||
order with sorted keys. Editing a note changes one line of `tables/notes.jsonl`.
|
||||
- **Attachments never change**, so they deduplicate run after run.
|
||||
- **Only `manifest.json` changes every run.** It holds the time and the counts; the
|
||||
tar headers are fixed.
|
||||
|
||||
In practice a day's new backup adds roughly the size of that day's edits and uploads
|
||||
to the off-site repository, not the size of the whole file.
|
||||
|
||||
A backup **never** holds a credential: not the SMTP password, not the bucket's secret
|
||||
key, and not the server's own signing key. A copy kept off-site cannot be used to
|
||||
reach the server or the bucket it came from. One-hour password reset links are left
|
||||
out too.
|
||||
|
||||
## Where it goes
|
||||
|
||||
Inkwell only ever **adds** a file. It never deletes or replaces one. Keeping a
|
||||
history and thinning it (keep 7 daily, 4 weekly, 12 monthly, say) is the
|
||||
destination's job.
|
||||
|
||||
### A folder
|
||||
|
||||
Mount a share into the container and give its path in Settings:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
volumes:
|
||||
- inkwell-data:/var/inkwell
|
||||
- /mnt/archives/inkwell:/backups # add this line
|
||||
```
|
||||
|
||||
The file is written as `.inkwell-….tar.part` and renamed when it is complete, so a
|
||||
service that watches the folder and skips names starting with a dot never picks up
|
||||
half a backup.
|
||||
|
||||
### An S3-compatible bucket
|
||||
|
||||
Any S3 server that accepts path-style addresses: Fabled Archives, MinIO, Garage,
|
||||
Backblaze B2 and AWS itself. Give the address **with its port**, the bucket, the
|
||||
region (`us-east-1` suits most self-hosted servers) and a key.
|
||||
|
||||
Inkwell only puts and lists, so a key that may **only add objects** is enough. It is
|
||||
also the safer key: a stolen copy cannot delete the history it writes into.
|
||||
**Test destination** only lists the bucket, so it works with such a key.
|
||||
|
||||
Uploads are capped at 10 MB/s by default (**Upload limit**), so a large backup never
|
||||
crowds the network. Set it to 0 for no limit.
|
||||
|
||||
### Fabled Archives
|
||||
|
||||
This is the worked example. Make a backup target in Archives with retention set to
|
||||
**Archives** (one file per backup, thinned like restic's keep-daily/weekly/monthly)
|
||||
and either:
|
||||
|
||||
- **S3:** set the destination to the bucket, using Archives' S3 address including the
|
||||
port (for example `http://192.168.0.10:8081`). Without the port the request lands on
|
||||
port 80, and the answer is a plain 404, shown as "nothing answered as S3 at that
|
||||
address; check the port". Give Inkwell an append-only key.
|
||||
- **Folder:** mount the target's folder into the container (above) and point the
|
||||
folder setting at it.
|
||||
|
||||
Archives then copies the target off-site with restic, and the shape above is what
|
||||
lets that copy deduplicate.
|
||||
|
||||
## When it runs
|
||||
|
||||
Once a day at the hour you choose (UTC on the server; the page shows the next run in
|
||||
your own time). A backup missed while the server was down runs as soon as it is back.
|
||||
Switching backups on runs the first one within a few minutes, which is also the
|
||||
quickest check that the destination works.
|
||||
|
||||
A failed run is tried again after an hour, then two, then four, and then waits for
|
||||
the next day. The page says when backups started failing and why. Every run has a
|
||||
one-hour deadline, and the page keeps the last 30.
|
||||
|
||||
**Back up now** makes one straight away.
|
||||
|
||||
## Restoring
|
||||
|
||||
Restore into a **new, empty** Inkwell:
|
||||
|
||||
1. Start the new instance once, so its database is created. Don't sign up.
|
||||
2. Put the backup file where the container can read it, then run:
|
||||
|
||||
```sh
|
||||
docker compose exec app python -m inkwell.restore /backups/inkwell-20261016T020000Z.tar
|
||||
```
|
||||
|
||||
3. Sign in with an account from the backup.
|
||||
4. In Settings, enter the SMTP password and the bucket's secret key again. They are
|
||||
never in a backup.
|
||||
|
||||
The restore refuses a database that already has accounts, because loading one
|
||||
instance's notes over another's would mix the two with no way to separate them.
|
||||
|
||||
Everyone signs in again after a restore, because the signing key is not in the
|
||||
backup. Linked desktop and Android apps keep their links, and should be re-synced.
|
||||
Reference in New Issue
Block a user