Compare commits

...
2 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 b45fc9efba feat(523): restore reads the backup tar; python -m scribe.restore brings a fresh instance back
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Failing after 1m8s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Skipped
The restore route takes the tar as a raw body beside the JSON download.
scribe/restore.py reads either format, refuses a database that already
has an account (a restore re-creates everything with new ids, so into a
live instance it would duplicate rather than replace), has no size
limit, and says to restart so search indexes rebuild. An integration
test restores a written tar through the real restore_full_backup and
checks every id is remapped. docs/backups.md explains the file, why it
is uncompressed, both destinations, Fabled Archives as the worked
example, and restoring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:40:24 -04:00
bvandeusenandClaude Opus 5.5 be59829dde feat(523): Settings -> Data -> Scheduled backups - the verdict first, then destination, schedule, Test, Back up now and recent runs
BackupScheduleSettings.vue leads with whether the data is backed up
(last good backup, or Overdue), then what is wrong (last failure in
words, an S3 secret this SECRET_KEY cannot open, incomplete config),
then the form: daily toggle and UTC hour, folder or S3 bucket, Test
destination with unsaved values, Back up now (polls while running,
re-armed in finally), and the last runs. The secret is write-only.
formatBytes moves to utils/bytes.ts for both views. Restore from
Backup now also takes the .tar, and says to use the CLI on a 413.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:40:24 -04:00
9 changed files with 864 additions and 15 deletions
+158
View File
@@ -0,0 +1,158 @@
# Backups
Scribe can back itself up every day to a **folder** or an **S3-compatible
bucket**. Each run adds one file; Scribe never deletes an old one. Keeping and
thinning the history is the destination's job.
Set it up under **Settings → Data → Scheduled backups** (admins only).
## What a backup is
One file per run, named for the moment it was taken in UTC:
```
scribe-20261015T020000Z.tar
```
It is a plain, **uncompressed** tar:
- `manifest.json` comes first. It holds the backup format version, the Scribe
version, when the backup was taken, and how many rows each table has.
- Then there is one `<table>.jsonl` per table, one row per line.
It holds everything the admin **Full Backup** download holds, laid out
differently. It does **not** hold:
- **Search indexes.** They are rebuilt from the records when Scribe starts.
- **Sharing:** groups, project shares and note shares.
- **Credentials:** API keys and forge connections. After a restore, issue new
API keys and add forge connections again.
- **Logs and notifications.**
- **The backup destination's own S3 secret.**
The file contains hashed passwords, so store it somewhere only trusted people
can read.
### Why it is shaped this way
The file is meant to be copied off-site by a deduplicating tool such as
restic, which stores a chunk of data only once no matter how many backups
contain it. Every choice serves that:
- **It is never compressed.** Compression changes every byte after the first
edit, so nothing would match the day before. restic compresses its own
repository after deduplicating, so nothing is lost.
- **Rows are in primary-key order, with their keys sorted.** A row that did
not change is the same bytes in the same place every day. An edited row
changes one line, and new rows land at the end of their table's file.
- **Each table is its own file**, so a busy table never shifts the bytes of a
quiet one.
- **The parts that change every run are kept in `manifest.json`**: the time
and the counts.
- **Each run gets a new, dated name.** Overwriting one name would keep one
copy and no history.
Day to day, the off-site copy then stores roughly what changed, not the whole
file again.
## Destinations
### A folder
A path **inside the Scribe container**, normally a share mounted there:
```yaml
services:
scribe:
volumes:
- /nfs/backup/scribe:/backups
```
Scribe writes each file under a hidden name (`.scribe-….tar.part`) and renames
it once it is complete. A program watching the folder never sees half a
backup.
### An S3-compatible bucket
You need four things:
- **Address:** the service itself, like `https://backup.example.com`. Don't
include the bucket; Scribe addresses buckets by path.
- **Bucket.**
- **Access key.**
- **Secret key.**
Scribe uploads and lists; it never deletes. A key that may **only add
objects** is therefore enough, and it is the safer key to hand out: a stolen
one cannot wipe the history.
The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
never shown again, and never included in a backup. If an instance's
`SECRET_KEY` changes, for example after a restore onto a new instance, the
page asks for the secret again.
**Test destination** checks the destination without sending a backup:
- a folder: it writes and removes a hidden file;
- a bucket: it only lists, because an add-only key could never remove a test
file.
## Schedule and history
- Backups run once a day at the hour you choose, in UTC. The default is 02:00,
before the nightly trash purge and database maintenance.
- **Back up now** runs one straight away.
- The page leads with when the last good backup was taken. It says
**Overdue** when backups are on and none has worked for 26 hours.
- It lists the last 30 runs, and gives the reason for each failure in words.
- A run that hasn't finished after an hour is abandoned and recorded as
failed.
## Example: Fabled Archives
[Fabled Archives](https://git.fabledsword.com/bvandeusen/fabledarchives)
keeps service backups as "backup targets". Either destination works with it.
1. On Archives' **Backups** page, make a target for Scribe, named `scribe`
for example.
2. Choose how Scribe's files arrive:
- **S3**: turn on the S3 way in for the target and make an **append-only**
S3 key. In Scribe, choose **An S3 bucket**:
- address: Archives' backup address (`http://<archives>:8081`);
- bucket: the target's name;
- access key and secret key: the key you just made.
- **A folder**: mount the share Archives watches into Scribe as
`/backups/scribe`, and point the target at that folder. In Scribe, choose
**A folder** with `/backups/scribe`.
3. Set the target's **Retention** to **Archives**. Archives then keeps the
newest backups and thins the rest by day, week and month. Scribe only adds.
4. Give the target an expected cadence of **daily**, so a missed backup turns
it late in Archives as well as overdue in Scribe.
5. Add an off-site **restic** replication job for the target. The uncompressed
layout above is what lets that job send only what changed each day.
## Restoring
### Into a fresh instance
Start Scribe on an empty database, copy the backup file into the container,
and run:
```sh
docker exec -it scribe python -m scribe.restore /backups/scribe-20261015T020000Z.tar
```
- The command refuses a database that already has an account. A restore
re-creates everything with new ids, so on a live instance it would add a
second copy instead of replacing anything.
- Nothing limits the file's size.
- It also reads the JSON from the admin **Full Backup** download.
Afterwards, **restart Scribe** so the search indexes are rebuilt. Then issue
new API keys, add forge connections again, and set up any sharing.
### From the web
**Settings → Data → Restore from Backup** takes the `.tar` as well as the
JSON download. The upload is held to the server's request-size limit. For a
bigger file, use the command above.
+6 -2
View File
@@ -127,8 +127,12 @@ Every read and mutation is scoped by owner + direct shares + group shares.
- **Personal export** — download all your notes/tasks as a Markdown ZIP (YAML
frontmatter) or a JSON array.
- **Admin backup** — full application backup/restore (format v4) with ID remapping on
- **Admin backup** — full application backup/restore with ID remapping on
restore for cross-instance migration.
- **Scheduled backups** — a daily, dated, uncompressed `.tar` to a folder or an
S3-compatible bucket, laid out so a deduplicating tool such as restic stores
only what changed; `python -m scribe.restore` brings a fresh instance back
from one. See [Backups](backups.md).
## Progressive Web App
@@ -148,7 +152,7 @@ reset, and session invalidation are included. See [SSO / OAuth](sso-oauth.md).
| Account | Email change, password change, session invalidation |
| Profile | Per-user profile fields |
| Integrations | SearXNG status |
| Data | Personal export; backup / restore (admin) |
| Data | Personal export; backup / restore and scheduled backups (admin) |
| API Keys | Create/revoke `fmcp_` keys for the MCP endpoint |
| Config (admin) | Base URL, SMTP, DB-maintenance schedule |
| Users (admin) | User list, invite links, registration toggle |
+73
View File
@@ -0,0 +1,73 @@
import { apiGet, apiPost, apiPut } from "@/api/client";
/**
* Scheduled backups (milestone 523) — `/api/admin/backups`. One dated,
* uncompressed tar per run, delivered to a folder or an S3 bucket.
*/
export interface BackupConfig {
enabled: boolean;
/** UTC hour the daily backup runs at, 0–23. */
hour: number;
destination: "folder" | "s3";
folder_path: string;
s3_endpoint: string;
s3_bucket: string;
s3_access_key: string;
s3_region: string;
s3_verify_tls: boolean;
}
export interface BackupRun {
started_at: string;
finished_at: string;
trigger: "scheduled" | "manual";
status: "ok" | "failed";
destination?: string;
file?: string;
size_bytes?: number;
where?: string;
error?: string;
duration_s: number;
}
export interface BackupsOverview {
config: BackupConfig;
/** Never the secret itself: whether one is kept, and whether this
* instance can still open it. */
secret: { set: boolean; readable: boolean };
/** What stops the saved config from sending a backup, in words. */
problems: string[];
running: boolean;
/** Newest first, at most 30. */
runs: BackupRun[];
last_run: BackupRun | null;
last_success: BackupRun | null;
overdue: boolean;
}
/** What the form sends: config fields, plus the secret when one was typed. */
export type BackupChanges = Partial<BackupConfig> & {
s3_secret?: string;
clear_s3_secret?: boolean;
};
export function getBackups(): Promise<BackupsOverview> {
return apiGet<BackupsOverview>("/api/admin/backups");
}
export function saveBackups(changes: BackupChanges): Promise<BackupsOverview> {
return apiPut<BackupsOverview>("/api/admin/backups", changes);
}
/** Ask the destination whether it would take a backup, with the form's
* unsaved values. The server gives the destination 45s; this waits a little
* longer so its own answer, not the client's deadline, is what arrives. */
export function checkBackupDestination(
changes: BackupChanges,
): Promise<{ ok: boolean; message: string }> {
return apiPost<{ ok: boolean; message: string }>("/api/admin/backups/check", changes, { timeoutMs: 60_000 });
}
export function runBackupNow(): Promise<{ status: string }> {
return apiPost<{ status: string }>("/api/admin/backups/run", {});
}
@@ -0,0 +1,388 @@
<script setup lang="ts">
import { computed, onBeforeUnmount, onMounted, ref } from "vue";
import { apiErrorMessage } from "@/api/client";
import {
checkBackupDestination, getBackups, runBackupNow, saveBackups,
type BackupChanges, type BackupConfig, type BackupRun, type BackupsOverview,
} from "@/api/backups";
import { relativeTime } from "@/composables/useRelativeTime";
import { formatBytes } from "@/utils/bytes";
import { fmtStamp } from "@/utils/dateFormat";
import { useToastStore } from "@/stores/toast";
/**
* Scheduled backups (milestone 523): one dated, uncompressed tar per run,
* delivered to a folder or an S3 bucket. The page answers "is my data backed
* up?" first — the verdict, then anything wrong — and the settings follow.
*/
const toastStore = useToastStore();
const data = ref<BackupsOverview | null>(null);
const loading = ref(true);
const loadError = ref("");
const form = ref<BackupConfig | null>(null);
// The secret is never sent back by the server; this holds only what the
// admin types, and is sent only when non-empty.
const secretInput = ref("");
const saving = ref(false);
const checking = ref(false);
const checkResult = ref<{ ok: boolean; message: string } | null>(null);
const starting = ref(false);
const showAllRuns = ref(false);
const RUNS_SHOWN = 8;
let pollTimer: ReturnType<typeof setTimeout> | null = null;
let unmounted = false;
async function load() {
loadError.value = "";
try {
const fresh = await getBackups();
data.value = fresh;
// Keep the admin's unsaved edits; take the server's config only on the
// first load or right after a save.
if (!form.value) form.value = { ...fresh.config };
} catch (e: unknown) {
loadError.value = apiErrorMessage(e, "Could not load the backup settings");
} finally {
loading.value = false;
schedulePoll();
}
}
// While a backup runs, follow it. Re-armed from load()'s finally (rule 157),
// so one failed read does not stop the page following the run.
function schedulePoll() {
if (pollTimer) clearTimeout(pollTimer);
pollTimer = null;
if (!unmounted && data.value?.running) pollTimer = setTimeout(load, 3000);
}
onMounted(load);
onBeforeUnmount(() => {
unmounted = true;
if (pollTimer) clearTimeout(pollTimer);
});
const hourLabel = (h: number) => `${String(h).padStart(2, "0")}:00 UTC`;
// ── The verdict, and what is wrong ─────────────────────────────────────────
const verdict = computed(() => {
const d = data.value;
if (!d) return { tone: "", text: "" };
if (d.running) return { tone: "", text: "Backing up now…" };
if (!d.config.enabled) {
return { tone: "muted", text: "Scheduled backups are off. Nothing is backed up unless you run one." };
}
const ok = d.last_success;
if (d.overdue) {
return {
tone: "error",
text: ok
? `Overdue — the last backup that worked was ${relativeTime(ok.finished_at)} (${fmtStamp(ok.finished_at)}).`
: "Overdue — no backup has worked yet.",
};
}
if (!ok) return { tone: "", text: `On — the first backup runs at ${hourLabel(d.config.hour)}.` };
return {
tone: "ok",
text: `Backed up ${relativeTime(ok.finished_at)}: ${ok.file}, ${formatBytes(ok.size_bytes ?? 0)}, to ${ok.where}.`,
};
});
const lastFailure = computed(() => {
const last = data.value?.last_run;
return last && last.status === "failed" ? last : null;
});
const secretUnreadable = computed(() => {
const s = data.value?.secret;
return !!s && s.set && !s.readable;
});
const visibleRuns = computed<BackupRun[]>(() => {
const runs = data.value?.runs ?? [];
return showAllRuns.value ? runs : runs.slice(0, RUNS_SHOWN);
});
const hiddenRuns = computed(() => Math.max(0, (data.value?.runs.length ?? 0) - RUNS_SHOWN));
// ── Actions ────────────────────────────────────────────────────────────────
function changes(): BackupChanges {
const out: BackupChanges = { ...(form.value as BackupConfig) };
if (secretInput.value) out.s3_secret = secretInput.value;
return out;
}
async function save() {
if (!form.value) return;
saving.value = true;
try {
data.value = await saveBackups(changes());
form.value = { ...data.value.config };
secretInput.value = "";
toastStore.show("Backup settings saved");
} catch (e: unknown) {
toastStore.show(apiErrorMessage(e, "Could not save the backup settings"), "error");
} finally {
saving.value = false;
}
}
async function removeSecret() {
try {
data.value = await saveBackups({ clear_s3_secret: true });
toastStore.show("S3 secret removed");
} catch (e: unknown) {
toastStore.show(apiErrorMessage(e, "Could not remove the secret"), "error");
}
}
async function check() {
if (!form.value) return;
checking.value = true;
checkResult.value = null;
try {
checkResult.value = await checkBackupDestination(changes());
} catch (e: unknown) {
checkResult.value = { ok: false, message: apiErrorMessage(e, "The check could not be run") };
} finally {
checking.value = false;
}
}
async function backUpNow() {
starting.value = true;
try {
await runBackupNow();
if (data.value) data.value.running = true;
schedulePoll();
} catch (e: unknown) {
toastStore.show(apiErrorMessage(e, "Could not start a backup"), "error");
} finally {
starting.value = false;
}
}
</script>
<template>
<div class="backups">
<p v-if="loading && !data" class="state">Loading…</p>
<p v-else-if="loadError && !data" class="state state-error" role="alert">
{{ loadError }}
<button type="button" class="btn-text" @click="load">Retry</button>
</p>
<template v-else-if="data && form">
<p :class="['verdict', `verdict-${verdict.tone}`]">{{ verdict.text }}</p>
<p v-if="lastFailure && !data.running" class="problem" role="alert">
The last run failed {{ relativeTime(lastFailure.finished_at) }}: {{ lastFailure.error }}
</p>
<p v-if="secretUnreadable" class="problem" role="alert">
The stored S3 secret can't be opened here — it was saved on an instance with a different
SECRET_KEY. Enter it again below.
</p>
<ul v-if="data.config.enabled && data.problems.length" class="problem-list" role="alert">
<li v-for="p in data.problems" :key="p">{{ p }}</li>
</ul>
<p class="intro">
Each run adds one dated file, <code>scribe-&lt;date&gt;.tar</code>, holding every table
as plain lines. It is not compressed, so a deduplicating tool such as restic stores only what
changed since the day before. Scribe never deletes old backups — the destination keeps and
thins the history. Sharing, API keys and forge connections are not included.
</p>
<div class="row">
<label class="check">
<input v-model="form.enabled" type="checkbox" />
Back up daily
</label>
<label class="inline-field">
at
<select v-model.number="form.hour" class="fs-input" aria-label="Backup hour (UTC)">
<option v-for="h in 24" :key="h - 1" :value="h - 1">{{ hourLabel(h - 1) }}</option>
</select>
</label>
</div>
<fieldset class="destination">
<legend>Send each backup to</legend>
<label class="check">
<input v-model="form.destination" type="radio" value="folder" />
A folder
</label>
<label class="check">
<input v-model="form.destination" type="radio" value="s3" />
An S3 bucket
</label>
</fieldset>
<div v-if="form.destination === 'folder'" class="fields">
<label class="field">
<span>Folder</span>
<input v-model.trim="form.folder_path" class="fs-input" placeholder="/backups" />
<span class="field-hint">
A path inside the Scribe container — usually a share mounted there that a backup
service watches.
</span>
</label>
</div>
<div v-else class="fields">
<label class="field">
<span>Address</span>
<input v-model.trim="form.s3_endpoint" class="fs-input" placeholder="https://backup.example.com" />
<span class="field-hint">The S3 service itself, without the bucket. Buckets are addressed by path.</span>
</label>
<label class="field">
<span>Bucket</span>
<input v-model.trim="form.s3_bucket" class="fs-input" placeholder="scribe" />
</label>
<label class="field">
<span>Access key</span>
<input v-model.trim="form.s3_access_key" class="fs-input" autocomplete="off" />
</label>
<label class="field">
<span>Secret key</span>
<input
v-model="secretInput"
type="password"
class="fs-input"
autocomplete="new-password"
:placeholder="data.secret.set && data.secret.readable ? 'Saved — type a new one to replace it' : ''"
/>
<span class="field-hint">
Stored encrypted, and never included in a backup. A key that may only add files is enough.
<button
v-if="data.secret.set"
type="button"
class="btn-text"
@click="removeSecret"
>Remove the saved secret</button>
</span>
</label>
<label class="field">
<span>Region</span>
<input v-model.trim="form.s3_region" class="fs-input" placeholder="us-east-1" />
</label>
<label class="check">
<input v-model="form.s3_verify_tls" type="checkbox" />
Check the address's TLS certificate
</label>
</div>
<div class="actions">
<button type="button" class="btn-primary" :disabled="saving" @click="save">
{{ saving ? "Saving…" : "Save" }}
</button>
<button type="button" class="btn-secondary" :disabled="checking" @click="check">
{{ checking ? "Checking…" : "Test destination" }}
</button>
<button
type="button"
class="btn-secondary"
:disabled="starting || data.running"
@click="backUpNow"
>
{{ data.running ? "Backing up…" : "Back up now" }}
</button>
</div>
<p v-if="checkResult" :class="['check-result', checkResult.ok ? 'check-ok' : 'check-bad']" role="status">
{{ checkResult.message }}
</p>
<h3 class="runs-label">Recent runs</h3>
<p v-if="!data.runs.length" class="state">None yet.</p>
<div v-else class="runs-scroll">
<table class="runs">
<thead>
<tr><th>When</th><th>Result</th><th>File</th><th class="num">Size</th><th class="num">Took</th></tr>
</thead>
<tbody>
<tr v-for="r in visibleRuns" :key="r.started_at" :class="{ failed: r.status === 'failed' }">
<td :title="fmtStamp(r.started_at)">
{{ relativeTime(r.started_at) }}
<span v-if="r.trigger === 'manual'" class="by-hand">by hand</span>
</td>
<td>{{ r.status === "ok" ? "Backed up" : r.error }}</td>
<td><code v-if="r.file">{{ r.file }}</code><span v-else>—</span></td>
<td class="num">{{ r.size_bytes != null ? formatBytes(r.size_bytes) : "—" }}</td>
<td class="num">{{ r.duration_s }} s</td>
</tr>
</tbody>
</table>
<button v-if="hiddenRuns && !showAllRuns" type="button" class="btn-text" @click="showAllRuns = true">
+{{ hiddenRuns }} earlier
</button>
</div>
</template>
</div>
</template>
<style scoped>
.backups { display: flex; flex-direction: column; gap: var(--fs-space-3); }
.state { margin: 0; font-size: var(--fs-size-body-sm); color: var(--fs-text-secondary); }
.state-error { color: var(--fs-error); }
.verdict { margin: 0; font-size: var(--fs-size-body); color: var(--fs-text-primary); }
.verdict-ok { color: var(--fs-success); }
.verdict-error { color: var(--fs-error); }
.verdict-muted { color: var(--fs-text-secondary); }
.problem, .problem-list {
margin: 0;
font-size: var(--fs-size-body-sm);
color: var(--fs-error);
}
.problem-list { padding-left: var(--fs-space-4); }
.intro { margin: 0; font-size: var(--fs-size-body-sm); color: var(--fs-text-secondary); line-height: var(--fs-leading-body); }
.intro code, .runs code { font-family: var(--fs-font-mono); font-size: var(--fs-size-tiny); }
.row { display: flex; flex-wrap: wrap; align-items: center; gap: var(--fs-space-3); }
.check { display: inline-flex; align-items: center; gap: var(--fs-space-2); font-size: var(--fs-size-body-sm); color: var(--fs-text-primary); }
.inline-field { display: inline-flex; align-items: center; gap: var(--fs-space-2); font-size: var(--fs-size-body-sm); color: var(--fs-text-secondary); }
.inline-field select { width: auto; }
.destination {
margin: 0; padding: 0; border: 0;
display: flex; flex-wrap: wrap; align-items: center; gap: var(--fs-space-3);
}
.destination legend {
float: left; margin-right: var(--fs-space-2);
font-size: var(--fs-size-body-sm); color: var(--fs-text-secondary);
}
.fields { display: flex; flex-direction: column; gap: var(--fs-space-3); max-width: 32rem; }
.field { display: flex; flex-direction: column; gap: var(--fs-space-1); }
.field > span:first-child { font-size: var(--fs-size-body-sm); color: var(--fs-text-secondary); }
.actions { display: flex; flex-wrap: wrap; gap: var(--fs-space-2); }
.check-result { margin: 0; font-size: var(--fs-size-body-sm); }
.check-ok { color: var(--fs-success); }
.check-bad { color: var(--fs-error); }
.runs-label {
margin: var(--fs-space-2) 0 0;
font-size: var(--fs-size-tiny); font-weight: normal;
text-transform: uppercase; letter-spacing: var(--fs-tracking-tiny);
color: var(--fs-text-tertiary);
}
.runs-scroll { overflow-x: auto; }
.runs { width: 100%; border-collapse: collapse; font-size: var(--fs-size-body-sm); }
.runs th {
text-align: left; font-weight: normal;
font-size: var(--fs-size-tiny); color: var(--fs-text-tertiary);
padding: var(--fs-space-1) var(--fs-space-2);
border-bottom: 1px solid var(--fs-border-color);
}
.runs td {
padding: var(--fs-space-1) var(--fs-space-2);
color: var(--fs-text-primary);
border-bottom: 1px solid var(--fs-border-color);
vertical-align: baseline;
}
.runs .num { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; }
.runs tr.failed td { color: var(--fs-error); }
.by-hand { margin-left: var(--fs-space-1); font-size: var(--fs-size-tiny); color: var(--fs-text-tertiary); }
</style>
+8
View File
@@ -0,0 +1,8 @@
/** "1.4 MB" — a byte count for people. Binary units, one decimal under 10. */
export function formatBytes(n: number): string {
if (n < 1024) return `${n} B`;
const units = ["KB", "MB", "GB", "TB"];
let v = n / 1024, i = 0;
while (v >= 1024 && i < units.length - 1) { v /= 1024; i++; }
return `${v.toFixed(v >= 10 || i === 0 ? 0 : 1)} ${units[i]}`;
}
+18 -12
View File
@@ -11,6 +11,8 @@ import PaginationBar from "@/components/PaginationBar.vue";
import TagInput from "@/components/TagInput.vue";
import MomentsSettings from "@/components/MomentsSettings.vue";
import ReplyShapesSettings from "@/components/ReplyShapesSettings.vue";
import BackupScheduleSettings from "@/components/BackupScheduleSettings.vue";
import { formatBytes } from "@/utils/bytes";
import { fmtDate, fmtLogStamp } from "@/utils/dateFormat";
import { fetchVersion, type VersionPayload } from "@/api/version";
@@ -640,13 +642,6 @@ const dbHealth = ref<DbHealth | null>(null);
const loadingHealth = ref(false);
const DEAD_PCT_WARN = 20; // dead-tuple ratio above this = autovacuum falling behind
function formatBytes(n: number): string {
if (n < 1024) return `${n} B`;
const units = ["KB", "MB", "GB", "TB"];
let v = n / 1024, i = 0;
while (v >= 1024 && i < units.length - 1) { v /= 1024; i++; }
return `${v.toFixed(v >= 10 || i === 0 ? 0 : 1)} ${units[i]}`;
}
const pluginInstallCommands = computed(() => {
const mkt = (pluginMarketplaceUrl.value || '').trim()
|| serverMarketplaceUrl.value
@@ -1368,14 +1363,21 @@ async function handleRestoreFile(event: Event) {
if (!file) return;
restoring.value = true;
try {
const text = await file.text();
const data = JSON.parse(text);
// The scheduled backup's .tar goes up as it is; the JSON download is
// parsed first so a damaged file fails here, before the upload.
const isTar = file.name.toLowerCase().endsWith(".tar");
const res = await fetch("/api/admin/restore", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(data),
headers: { "Content-Type": isTar ? "application/x-tar" : "application/json" },
body: isTar ? file : JSON.stringify(JSON.parse(await file.text())),
signal: bulkDeadline(),
});
if (res.status === 413) {
throw new Error(
"the file is bigger than the web upload allows. Restore it with " +
"`python -m scribe.restore <file>` inside the Scribe container instead.",
);
}
if (!res.ok) {
const body = await res.json().catch(() => ({ error: `Error ${res.status}` }));
throw new Error((body as Record<string, string>).error || `Error ${res.status}`);
@@ -2650,12 +2652,16 @@ async function deleteUser(userId: number) {
<input
ref="restoreFileInput"
type="file"
accept=".json"
accept=".json,.tar"
class="hidden-file-input"
@change="handleRestoreFile"
/>
</div>
</section>
<section class="settings-section full-width">
<h2>Scheduled backups</h2>
<BackupScheduleSettings />
</section>
</template>
</div>
+95
View File
@@ -0,0 +1,95 @@
"""Restore a Scribe backup into an empty instance, from the command line.
python -m scribe.restore /backups/scribe-20261015T020000Z.tar
This is how a fresh instance is brought back after losing the old one. Run it
inside the Scribe container, which already has the database settings and has
already migrated the schema on start. It reads either the scheduled backup's
tar (services/backup_file.py) or the JSON the admin download produces.
It refuses a database that already holds an account. A restore re-creates
every user, project and record from the file with new ids; into a live
instance it would add a second copy of everything rather than replace it.
(Settings → Backup & Restore still offers the upload for that case, on
purpose and with its own warning.)
Unlike the upload, nothing limits the file's size here, so this is also the
way in for a backup too big for the web upload.
Afterwards, restart Scribe: search indexes are not part of a backup and are
rebuilt from the restored records at startup.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import sys
from pathlib import Path
from sqlalchemy import func, select
def load_backup(path: Path) -> dict:
"""The payload in `path`, whichever of the two backup formats it is."""
from scribe.services.backup_file import is_archive, read_archive
with open(path, "rb") as f:
if is_archive(f):
return read_archive(f)
return json.load(f)
async def _account_count() -> int:
from scribe.models import async_session
from scribe.models.user import User
async with async_session() as session:
return (await session.execute(select(func.count()).select_from(User))).scalar_one()
async def run(path: Path) -> int:
from scribe.models.engine import engine
from scribe.services.backup import restore_full_backup
from scribe.services.backup_file import ArchiveError
try:
try:
payload = load_backup(path)
except FileNotFoundError:
print(f"There is no file at {path}.", file=sys.stderr)
return 2
except (ArchiveError, ValueError) as exc:
print(f"{path} isn't a backup this Scribe can read: {exc}", file=sys.stderr)
return 2
accounts = await _account_count()
if accounts:
print(
f"This database already holds {accounts} account(s). A restore adds a second "
"copy of everything rather than replacing it, so it only runs into an empty "
"instance. Start Scribe on an empty database and run this again.",
file=sys.stderr,
)
return 3
stats = await restore_full_backup(payload)
print(json.dumps({"restored": stats}, indent=2, default=str))
print("Restored. Restart Scribe so the search indexes are rebuilt.", file=sys.stderr)
return 0
finally:
await engine.dispose()
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m scribe.restore",
description="Restore a Scribe backup (.tar or .json) into an empty instance.",
)
parser.add_argument("file", type=Path, help="the backup file")
args = parser.parse_args(argv)
return asyncio.run(run(args.file))
if __name__ == "__main__":
sys.exit(main())
+15 -1
View File
@@ -59,7 +59,21 @@ async def backup():
@admin_bp.route("/restore", methods=["POST"])
@admin_required
async def restore():
data = await request.get_json()
"""Restore from an uploaded backup: the JSON download (application/json)
or the scheduled backup's tar (sent as the raw body). The upload is held
to the app's request-size limit; `python -m scribe.restore` has none, and
is the way in for a bigger file or a fresh instance."""
if request.mimetype == "application/json":
data = await request.get_json()
else:
import io
from scribe.services.backup_file import ArchiveError, read_archive
try:
data = read_archive(io.BytesIO(await request.get_data()))
except ArchiveError as exc:
return jsonify({"error": f"That file isn't a Scribe backup: {exc}"}), 400
if not data:
return jsonify({"error": "No backup data provided"}), 400
@@ -0,0 +1,103 @@
"""Real-Postgres proof that the scheduled backup's FILE restores (milestone
523): a payload written as the tar, read back through `scribe.restore`'s
loader, and fed to the real `restore_full_backup`.
The unit lane already shows the tar hands back the dict it was given. What
only this lane shows is that the dict it hands back is one the restore
accepts — every id remapped to the restored rows, nothing dropped — and that
the command-line restore refuses a database that already has accounts.
"""
import pytest
import pytest_asyncio
from sqlalchemy import select
from scribe.models import async_session
from scribe.models.milestone import Milestone
from scribe.models.note import Note
from scribe.models.project import Project
from scribe.models.user import User
from scribe.restore import load_backup, run
from scribe.services import backup
from scribe.services.backup_file import write_archive
from tests.helpers import ensure_user
pytestmark = [pytest.mark.integration, pytest.mark.usefixtures("_dispose_engine")]
OWNER = "backup_file_restore_owner"
RESTORED = "backup_file_restore_restored"
async def _purge(username: str) -> None:
async with async_session() as s:
for user in (await s.execute(select(User).where(User.username == username))).scalars().all():
for model in (Note, Milestone, Project):
for row in (await s.execute(
select(model).where(model.user_id == user.id)
)).scalars().all():
await s.delete(row)
await s.flush()
await s.delete(user)
await s.commit()
@pytest_asyncio.fixture(autouse=True)
async def _no_leftovers():
"""Setup only — see the rule_version roundtrip module for why."""
await _purge(RESTORED)
await _purge(OWNER)
@pytest_asyncio.fixture
async def written(tmp_path):
"""A project with a milestone and a task in it, exported with the real
row builders, renamed so the restore mints a separate user, and written
as the backup tar."""
async with async_session() as s:
user = await ensure_user(s, OWNER)
project = Project(user_id=user.id, title="Restore drill")
s.add(project)
await s.flush()
milestone = Milestone(user_id=user.id, project_id=project.id, title="Plan")
s.add(milestone)
await s.flush()
note = Note(user_id=user.id, project_id=project.id, milestone_id=milestone.id,
title="Prove the restore", body="From the tar, not the JSON.",
status="todo", note_type="note", task_kind="work")
s.add(note)
await s.commit()
payload = {
"version": backup.BACKUP_VERSION,
"scope": "full",
"users": backup._user_rows([user]),
"projects": backup._project_rows([project]),
"milestones": backup._milestone_rows([milestone]),
"notes": backup._note_rows([note]),
}
payload["users"][0]["username"] = RESTORED
from datetime import datetime, timezone
path = write_archive(payload, tmp_path, datetime.now(timezone.utc), app_version="test")
yield path
await _purge(RESTORED)
await _purge(OWNER)
async def test_the_tar_restores_with_every_id_remapped(written):
await backup.restore_full_backup(load_backup(written))
async with async_session() as s:
user = (await s.execute(select(User).where(User.username == RESTORED))).scalars().one()
project = (await s.execute(select(Project).where(Project.user_id == user.id))).scalars().one()
milestone = (await s.execute(select(Milestone).where(Milestone.user_id == user.id))).scalars().one()
note = (await s.execute(select(Note).where(Note.user_id == user.id))).scalars().one()
assert project.title == "Restore drill"
assert (note.project_id, note.milestone_id) == (project.id, milestone.id)
assert note.body == "From the tar, not the JSON."
assert note.status == "todo"
async def test_the_command_refuses_a_database_with_accounts(written):
"""The source user exists, so this database is not empty: the command
must stop before restoring anything."""
assert await run(written) == 3
async with async_session() as s:
restored = (await s.execute(select(User).where(User.username == RESTORED))).scalars().all()
assert restored == []