The nightly database backup (and "Run DB backup now") now sends pg_dump's
output straight to the off-site bucket as a multipart upload, with no copy
staged on this server. The object is named database/fabledcurator-<UTC
time>.dump and appears only once pg_dump has exited cleanly and every part
has landed; a failed dump aborts the upload and records pg_dump's last line.
Without a destination, dumps are written to /images/_backups as before.
- backup_s3: Upload (start / add part / complete / abort), with a busy or
silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and
a Pacer that holds uploads to the new bandwidth limit.
- Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default,
0 for none (migration 0120).
- pg_dump runs under nice (and ionice where installed), locally too.
- Backups card says where dumps go; the runs table gains "Saved to" and
shows a failed run's error. Off-site rows have no restore/tag/delete here:
restoring from the bucket is a later step (#6880).
Task #6878, milestone 586.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Listing a 2026-05 tarball while investigating the 4.3T `_backups` pile showed
its second and third entries:
images/secrets/
images/secrets/credential_key.b64
That is the key that decrypts the stored Patreon/SubscribeStar session
credentials, and `cookies/` sat beside it — both unexcluded, so this was true
of every images backup taken today, not just the old ones. An images tarball
is supposed to be a media archive; one that carries the operator's account
keys is a credential leak wearing a backup's name, in a single file that is
easy to copy to another disk or restore somewhere less protected. Encryption
at rest buys nothing when the key travels in the same archive.
`secrets` and `cookies` join `_backups` and `_quarantine` in one named tuple,
each with its reason recorded — the recursion that produced 4.3T of nested
tarballs is the cautionary tale for why the list is worth explaining rather
than just listing.
A restore no longer re-establishes credentials. You sign in again, which is
the correct outcome for a media backup.
Tests cover both new names and that every exclude stays root-relative — a bare
`secrets` would also match an artist folder of that name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
DB backup polish (plan-task #764 Q3):
- pg_dump now uses custom format (-Fc): compressed (much smaller on NFS) and
restored via pg_restore. Artifact extension .sql → .dump; restore_db swaps
psql -f for pg_restore -d. BackupRun.sql_path field name kept (it's just the
db artifact path).
- Reconcile the subprocess guardrails: the DB timeout was 720s with a stale
'Celery soft is 10 min' comment, but backup_db_task's soft limit is actually
1800s — so the bounded-kill fired 18 min early. Set DB=1700s / images=21000s,
each just under its task's Celery soft limit so _run_bounded stays the
primary guard (an NFS D-state hang defeats Celery's own SIGKILL).
Real shrink of the DB is the #764 prune; this makes each dump smaller/faster
on top of that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5