DB backup polish (plan-task #764 Q3):
- pg_dump now uses custom format (-Fc): compressed (much smaller on NFS) and
restored via pg_restore. Artifact extension .sql → .dump; restore_db swaps
psql -f for pg_restore -d. BackupRun.sql_path field name kept (it's just the
db artifact path).
- Reconcile the subprocess guardrails: the DB timeout was 720s with a stale
'Celery soft is 10 min' comment, but backup_db_task's soft limit is actually
1800s — so the bounded-kill fired 18 min early. Set DB=1700s / images=21000s,
each just under its task's Celery soft limit so _run_bounded stays the
primary guard (an NFS D-state hang defeats Celery's own SIGKILL).
Real shrink of the DB is the #764 prune; this makes each dump smaller/faster
on top of that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
(D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
(NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
writes (killable) instead of an NFS write that hangs. backup_images keeps
bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
images' 7h), so a hung DB backup is flipped to error promptly.
#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.
Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen