14 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	14936b62dc	refactor: retire the images tarball backup ... Images and attachments are backed up off-site, one object per file, so the tar+zstd of /images goes: its service and restore, both tasks, the trigger and restore routes, the Keep last setting, the screen's Images section, and the tests. Migration 0122 drops backup_run.tar_path and import_settings.backup_images_keep_last_n and deletes any images runs. The local database dump stays for an install with no off-site destination, and the Backups card now says plainly that images and attachments are backed up only off-site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:07:44 -05:00
bvandeusenandClaude Opus 5.5	75008ad048	feat: test a database backup without touching the install ... "Test a backup" on the Backups card loads a dump into a separate, temporary database on the same Postgres server, the same way a restore does, checks every table against the row counts recorded with the dump, brings the copy up to this version, and drops it. The live database is only read. Every dump now records its row counts, read in the snapshot pg_dump dumps from (pg_export_snapshot + --snapshot), so the comparison is exact. Off-site the counts sit beside the dump as <name>.counts.json; locally they go in the manifest file. A dump taken before this is compared with the live database instead. A test cut off by a restart is ended, and its database dropped, when the long maintenance lane starts. A test doesn't block backups; a restore and a second test wait for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:21:25 -05:00
bvandeusenandClaude Opus 5.5	66045794f0	feat: the Backups screen says whether backups are keeping up, and the System view warns when they aren't ... - A Backups row on Settings → System and the nav dot turn amber when the newest database dump is more than a day old or failed, or the weekly bucket check found files missing that no media run has sent since. No row when backups aren't set up. - The weekly bucket check is kept as a run (kind "reconcile") with its counts, trimmed to the last 12, and listed on the Backups card. - The Backups card leads with the same verdict and the newest dump's size, shows bytes beside the running media job's file count, and lets each run open its list of failed files with why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 17:51:39 -05:00
bvandeusenandClaude Opus 5.5	1e63f70bf7	fix: a media backup or restore cut off by a restart carries on where it stopped ... A media backup and a restore's file phase run as chains of 45-minute chunks, each queuing the next when it ends. A restart killed the chunk in flight, so nothing queued the next one; the run's row stayed "running", "Back up now" skipped it as already running, and only the 7-hour stall sweep ended it. - When the long-maintenance lane starts, it queues the next chunk of every media run still running and every restore fetching its files. - Each chunk writes its token to the run and hands it to the chunk it queues. A chunk queued by any other chunk is dropped (a killed chunk's message redelivered by acks_late, say), so a resume never doubles the chain; a chunk taken over stops at its next file. - The integration job installs postgresql-client when the CI image lacks it: the backup round trip in 375e7e2 runs the real pg_dump and pg_restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:53:57 -05:00
bvandeusenandClaude Opus 5.5	375e7e2eda	feat: restore a new install from the off-site bucket, database first, then every file ... Settings → Maintenance → Backups gains "Restore from the bucket": pick one of the bucket's dated dumps, type its confirmation, and FabledCurator saves the current database to the bucket, replaces it with the dump, migrates it to this version, then fetches every image and attachment back to the path its row records. Each file is checked against its sha256 on arrival and moved into place only then; a file already on disk is left alone, so "Fetch missing files" repairs a library that lost files without touching the database. - services/backup_restore.py: list/find/fetch dumps, migrate, keep this install's destination (the dump's sealed secret can't be opened here), library_files work list, restore_file (one fetch per sha, copies for duplicate paths, refuses paths outside /images). - restore_offsite_task: the database phase, then 45-minute file chunks with a heartbeat; a failed load leaves an empty migrated database and says why. The dump's own run row is marked ok, other runs then going "Interrupted". - backup_s3.get_file: sha/size check on arrival, ObjectMissing for a 404, download pacing. - backup_service: every dump leaves the credential table's rows out; restore_db reports pg_restore's last line, stops at the first error, ends other connections and waits at most 60 s for locks. - Backups and orphan repair wait while a restore runs; the stall sweep judges restores by heartbeat; restore runs are pruned to 20. - CLI: python -m backend.app.scripts.restore list|database|files. - README: backup and restore section; logins are in no backup. - Tests: a round trip through an in-process S3 bucket (tests/fake_s3.py) with real pg_dump/pg_restore, plus file, run, API and get_file cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:34:47 -05:00
bvandeusenandClaude Opus 5.5	0cf8edaa10	feat: images and attachments go to the off-site bucket once each, by fingerprint ... With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:07:15 -05:00
bvandeusenandClaude Opus 5.5	772482d794	feat: with an off-site destination set up, database backups stream to it ... The nightly database backup (and "Run DB backup now") now sends pg_dump's output straight to the off-site bucket as a multipart upload, with no copy staged on this server. The object is named database/fabledcurator-<UTC time>.dump and appears only once pg_dump has exited cleanly and every part has landed; a failed dump aborts the upload and records pg_dump's last line. Without a destination, dumps are written to /images/_backups as before. - backup_s3: Upload (start / add part / complete / abort), with a busy or silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and a Pacer that holds uploads to the new bandwidth limit. - Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default, 0 for none (migration 0120). - pg_dump runs under nice (and ionice where installed), locally too. - Backups card says where dumps go; the runs table gains "Saved to" and shows a failed run's error. Off-site rows have no restore/tag/delete here: restoring from the bucket is a later step (#6880). Task #6878, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:11:12 -05:00
bvandeusenandClaude Opus 4.8	a183be7e6e	fix(infra): bump Postgres shm_size (vacuum DiskFull) + raise DB-backup time limit ... Two more maintenance-queue failures from the operator's 24h list: - vacuum_analyze died with "could not resize shared memory segment to 67MB: No space left on device" — Docker's default /dev/shm is 64MB, too small for VACUUM (ANALYZE)'s parallel-worker shared memory. Set the postgres service shm_size: 512m. - backup_db_task timed out at its 12-min limit once the DB grew; a pg_dump can't be chunked, so raise it to 30/35 min. (A long backup still briefly holds the concurrency-1 lane — the structural fix is a dedicated lane for long one-shots.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 00:08:19 -04:00
bvandeusen	e30f50e6fe	fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts ... Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days).	2026-06-02 14:30:46 -04:00
bvandeusenandClaude Opus 4.7	171c486939	refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row ... The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was repeated 15× across services, API, and 5 task modules. Added async load() + sync load_sync() classmethods on the model and migrated all 15 full-row sites (callers already imported ImportSettings, so no new imports; dropped download's now-orphaned select import). Left maintenance.py's deliberate column-select (import_scan_path only) as-is. Rest of the service layer was already adequately DRY — the Record/to_dict pattern is only 2 instances and the savepoint find-or-create recovery is correctly per-entity, so neither was forced into a shared abstraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 14:03:26 -04:00
bvandeusenandClaude Opus 4.7	718cc79905	fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 23:37:06 -04:00
bvandeusenandClaude Opus 4.7	06d527cb92	fc3h: prune_backups (daily retention) + backup_db_nightly (hourly tick, settings-gated) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 22:55:19 -04:00
bvandeusenandClaude Opus 4.7	e9ea376aed	fc3h: restore_db_task + restore_images_task — restore creates 'restoring' marker row linked via restored_from_id ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 22:53:50 -04:00
bvandeusenandClaude Opus 4.7	882cb491ba	fc3h: backup_db_task + backup_images_task Celery tasks ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 22:53:26 -04:00