Commit Graph
4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 66045794f0 feat: the Backups screen says whether backups are keeping up, and the System view warns when they aren't
CI and images / extension-version (push) Successful in 2s
CI and images / lint (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 45s
CI and images / integration (push) Successful in 2m31s
CI and images / extension-test (push) Successful in 18s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m54s
CI and images / promote (push) Successful in 1s
CI and images / smoke-web (push) Successful in 48s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
- A Backups row on Settings → System and the nav dot turn amber when the
  newest database dump is more than a day old or failed, or the weekly
  bucket check found files missing that no media run has sent since.
  No row when backups aren't set up.
- The weekly bucket check is kept as a run (kind "reconcile") with its
  counts, trimmed to the last 12, and listed on the Backups card.
- The Backups card leads with the same verdict and the newest dump's size,
  shows bytes beside the running media job's file count, and lets each run
  open its list of failed files with why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 17:51:39 -05:00
bvandeusenandClaude Opus 5.5 1e63f70bf7 fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.

- When the long-maintenance lane starts, it queues the next chunk of every
  media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
  A chunk queued by any other chunk is dropped (a killed chunk's message
  redelivered by acks_late, say), so a resume never doubles the chain; a chunk
  taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
  the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:53:57 -05:00
bvandeusenandClaude Opus 5.5 d465198c4a fix: the media backup task has its own stall threshold, and its test seeds a post for a shared archive
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 27s
CI and images / extension-test (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 47s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 2m12s
CI and images / smoke-web (push) Successful in 50s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / promote (push) Successful in 2s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:15:26 -05:00
bvandeusenandClaude Opus 5.5 0cf8edaa10 feat: images and attachments go to the off-site bucket once each, by fingerprint
CI and images / lint (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / extension-version (push) Successful in 2s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
With an off-site destination set up, the library's files are sent to the
bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database
dumps under database/. Each file goes once: a new backup_blob table records
what each destination holds (migration 0121), and a run's work list is every
image and attachment sha with no row yet, so runs are incremental and a new
bucket starts its copy from the beginning.

- The bucket checks each file against the sha recorded at import; a file
  over 1 GiB goes in parts and is checked before the upload is completed.
  A file missing or changed on disk is reported, not stored.
- A run is chunked: each 45-minute chunk commits after every file and queues
  the next, so other long-maintenance jobs get turns. One run at a time.
  The stall sweep judges a media run by its progress heartbeat
  (backup_run.last_progress_at), not its start.
- A file that can't be sent is listed on the run (status ok, with the error
  line) and tried again by the next run. A refused put for an object
  already in an append-only bucket counts as sent.
- Nightly, with a destination set, runs the dump and then the media.
- Weekly reconcile lists media/ and re-sends what the bucket lost, and
  adopts what it already holds (a restored database sends nothing twice).
- Uploads use the bandwidth limit; read transactions end before transfers.
- Backups card: with a destination, an "Images and attachments" section
  shows coverage (files and bytes in the bucket), "Send new files now", and
  the media runs. Without one, the local tarball section is unchanged.
- Media run history keeps the latest 60 rows.

Task #6879, milestone 586.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:07:15 -05:00