fix: downloads stop leaving files on disk that the library doesn't use
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

- An archive whose images were extracted was kept at its download path
  as well as in the attachments store. Both download paths now remove
  the downloaded copy.
- A downloaded file the import filters reject (too small, transparent,
  single colour, unreadable) was left on disk with no record. A later
  walk found it, imported it again and rejected it again, every time.
  Rejected files are now removed; the seen ledger keeps them from being
  downloaded again.
- Both paths share one policy, Importer.discard_after_attach, which
  never removes a file an ImageRecord points at.
- An external-link fetch that fails, or whose worker was killed, left
  partial files (megatools' .megatmp.*) in the link's dir, and a killed
  fetch left its link 'downloading' for good. A fetch now clears
  unrecorded leftovers before starting and after failing. The sweep
  returns a link to retry once it has been 'downloading' past the
  task's hard limit, using a new external_link.started_at column
  (migration 0117).
- The orphaned temp-file sweep now recognises .megatmp.* as well as
  .part and .partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 18:00:31 -05:00
co-authored by Claude Opus 5.5
parent 47fb3f2193
commit 157f2803c0
9 changed files with 399 additions and 64 deletions
+16 -5
View File
@@ -88,8 +88,18 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# keeps it from deleting an in-flight download's staging file mid-write.
IMAGES_ROOT = Path("/images")
TEMP_STAGING_SUFFIXES = (".part", ".partial")
# megatools stages a Mega download as `.megatmp.<id>` in the target dir.
TEMP_STAGING_PREFIXES = (".megatmp.",)
ORPHAN_TEMP_MIN_AGE_HOURS = 6
def is_temp_staging_file(path: Path) -> bool:
"""True for a downloader's staging file: one that never becomes library
content under that name, so a stale one is safe to delete."""
return path.suffix in TEMP_STAGING_SUFFIXES or path.name.startswith(
TEMP_STAGING_PREFIXES
)
# Audit 2026-06-02: per-entity recovery sweep thresholds. Each must be
# > the entity's longest legitimate runtime (its task's time_limit + a
# small buffer) so the sweep never flags in-flight work.
@@ -373,16 +383,17 @@ def cleanup_old_tasks() -> int:
@celery.task(name="backend.app.tasks.maintenance.cleanup_orphaned_temp_files")
def cleanup_orphaned_temp_files() -> int:
"""Delete orphaned .part/.partial staging files under the images root, left by
a download/import killed mid-write. Only removes files older than
ORPHAN_TEMP_MIN_AGE_HOURS so an in-flight download's staging file is never
pulled out from under it. Returns the count removed."""
"""Delete orphaned staging files (.part, .partial, megatools' .megatmp.*)
under the images root, left by a download/import killed mid-write. Only
removes files older than ORPHAN_TEMP_MIN_AGE_HOURS so an in-flight
download's staging file is never pulled out from under it. Returns the
count removed."""
if not IMAGES_ROOT.is_dir():
return 0
cutoff = datetime.now(UTC).timestamp() - ORPHAN_TEMP_MIN_AGE_HOURS * 3600
removed = 0
for path in IMAGES_ROOT.rglob("*"):
if path.suffix not in TEMP_STAGING_SUFFIXES or not path.is_file():
if not is_temp_staging_file(path) or not path.is_file():
continue
try:
if path.stat().st_mtime >= cutoff: