fix: downloads stop leaving files on disk that the library doesn't use
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

- An archive whose images were extracted was kept at its download path
  as well as in the attachments store. Both download paths now remove
  the downloaded copy.
- A downloaded file the import filters reject (too small, transparent,
  single colour, unreadable) was left on disk with no record. A later
  walk found it, imported it again and rejected it again, every time.
  Rejected files are now removed; the seen ledger keeps them from being
  downloaded again.
- Both paths share one policy, Importer.discard_after_attach, which
  never removes a file an ImageRecord points at.
- An external-link fetch that fails, or whose worker was killed, left
  partial files (megatools' .megatmp.*) in the link's dir, and a killed
  fetch left its link 'downloading' for good. A fetch now clears
  unrecorded leftovers before starting and after failing. The sweep
  returns a link to retry once it has been 'downloading' past the
  task's hard limit, using a new external_link.started_at column
  (migration 0117).
- The orphaned temp-file sweep now recognises .megatmp.* as well as
  .part and .partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 18:00:31 -05:00
co-authored by Claude Opus 5.5
parent 47fb3f2193
commit 157f2803c0
9 changed files with 399 additions and 64 deletions
+9 -30
View File
@@ -369,11 +369,8 @@ class DownloadService:
if do_embed:
embed_image.delay(img_id)
elif result.status == "attached":
# Non-media or extracted archive captured as PostAttachment
# (FC-2d-iii). The canonical copy lives in the attachments
# store; the original download path is now redundant —
# mirror duplicate_hash cleanup so we don't keep two copies.
# Operator-flagged 2026-06-02 (Lustria OST zip).
# Non-media, or an archive with no new images, captured as a
# PostAttachment; the attachments store holds the copy.
import_summary["attached"] += 1
# An archive captured WITHOUT extracting any image carries a
# reason on result.error — record it so the event explains the
@@ -390,35 +387,13 @@ class DownloadService:
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
except OSError:
pass
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped" and result.skip_reason and result.skip_reason.value in (
"duplicate_hash", "duplicate_phash",
):
import_summary["skipped"] += 1
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "skipped":
# Soft skip (too_small, too_transparent, invalid_image) —
# the file just didn't qualify, not a download/ingest
# failure. Don't flag the run as error; the file stays
# on disk for operator inspection.
# A duplicate, or a file the import filters rejected. Neither
# is a download failure, so the run is not flagged.
import_summary["skipped"] += 1
elif result.status == "failed":
# Hard failure (today only: archive probe crash/timeout).
# The original archive sits in /images/ as an orphan; the
# filesystem scanner would re-import and re-crash on the
# same file, so delete the source file and surface the
# error in import_summary. Audit 2026-06-02.
# Hard failure: an archive probe crashed or timed out.
import_summary["errors"] += 1
try:
path.unlink(missing_ok=True) # noqa: ASYNC240
except OSError:
pass
elif result.status == "refreshed":
# Currently unreachable from attach_in_place (the download
# path never runs in deep=True mode), but the importer's
@@ -427,6 +402,10 @@ class DownloadService:
import_summary["attached"] += 1
else:
import_summary["errors"] += 1
# Remove the download when the library no longer needs it at this
# path: an attachment or archive now in the store, a duplicate, a
# rejected file. One policy for every download path.
self.importer.discard_after_attach(path, result)
# Post-only records: media-less posts (pure-text) the native ingester
# captured so the artist archive is complete. Upsert each (keyed on