fix: downloads stop leaving files on disk that the library doesn't use
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
- An archive whose images were extracted was kept at its download path as well as in the attachments store. Both download paths now remove the downloaded copy. - A downloaded file the import filters reject (too small, transparent, single colour, unreadable) was left on disk with no record. A later walk found it, imported it again and rejected it again, every time. Rejected files are now removed; the seen ledger keeps them from being downloaded again. - Both paths share one policy, Importer.discard_after_attach, which never removes a file an ImageRecord points at. - An external-link fetch that fails, or whose worker was killed, left partial files (megatools' .megatmp.*) in the link's dir, and a killed fetch left its link 'downloading' for good. A fetch now clears unrecorded leftovers before starting and after failing. The sweep returns a link to retry once it has been 'downloading' past the task's hard limit, using a new external_link.started_at column (migration 0117). - The orphaned temp-file sweep now recognises .megatmp.* as well as .part and .partial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -369,11 +369,8 @@ class DownloadService:
|
||||
if do_embed:
|
||||
embed_image.delay(img_id)
|
||||
elif result.status == "attached":
|
||||
# Non-media or extracted archive captured as PostAttachment
|
||||
# (FC-2d-iii). The canonical copy lives in the attachments
|
||||
# store; the original download path is now redundant —
|
||||
# mirror duplicate_hash cleanup so we don't keep two copies.
|
||||
# Operator-flagged 2026-06-02 (Lustria OST zip).
|
||||
# Non-media, or an archive with no new images, captured as a
|
||||
# PostAttachment; the attachments store holds the copy.
|
||||
import_summary["attached"] += 1
|
||||
# An archive captured WITHOUT extracting any image carries a
|
||||
# reason on result.error — record it so the event explains the
|
||||
@@ -390,35 +387,13 @@ class DownloadService:
|
||||
bytes_downloaded += path.stat().st_size # noqa: ASYNC240
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
path.unlink(missing_ok=True) # noqa: ASYNC240
|
||||
except OSError:
|
||||
pass
|
||||
elif result.status == "skipped" and result.skip_reason and result.skip_reason.value in (
|
||||
"duplicate_hash", "duplicate_phash",
|
||||
):
|
||||
import_summary["skipped"] += 1
|
||||
try:
|
||||
path.unlink(missing_ok=True) # noqa: ASYNC240
|
||||
except OSError:
|
||||
pass
|
||||
elif result.status == "skipped":
|
||||
# Soft skip (too_small, too_transparent, invalid_image) —
|
||||
# the file just didn't qualify, not a download/ingest
|
||||
# failure. Don't flag the run as error; the file stays
|
||||
# on disk for operator inspection.
|
||||
# A duplicate, or a file the import filters rejected. Neither
|
||||
# is a download failure, so the run is not flagged.
|
||||
import_summary["skipped"] += 1
|
||||
elif result.status == "failed":
|
||||
# Hard failure (today only: archive probe crash/timeout).
|
||||
# The original archive sits in /images/ as an orphan; the
|
||||
# filesystem scanner would re-import and re-crash on the
|
||||
# same file, so delete the source file and surface the
|
||||
# error in import_summary. Audit 2026-06-02.
|
||||
# Hard failure: an archive probe crashed or timed out.
|
||||
import_summary["errors"] += 1
|
||||
try:
|
||||
path.unlink(missing_ok=True) # noqa: ASYNC240
|
||||
except OSError:
|
||||
pass
|
||||
elif result.status == "refreshed":
|
||||
# Currently unreachable from attach_in_place (the download
|
||||
# path never runs in deep=True mode), but the importer's
|
||||
@@ -427,6 +402,10 @@ class DownloadService:
|
||||
import_summary["attached"] += 1
|
||||
else:
|
||||
import_summary["errors"] += 1
|
||||
# Remove the download when the library no longer needs it at this
|
||||
# path: an attachment or archive now in the store, a duplicate, a
|
||||
# rejected file. One policy for every download path.
|
||||
self.importer.discard_after_attach(path, result)
|
||||
|
||||
# Post-only records: media-less posts (pure-text) the native ingester
|
||||
# captured so the artist archive is complete. Upsert each (keyed on
|
||||
|
||||
Reference in New Issue
Block a user