fix: downloads stop leaving files on disk that the library doesn't use
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m3s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

- An archive whose images were extracted was kept at its download path
  as well as in the attachments store. Both download paths now remove
  the downloaded copy.
- A downloaded file the import filters reject (too small, transparent,
  single colour, unreadable) was left on disk with no record. A later
  walk found it, imported it again and rejected it again, every time.
  Rejected files are now removed; the seen ledger keeps them from being
  downloaded again.
- Both paths share one policy, Importer.discard_after_attach, which
  never removes a file an ImageRecord points at.
- An external-link fetch that fails, or whose worker was killed, left
  partial files (megatools' .megatmp.*) in the link's dir, and a killed
  fetch left its link 'downloading' for good. A fetch now clears
  unrecorded leftovers before starting and after failing. The sweep
  returns a link to retry once it has been 'downloading' past the
  task's hard limit, using a new external_link.started_at column
  (migration 0117).
- The orphaned temp-file sweep now recognises .megatmp.* as well as
  .part and .partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 18:00:31 -05:00
co-authored by Claude Opus 5.5
parent 47fb3f2193
commit 157f2803c0
9 changed files with 399 additions and 64 deletions
@@ -0,0 +1,33 @@
"""external_link.started_at: when a fetch claimed the link.
A worker killed mid-fetch (the Celery hard limit, an OOM, a restart) never
reaches the code that moves its link out of 'downloading'. The external-link
sweep returns such a link to the retry path once it has been 'downloading'
longer than the task's hard limit, which needs the claim time.
A row in 'downloading' with no claim time is treated as stale: no worker
survives the deploy that adds this column.
Revision ID: 0117
Revises: 0116
Create Date: 2026-11-05
"""
import sqlalchemy as sa
from alembic import op
revision = "0117"
down_revision = "0116"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"external_link",
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_column("external_link", "started_at")