External downloads import IN PLACE, so the post-attach dedup-skip unlink could
delete a file that IS an ImageRecord's backing file — orphaning the record and
404-ing on playback. Two sources of that:
- Two links on the same post (same film from mega + gdrive) emitted the same
filename into one external/<post_id>/ dir; the second overwrote the first.
Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path.
- The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it
path-safe: only unlink when `f` is NOT the existing record's canonical file.
Plus an operator-triggered orphan-repair maintenance task
(prune_missing_file_records_task) to clean up records already orphaned by the
bug: scans ImageRecords, deletes those whose file is gone (cascade), with an
NFS-stall guard that aborts without deleting if a large sample is mostly missing.
Wired through POST /api/admin/maintenance/prune-missing-files and a
MissingFileRepairCard in the Maintenance panel.
Tests: refetch-same-link keeps the canonical file; orphan repair deletes only
real orphans and aborts on the mostly-missing guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).
- _route_files now mirrors download_service._phase3_persist branch-for-branch:
imported/superseded → collect member_image_ids+image_id (provenance-linked via
the synthesized sidecar, same as extracted-zip members) → caller enqueues
tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
fetch result (files/bytes) or fetch failure, per-file import decision, dead-
letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
exactly the member images.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
requeue-with-countdown if busy), fetch via external_fetch into the artist
library tree, then route each file through importer.attach_in_place via a
synthesized sidecar so it links to the SAME post (archive→ImageRecords,
else→PostAttachment; on-disk original removed for captured files, art stays);
thumbnail+ML enqueue for new images; status downloaded | failed | dead with
attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
columns in 4d — defaults on, rule #26).
Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.
mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen