Operator-requested: a worker download must be tagged + provenance-associated
exactly like an extracted zip, and the path must log well (we won't get it right
first try).
- _route_files now mirrors download_service._phase3_persist branch-for-branch:
imported/superseded → collect member_image_ids+image_id (provenance-linked via
the synthesized sidecar, same as extracted-zip members) → caller enqueues
tag_and_embed + generate_thumbnail; attached → drop on-disk original, and warn
on an UNEXTRACTED archive (#718 symptom); skipped duplicate → unlink; failed →
unlink + warn.
- Logging at every stage: start (link/host/post/artist/attempt/url), requeue,
fetch result (files/bytes) or fetch failure, per-file import decision, dead-
letter transitions, and done (files/images/duration).
- Parity test: an archive downloaded by the worker is extracted, provenance-
linked to the SAME post, and tag_and_embed+generate_thumbnail are queued for
exactly the member images.
Refs FC #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
requeue-with-countdown if busy), fetch via external_fetch into the artist
library tree, then route each file through importer.attach_in_place via a
synthesized sidecar so it links to the SAME post (archive→ImageRecords,
else→PostAttachment; on-disk original removed for captured files, art stays);
thumbnail+ML enqueue for new images; status downloaded | failed | dead with
attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
columns in 4d — defaults on, rule #26).
Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.
mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen