feat(external): download worker for file-host links (Phase 4b)
tasks/external.py drives the external_link ledger: - fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern; requeue-with-countdown if busy), fetch via external_fetch into the artist library tree, then route each file through importer.attach_in_place via a synthesized sidecar so it links to the SAME post (archive→ImageRecords, else→PostAttachment; on-disk original removed for captured files, art stays); thumbnail+ML enqueue for new images; status downloaded | failed | dead with attempts/last_error/completed_at/duration. - sweep_external_links(): enqueue a bounded batch of actionable links. - recover_external_links() + prune_external_links(): recovery + retention (#89). - per-host enable read via getattr (forward-compatible; Settings UI adds the columns in 4d — defaults on, rule #26). Wiring: celery include + route (download lane) + beat (sweep 10m, recover + prune daily); download_service phase 3 enqueues a sweep after recording links. Integration tests: download+attach, failure, dead-letter, non-claimable, sweep. mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -396,6 +396,13 @@ class DownloadService:
|
||||
|
||||
await loop.run_in_executor(None, _upsert)
|
||||
|
||||
# Kick the off-platform file-host downloader for any links this run
|
||||
# recorded (mega/gdrive/…). Global + idempotent (only claims pending/
|
||||
# retryable rows); the beat sweep is the backstop. Lazy import dodges a
|
||||
# task-module import cycle.
|
||||
from ..tasks.external import sweep_external_links
|
||||
sweep_external_links.delay()
|
||||
|
||||
ev = (await self.async_session.execute(
|
||||
select(DownloadEvent).where(DownloadEvent.id == event_id)
|
||||
)).scalar_one()
|
||||
|
||||
Reference in New Issue
Block a user