feat(external): download worker for file-host links (Phase 4b)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Failing after 3m14s

tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
  duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
  requeue-with-countdown if busy), fetch via external_fetch into the artist
  library tree, then route each file through importer.attach_in_place via a
  synthesized sidecar so it links to the SAME post (archive→ImageRecords,
  else→PostAttachment; on-disk original removed for captured files, art stays);
  thumbnail+ML enqueue for new images; status downloaded | failed | dead with
  attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
  columns in 4d — defaults on, rule #26).

Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.

mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 13:44:07 -04:00
parent 13253b18d1
commit 96e984cded
4 changed files with 455 additions and 0 deletions
+19
View File
@@ -30,6 +30,7 @@ def make_celery() -> Celery:
"backend.app.tasks.maintenance",
"backend.app.tasks.ml",
"backend.app.tasks.download",
"backend.app.tasks.external",
"backend.app.tasks.backup",
"backend.app.tasks.admin",
"backend.app.tasks.library_audit",
@@ -42,6 +43,9 @@ def make_celery() -> Celery:
"backend.app.tasks.ml.*": {"queue": "ml"},
"backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
"backend.app.tasks.download.*": {"queue": "download"},
# External file-host fetches are downloads — same lane (they can run
# long, but the download worker already tolerates long backfills).
"backend.app.tasks.external.*": {"queue": "download"},
"backend.app.tasks.scan.*": {"queue": "scan"},
# `maintenance` is the QUICK lane — recovery sweeps, vacuum, cleanup
# (concurrency-1 on the scheduler). The long one-shots (DB backups,
@@ -150,6 +154,21 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
"schedule": 86400.0,
},
# External file-host downloads (#830): a steady sweep catches links
# the post-download hook missed (worker down, etc.); recovery re-tries
# dead links daily; retention prunes long-dead rows.
"extdl-sweep": {
"task": "backend.app.tasks.external.sweep_external_links",
"schedule": 600.0, # every 10 min
},
"extdl-recover-daily": {
"task": "backend.app.tasks.external.recover_external_links",
"schedule": 86400.0,
},
"extdl-prune-daily": {
"task": "backend.app.tasks.external.prune_external_links",
"schedule": 86400.0,
},
},
timezone="UTC",
)