feat(external): download worker for file-host links (Phase 4b)
tasks/external.py drives the external_link ledger: - fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern; requeue-with-countdown if busy), fetch via external_fetch into the artist library tree, then route each file through importer.attach_in_place via a synthesized sidecar so it links to the SAME post (archive→ImageRecords, else→PostAttachment; on-disk original removed for captured files, art stays); thumbnail+ML enqueue for new images; status downloaded | failed | dead with attempts/last_error/completed_at/duration. - sweep_external_links(): enqueue a bounded batch of actionable links. - recover_external_links() + prune_external_links(): recovery + retention (#89). - per-host enable read via getattr (forward-compatible; Settings UI adds the columns in 4d — defaults on, rule #26). Wiring: celery include + route (download lane) + beat (sweep 10m, recover + prune daily); download_service phase 3 enqueues a sweep after recording links. Integration tests: download+attach, failure, dead-letter, non-claimable, sweep. mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30,6 +30,7 @@ def make_celery() -> Celery:
|
||||
"backend.app.tasks.maintenance",
|
||||
"backend.app.tasks.ml",
|
||||
"backend.app.tasks.download",
|
||||
"backend.app.tasks.external",
|
||||
"backend.app.tasks.backup",
|
||||
"backend.app.tasks.admin",
|
||||
"backend.app.tasks.library_audit",
|
||||
@@ -42,6 +43,9 @@ def make_celery() -> Celery:
|
||||
"backend.app.tasks.ml.*": {"queue": "ml"},
|
||||
"backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
|
||||
"backend.app.tasks.download.*": {"queue": "download"},
|
||||
# External file-host fetches are downloads — same lane (they can run
|
||||
# long, but the download worker already tolerates long backfills).
|
||||
"backend.app.tasks.external.*": {"queue": "download"},
|
||||
"backend.app.tasks.scan.*": {"queue": "scan"},
|
||||
# `maintenance` is the QUICK lane — recovery sweeps, vacuum, cleanup
|
||||
# (concurrency-1 on the scheduler). The long one-shots (DB backups,
|
||||
@@ -150,6 +154,21 @@ def make_celery() -> Celery:
|
||||
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
|
||||
"schedule": 86400.0,
|
||||
},
|
||||
# External file-host downloads (#830): a steady sweep catches links
|
||||
# the post-download hook missed (worker down, etc.); recovery re-tries
|
||||
# dead links daily; retention prunes long-dead rows.
|
||||
"extdl-sweep": {
|
||||
"task": "backend.app.tasks.external.sweep_external_links",
|
||||
"schedule": 600.0, # every 10 min
|
||||
},
|
||||
"extdl-recover-daily": {
|
||||
"task": "backend.app.tasks.external.recover_external_links",
|
||||
"schedule": 86400.0,
|
||||
},
|
||||
"extdl-prune-daily": {
|
||||
"task": "backend.app.tasks.external.prune_external_links",
|
||||
"schedule": 86400.0,
|
||||
},
|
||||
},
|
||||
timezone="UTC",
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user