bvandeusen
13253b18d1
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
46 lines
1.2 KiB
Plaintext
46 lines
1.2 KiB
Plaintext
# Web
|
|
quart>=0.20,<0.21
|
|
hypercorn>=0.18,<0.19
|
|
|
|
# DB
|
|
sqlalchemy[asyncio]>=2.0,<2.1
|
|
asyncpg>=0.31,<0.32
|
|
psycopg[binary]>=3.3,<3.4
|
|
alembic>=1.18,<1.19
|
|
pgvector>=0.4,<0.5
|
|
|
|
# Task queue
|
|
celery>=5.6,<5.7
|
|
redis>=7.4,<8.0
|
|
|
|
# Crypto for credential storage (lands in FC-3, but pinned now for stability)
|
|
cryptography>=48,<49
|
|
|
|
# Image handling (lands in FC-2)
|
|
pillow>=12,<13
|
|
imagehash>=4.3,<4.4
|
|
|
|
# Gallery-dl wrapper (lands in FC-3)
|
|
gallery-dl>=1.32,<1.33
|
|
# Video extractor backend for gallery-dl. Without it, every video post
|
|
# attachment fails with `[downloader.ytdl][error] Cannot import yt-dlp`
|
|
# → `[download][error] Failed to download NN_video.mp4`. Operator-flagged
|
|
# 2026-05-31 after Patreon video posts produced empty downloads.
|
|
yt-dlp>=2025.1
|
|
|
|
# Utilities
|
|
python-dotenv>=1.2,<2.0
|
|
structlog>=25.5,<26.0
|
|
|
|
# HTML sanitization for scraped post descriptions (FC-2d provenance)
|
|
nh3>=0.2,<0.3
|
|
|
|
# Archive extraction (FC-2d-iii filesystem-import aid)
|
|
rarfile>=4.2,<5
|
|
py7zr>=1,<2
|
|
|
|
# Google Drive fetcher for off-platform file-host links (external downloads,
|
|
# #830). Handles Drive's confirm-token + virus-scan interstitial. mega.nz uses
|
|
# the MEGAcmd `mega-get` binary instead (added to the runtime image, not pip).
|
|
gdown>=5,<6
|