Use `megatools dl` (Debian-native apt package) for mega.nz public links rather
than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds
`megatools` to the runtime Dockerfile; the fetcher's mega backend now shells
`megatools dl --path <dir> <url>` (key in the #fragment is preserved by the
extractor). gdown (gdrive) is already a pip dep in the runtime image.
NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile
change is verified on the next dev→main merge, not by this dev push. The fetcher
code path is unit-tested via the mocked _run_mega_get seam.
With this, all 5 hosts download end-to-end once a celery download-worker runs.
Refs FC #830 (Phase 4c).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared, reusable fetchers for the 5 off-platform hosts behind one signature
(fetch_external(host, url, dest_dir, ...) -> FetchResult):
- dropbox : force dl=1 + stream GET
- pixeldrain : GET /api/file/{id}
- mediafire : scrape the download page for the direct link + stream GET
- gdrive : gdown (confirm-token + virus-scan interstitial); added to reqs
- mega : MEGAcmd `mega-get` subprocess (public link incl. #key)
HTTP/gdown/subprocess go through module seams so unit tests run without
network/gdown/MEGAcmd. fetch_external never raises — every backend failure
(transport, non-200, scrape miss, subprocess error, stop) is captured on
.error so the worker (next slice) records it and moves on. mega's binary lands
in the runtime image in a later slice; the code is complete + tested now.
Refs FC #830 (Phase 4a).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three coupled operator-reported pains from the 2026-05-31 download
event audit:
1. `[patreon][warning] Not allowed to view post N` was bubbling up as
an error event, bumping consecutive_failures and parking the source
in "needs attention." The classifier's tier-gated branch was gated
on `return_code in (1, 4)`. Gallery-dl returns a different exit
code for mixed-failure runs (e.g. paywall warnings + a missing
yt-dlp dep flipping the exit bits), so the branch never fired and
the path fell through to UNKNOWN_ERROR. Widen the gate: when no
source-level error fired AND tier-gated warnings are present,
classify as TIER_LIMITED regardless of return code.
2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
"stranded by recovery sweep (no terminal status after time_limit)"
+ empty stdout/stderr. Root cause: subprocess.run timeout (900s)
and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
wiped the in-memory captured output and the DownloadEvent ended up
empty-logged 18 minutes later when the sweep finalized it. Drop
gallery-dl's default subprocess timeout to 870s — a 30s margin
shy of Celery's soft limit — so subprocess.TimeoutExpired always
wins the race and captures the partial stdout/stderr via the
existing handler.
3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
firing on every video attachment, causing per-item download
failures that masked legitimate tier-gated classification.
Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
video posts download normally and the per-item failure noise
disappears.
Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
CI failed at 'pip install -r requirements.txt' with Pillow 10.4.0 sdist
building from source under Python 3.14 and exiting with KeyError: '__version__'
(no 3.14 wheel exists for the 10.x line). The same risk applied to
asyncpg 0.29 (also no 3.14 wheel) and to psycopg 3.1.x (dep resolver was
backtracking through every minor trying to find a 3.14-compatible combo).
Bumps:
- pillow: 10.2-10.x -> 11.1-11.x (11.x has 3.13+ wheels)
- asyncpg: 0.29 -> 0.30 (0.30 added 3.13/3.14 wheels)
- psycopg: 3.1.x -> 3.2.x (current line, stable resolver)
- cryptography: 42 -> 44-45 (current abi3 wheels; the 42 line
worked via abi3 but bumping aligns
with the newer cffi-on-3.14 path)
This is the operator-flagged risk from FC-1 ("scientific stack may still be
catching up on 3.14 wheels — worth confirming before locking in"), realized.
The fix is to pin newer minimums, not to back off to 3.13.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pins runtime and ML deps separately so the regular web image stays lean.
Configures ruff for py312 with bugbear, async, and pyupgrade lints enabled.
psycopg sync driver included up-front for alembic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen