fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
This commit is contained in:
@@ -22,6 +22,11 @@ imagehash>=4.3,<4.4
|
||||
|
||||
# Gallery-dl wrapper (lands in FC-3)
|
||||
gallery-dl>=1.32,<1.33
|
||||
# Video extractor backend for gallery-dl. Without it, every video post
|
||||
# attachment fails with `[downloader.ytdl][error] Cannot import yt-dlp`
|
||||
# → `[download][error] Failed to download NN_video.mp4`. Operator-flagged
|
||||
# 2026-05-31 after Patreon video posts produced empty downloads.
|
||||
yt-dlp>=2025.1
|
||||
|
||||
# Utilities
|
||||
python-dotenv>=1.2,<2.0
|
||||
|
||||
Reference in New Issue
Block a user