bvandeusen
66f19d67f5
Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
41 lines
970 B
Plaintext
41 lines
970 B
Plaintext
# Web
|
|
quart>=0.20,<0.21
|
|
hypercorn>=0.18,<0.19
|
|
|
|
# DB
|
|
sqlalchemy[asyncio]>=2.0,<2.1
|
|
asyncpg>=0.31,<0.32
|
|
psycopg[binary]>=3.3,<3.4
|
|
alembic>=1.18,<1.19
|
|
pgvector>=0.4,<0.5
|
|
|
|
# Task queue
|
|
celery>=5.6,<5.7
|
|
redis>=7.4,<8.0
|
|
|
|
# Crypto for credential storage (lands in FC-3, but pinned now for stability)
|
|
cryptography>=48,<49
|
|
|
|
# Image handling (lands in FC-2)
|
|
pillow>=12,<13
|
|
imagehash>=4.3,<4.4
|
|
|
|
# Gallery-dl wrapper (lands in FC-3)
|
|
gallery-dl>=1.32,<1.33
|
|
# Video extractor backend for gallery-dl. Without it, every video post
|
|
# attachment fails with `[downloader.ytdl][error] Cannot import yt-dlp`
|
|
# → `[download][error] Failed to download NN_video.mp4`. Operator-flagged
|
|
# 2026-05-31 after Patreon video posts produced empty downloads.
|
|
yt-dlp>=2025.1
|
|
|
|
# Utilities
|
|
python-dotenv>=1.2,<2.0
|
|
structlog>=25.5,<26.0
|
|
|
|
# HTML sanitization for scraped post descriptions (FC-2d provenance)
|
|
nh3>=0.2,<0.3
|
|
|
|
# Archive extraction (FC-2d-iii filesystem-import aid)
|
|
rarfile>=4.2,<5
|
|
py7zr>=1,<2
|