fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
Three coupled operator-reported pains from the 2026-05-31 download event audit: 1. `[patreon][warning] Not allowed to view post N` was bubbling up as an error event, bumping consecutive_failures and parking the source in "needs attention." The classifier's tier-gated branch was gated on `return_code in (1, 4)`. Gallery-dl returns a different exit code for mixed-failure runs (e.g. paywall warnings + a missing yt-dlp dep flipping the exit bits), so the branch never fired and the path fell through to UNKNOWN_ERROR. Widen the gate: when no source-level error fired AND tier-gated warnings are present, classify as TIER_LIMITED regardless of return code. 2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with "stranded by recovery sweep (no terminal status after time_limit)" + empty stdout/stderr. Root cause: subprocess.run timeout (900s) and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL wiped the in-memory captured output and the DownloadEvent ended up empty-logged 18 minutes later when the sweep finalized it. Drop gallery-dl's default subprocess timeout to 870s — a 30s margin shy of Celery's soft limit — so subprocess.TimeoutExpired always wins the race and captures the partial stdout/stderr via the existing handler. 3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was firing on every video attachment, causing per-item download failures that masked legitimate tier-gated classification. Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image, video posts download normally and the per-item failure noise disappears. Tests added: - pure tier-gated stderr with exit code 128 → TIER_LIMITED + success - mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
This commit is contained in:
@@ -164,6 +164,58 @@ async def test_validation_quarantines_truncated_files(tmp_path, monkeypatch):
|
||||
assert result.error_type == ErrorType.VALIDATION_FAILED
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_tier_limited_classified_as_success(gdl, monkeypatch):
|
||||
"""Pure tier-gated stderr (paywall warnings only, no source-level
|
||||
errors). Should be TIER_LIMITED with success=True so the
|
||||
DownloadEvent doesn't surface as 'needs attention' for an operator
|
||||
who's just not paying the top tier. Pre-fix: this fell through to
|
||||
UNKNOWN_ERROR when the return code wasn't 1 or 4."""
|
||||
stderr = (
|
||||
"[patreon][warning] Not allowed to view post 159421258\n"
|
||||
"[patreon][warning] Not allowed to view post 159247879\n"
|
||||
"[patreon][warning] Not allowed to view post 158111214\n"
|
||||
)
|
||||
# Use a non-(1,4) return code to prove the gate widened.
|
||||
monkeypatch.setattr(
|
||||
"backend.app.services.gallery_dl.subprocess.run",
|
||||
lambda *a, **k: _proc(stdout="", stderr=stderr, returncode=128),
|
||||
)
|
||||
result = await gdl.download(
|
||||
url="https://patreon.com/alice", artist_slug="alice", platform="patreon",
|
||||
source_config=SourceConfig(),
|
||||
)
|
||||
assert result.error_type == ErrorType.TIER_LIMITED
|
||||
assert result.success is True
|
||||
assert "3 posts" in (result.error_message or "")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_tier_limited_with_per_item_video_failures(gdl, monkeypatch):
|
||||
"""Real operator log shape (2026-05-31): hundreds of paywall warnings
|
||||
+ a handful of per-item video errors caused by missing yt-dlp. The
|
||||
video errors are classified as per-item (not source-level), so the
|
||||
classifier should still pick TIER_LIMITED. Once yt-dlp is installed
|
||||
the video errors stop firing; until then, the source-health doesn't
|
||||
flap to 'needs attention' over paywalled content."""
|
||||
stderr = (
|
||||
"[patreon][warning] Not allowed to view post 159421258\n"
|
||||
"[patreon][warning] Not allowed to view post 159247879\n"
|
||||
"[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl\n"
|
||||
"[download][error] Failed to download 02_video.mp4\n"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"backend.app.services.gallery_dl.subprocess.run",
|
||||
lambda *a, **k: _proc(stdout="", stderr=stderr, returncode=1),
|
||||
)
|
||||
result = await gdl.download(
|
||||
url="https://patreon.com/alice", artist_slug="alice", platform="patreon",
|
||||
source_config=SourceConfig(),
|
||||
)
|
||||
assert result.error_type == ErrorType.TIER_LIMITED
|
||||
assert result.success is True
|
||||
|
||||
|
||||
def test_compute_run_stats(gdl):
|
||||
stats = gdl._compute_run_stats(
|
||||
return_code=0,
|
||||
|
||||
Reference in New Issue
Block a user