fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 24s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 6m56s
CI / intcore (push) Successful in 7m35s

Three coupled operator-reported pains from the 2026-05-31 download
event audit:

1. `[patreon][warning] Not allowed to view post N` was bubbling up as
   an error event, bumping consecutive_failures and parking the source
   in "needs attention." The classifier's tier-gated branch was gated
   on `return_code in (1, 4)`. Gallery-dl returns a different exit
   code for mixed-failure runs (e.g. paywall warnings + a missing
   yt-dlp dep flipping the exit bits), so the branch never fired and
   the path fell through to UNKNOWN_ERROR. Widen the gate: when no
   source-level error fired AND tier-gated warnings are present,
   classify as TIER_LIMITED regardless of return code.

2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
   "stranded by recovery sweep (no terminal status after time_limit)"
   + empty stdout/stderr. Root cause: subprocess.run timeout (900s)
   and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
   wiped the in-memory captured output and the DownloadEvent ended up
   empty-logged 18 minutes later when the sweep finalized it. Drop
   gallery-dl's default subprocess timeout to 870s — a 30s margin
   shy of Celery's soft limit — so subprocess.TimeoutExpired always
   wins the race and captures the partial stdout/stderr via the
   existing handler.

3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
   firing on every video attachment, causing per-item download
   failures that masked legitimate tier-gated classification.
   Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
   video posts download normally and the per-item failure noise
   disappears.

Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
This commit is contained in:
2026-05-31 23:30:39 -04:00
parent 6fc8ae3106
commit 66f19d67f5
3 changed files with 78 additions and 8 deletions
+21 -8
View File
@@ -42,13 +42,16 @@ class ErrorType(StrEnum):
UNKNOWN_ERROR = "unknown_error"
# Pinned to download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). Anything larger and Celery kills the task before
# subprocess.run can raise TimeoutExpired — leaving the DownloadEvent
# stranded for the recovery sweep instead of capturing a clean timeout
# error. Per-source bumps live in source.config_overrides for legitimately
# long syncs. Operator-confirmed 2026-05-30 (~40-min hang investigation).
_DEFAULT_GDL_TIMEOUT_SECONDS = 900
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
# The 30s buffer absorbs scheduler jitter / GC pauses without making
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
# still live in source.config_overrides for legitimately long syncs.
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
@dataclass
@@ -369,7 +372,17 @@ class GalleryDLService:
if return_code in (1, 4) and (skip_line_count > 0 or has_skip_text) and not has_actual_error:
return ErrorType.NO_NEW_CONTENT, "No new content to download"
if return_code in (1, 4) and not has_actual_error:
# Tier-gated classification used to require `return_code in (1, 4)`,
# which silently fell through to UNKNOWN_ERROR when gallery-dl
# returned a different exit code for mixed-failure runs (e.g.
# paywall warnings + a missing yt-dlp dep flipping the exit bits).
# The artist then surfaced as "needs attention" purely because a
# paywall blocked posts the operator wasn't paying to see —
# operator-flagged 2026-05-31. Now: if no source-level error
# category fired AND tier-gated warnings are present, classify
# as TIER_LIMITED regardless of return code. Same priority order
# as before (auth/rate/access/not_found/network/http still win).
if not has_actual_error:
tier_gated_lines = [
line for line in combined.split("\n")
if "][warning]" in line and "not allowed to view post" in line