fix(downloader): don't misclassify tier-gated runs with per-item video 403s

When a Patreon source loses tier access, gallery-dl emits "Not allowed to
view post N" warnings for every post AND yt-dlp tries to fetch HLS manifests
which also 403. The ytdl error text contained "HTTP Error 403: Forbidden",
which the classifier's ACCESS_DENIED_PATTERNS matched against the full
combined stdout+stderr — so the run was labeled ACCESS_DENIED instead of
TIER_LIMITED.

Two fixes in one edit, because they're interlocked:
- Compute per-item vs source-level error lines upfront. has_actual_error
  now reflects only source-level errors (previously the per-item exclusion
  was gated on skip evidence, which tier-limited runs don't produce since
  no content was accessed).
- Strip per-item error lines from `combined` before downstream pattern
  matching so noise from recovered per-item failures doesn't latch onto
  ACCESS_DENIED / HTTP_ERROR / NOT_FOUND classifiers.

Regression test: tier-gated Patreon run with yt-dlp HLS 403s → TIER_LIMITED,
not ACCESS_DENIED.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-04-21 07:34:33 -04:00
parent 851df294f2
commit c2a2162c86
2 changed files with 75 additions and 33 deletions
+45 -33
View File
@@ -437,40 +437,52 @@ class GalleryDLService:
"download failed",
"extraction failed",
]
has_actual_error = any(err in combined for err in actual_error_indicators)
# Non-fatal error lines gallery-dl logs but recovers from. These shouldn't
# block skip detection when the overall download proceeded successfully:
# - "not allowed to view" / "unable to get post": per-item paywall warnings
# on Patreon /c/user/posts URLs where some posts are paywalled.
# - "failed to extract campaign id": logged when the vanity-URL HTML path
# fails, but gallery-dl falls back to the /cw/<vanity> redirect and
# recovers the campaign_id on its own.
# - "failed to download": per-item download failure (e.g., a single
# Patreon attachment whose media URL expired / was deleted). The
# downloader logs [download][error] and moves to the next item.
if has_actual_error and (skip_line_count > 0 or has_skip_text):
per_item_patterns = [
"not allowed to view",
"unable to get post",
"failed to extract campaign id",
"failed to download",
# yt-dlp/youtube-dl missing: each video that gallery-dl tries to
# download fails, but image/skip activity continues. Treat as
# per-item so it doesn't mask TIER_LIMITED/NO_NEW_CONTENT.
# Surface separately as a setup gap (see Dockerfile yt-dlp install).
"cannot import yt-dlp",
"cannot import youtube-dl",
]
error_lines = [
line for line in combined.split('\n')
if any(ind in line for ind in actual_error_indicators)
]
if error_lines and all(
any(p in line for p in per_item_patterns)
for line in error_lines
):
has_actual_error = False
# Non-fatal error lines gallery-dl logs but recovers from. These are
# per-item failures the run already moved past — they shouldn't cause
# source-level misclassification:
# - "not allowed to view" / "unable to get post": per-item paywall
# warnings on Patreon URLs with mixed tier access.
# - "failed to extract campaign id": vanity-URL HTML path fails,
# gallery-dl falls back to /cw/<vanity> and recovers.
# - "failed to download": per-item media failure (expired URL,
# yt-dlp HLS 403 on tier-gated video, deleted attachment).
# Downloader logs [download][error] and continues.
# - "cannot import yt-dlp"/"youtube-dl": video backend missing;
# surfaces via run_stats as a setup gap, not a source failure.
per_item_patterns = [
"not allowed to view",
"unable to get post",
"failed to extract campaign id",
"failed to download",
"cannot import yt-dlp",
"cannot import youtube-dl",
]
all_lines = combined.split('\n')
error_lines = [
line for line in all_lines
if any(ind in line for ind in actual_error_indicators)
]
per_item_error_lines = [
line for line in error_lines
if any(p in line for p in per_item_patterns)
]
source_level_error_lines = [
line for line in error_lines
if line not in set(per_item_error_lines)
]
# Only source-level errors block skip/tier-limited classification.
# Per-item failures are tracked separately in run_stats.
has_actual_error = bool(source_level_error_lines)
# Strip per-item error lines from the text used for downstream pattern
# matching. Otherwise noise like "HTTP Error 403: Forbidden" from a
# yt-dlp HLS failure latches onto ACCESS_DENIED and masks the real
# signal (tier gating, no new content, etc.).
if per_item_error_lines:
per_item_set = set(per_item_error_lines)
combined = '\n'.join(line for line in all_lines if line not in per_item_set)
# If we have skip evidence and no real errors, it's no_new_content
if (skip_line_count > 0 or has_skip_text) and not has_actual_error: