Three coupled operator-reported pains from the 2026-05-31 download
event audit:
1. `[patreon][warning] Not allowed to view post N` was bubbling up as
an error event, bumping consecutive_failures and parking the source
in "needs attention." The classifier's tier-gated branch was gated
on `return_code in (1, 4)`. Gallery-dl returns a different exit
code for mixed-failure runs (e.g. paywall warnings + a missing
yt-dlp dep flipping the exit bits), so the branch never fired and
the path fell through to UNKNOWN_ERROR. Widen the gate: when no
source-level error fired AND tier-gated warnings are present,
classify as TIER_LIMITED regardless of return code.
2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
"stranded by recovery sweep (no terminal status after time_limit)"
+ empty stdout/stderr. Root cause: subprocess.run timeout (900s)
and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
wiped the in-memory captured output and the DownloadEvent ended up
empty-logged 18 minutes later when the sweep finalized it. Drop
gallery-dl's default subprocess timeout to 870s — a 30s margin
shy of Celery's soft limit — so subprocess.TimeoutExpired always
wins the race and captures the partial stdout/stderr via the
existing handler.
3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
firing on every video attachment, causing per-item download
failures that masked legitimate tier-gated classification.
Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
video posts download normally and the per-item failure noise
disappears.
Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:
1. `SourceService.list()` returned every Source, no filter on URL.
alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
one canonical row per (artist, platform); when no real campaign URL
was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
enabled=false as a disabled anchor. The UI then listed those
anchors as if they were polls — disabled, but visible. Fix: `list()`
excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
opts back in for admin tooling.
2. `importer._source_for_sidecar` picked the lowest-id Source for
(artist, platform). When alembic 0022 had rewritten a per-post row
into a synthetic anchor (lower id) AND the operator later added the
real subscription (higher id), every gallery-dl download silently
attached its Post to the SYNTHETIC instead of the real Source. Fix:
prefer a non-`sidecar:%` URL when one exists; fall back to the
synthetic; only create a new synthetic when nothing exists for
(artist, platform).
alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.
bvandeusen