Sidecar synthetic anchor cleanup + tier-gated classifier fix #39

Merged

bvandeusen merged 2 commits from dev into main 2026-06-01 00:16:58 -04:00

Conversation 0 Commits 2 Files Changed 8

+380 -23

bvandeusen commented 2026-06-01 00:16:18 -04:00

Owner

Copy Link

Copy Source

Summary

Two coupled cleanup passes from the 2026-05-31 subscriptions audit:

6fc8ae3 — sidecar synthetic anchor cleanup

Operator-reported phantom subscriptions like sidecar:patreon:dpmaker in the Subscriptions hub.

• UI side: SourceService.list() now excludes url LIKE 'sidecar:%' by default; include_synthetic=True opt-in stays available for admin tooling.
• Data side: importer._source_for_sidecar now prefers non-sidecar Sources over synthetic anchors. Before this, gallery-dl downloads silently attached their Posts to older synthetic-anchor rows when both a synthetic and a real Source existed for the same (artist, platform) — a hangover from alembic 0022's consolidation pass.
• Migration: alembic 0028 collapses the synth+real coexisting case — pre-merges Post + ImageProvenance collisions on the canonical, bulk-repoints Posts/ImageProvenance/DownloadEvent.source_id, deletes the synthetic. Lone synthetics (no real twin, e.g. "Dymkens") stay intact — they anchor imported content the operator may still want.

66f19d6 — download event classification + missing dep

Three coupled pains from the same audit:

• Tier-gated mis-classified as error. [patreon][warning] Not allowed to view post N was bumping consecutive_failures and parking the source in "needs attention" because _categorize_error's TIER_LIMITED branch was gated on return_code in (1, 4). Widen to "no source-level error fired AND tier-gated warnings present → TIER_LIMITED regardless of return code."
• Empty-log events (Knuxy #38275): gallery-dl subprocess timeout (900s) raced Celery soft_time_limit (900s) — when Celery won, SIGKILL wiped the in-memory captured output. Drop subprocess timeout to 870s so TimeoutExpired always wins the race and the existing handler captures partial output.
• Missing yt-dlp: [downloader.ytdl][error] Cannot import yt-dlp fired on every video attachment. Add yt-dlp>=2025.1 to requirements.txt.

Test plan

• After deploy, Subscriptions tab no longer shows sidecar:* rows for any artist
• DPMaker collapses to one source row (the real Patreon URL); previously-attached gallery-dl Posts visible under that row
• Patreon syncs with paywalled posts render as 'ok' / TIER_LIMITED, not 'error'
• "Needs attention" rollup count drops by however many sources were only failing on tier-gates
• Video posts download cleanly (no [downloader.ytdl][error] lines)
• Empty-stdout/stderr "stranded by recovery sweep" events become rare

## Summary Two coupled cleanup passes from the 2026-05-31 subscriptions audit: ### `6fc8ae3` — sidecar synthetic anchor cleanup Operator-reported phantom subscriptions like `sidecar:patreon:dpmaker` in the Subscriptions hub. - **UI side**: `SourceService.list()` now excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True` opt-in stays available for admin tooling. - **Data side**: `importer._source_for_sidecar` now prefers non-sidecar Sources over synthetic anchors. Before this, gallery-dl downloads silently attached their Posts to older synthetic-anchor rows when both a synthetic and a real Source existed for the same (artist, platform) — a hangover from alembic 0022's consolidation pass. - **Migration**: alembic 0028 collapses the synth+real coexisting case — pre-merges Post + ImageProvenance collisions on the canonical, bulk-repoints Posts/ImageProvenance/DownloadEvent.source_id, deletes the synthetic. Lone synthetics (no real twin, e.g. "Dymkens") stay intact — they anchor imported content the operator may still want. ### `66f19d6` — download event classification + missing dep Three coupled pains from the same audit: - **Tier-gated mis-classified as error**. `[patreon][warning] Not allowed to view post N` was bumping `consecutive_failures` and parking the source in "needs attention" because `_categorize_error`'s TIER_LIMITED branch was gated on `return_code in (1, 4)`. Widen to "no source-level error fired AND tier-gated warnings present → TIER_LIMITED regardless of return code." - **Empty-log events** (Knuxy #38275): gallery-dl subprocess timeout (900s) raced Celery soft_time_limit (900s) — when Celery won, SIGKILL wiped the in-memory captured output. Drop subprocess timeout to 870s so `TimeoutExpired` always wins the race and the existing handler captures partial output. - **Missing yt-dlp**: `[downloader.ytdl][error] Cannot import yt-dlp` fired on every video attachment. Add `yt-dlp>=2025.1` to requirements.txt. ## Test plan - [ ] After deploy, Subscriptions tab no longer shows `sidecar:*` rows for any artist - [ ] DPMaker collapses to one source row (the real Patreon URL); previously-attached gallery-dl Posts visible under that row - [ ] Patreon syncs with paywalled posts render as 'ok' / TIER_LIMITED, not 'error' - [ ] "Needs attention" rollup count drops by however many sources were only failing on tier-gates - [ ] Video posts download cleanly (no `[downloader.ytdl][error]` lines) - [ ] Empty-stdout/stderr "stranded by recovery sweep" events become rare

bvandeusen added 2 commits 2026-06-01 00:16:19 -04:00

fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup ... 6fc8ae3106

Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing
"phantom" subscriptions like `sidecar:patreon:dpmaker`:

1. `SourceService.list()` returned every Source, no filter on URL.
   alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into
   one canonical row per (artist, platform); when no real campaign URL
   was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>`
   enabled=false as a disabled anchor. The UI then listed those
   anchors as if they were polls — disabled, but visible. Fix: `list()`
   excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True`
   opts back in for admin tooling.

2. `importer._source_for_sidecar` picked the lowest-id Source for
   (artist, platform). When alembic 0022 had rewritten a per-post row
   into a synthetic anchor (lower id) AND the operator later added the
   real subscription (higher id), every gallery-dl download silently
   attached its Post to the SYNTHETIC instead of the real Source. Fix:
   prefer a non-`sidecar:%` URL when one exists; fall back to the
   synthetic; only create a new synthetic when nothing exists for
   (artist, platform).

alembic 0028 is the data half: for every (artist, platform) with both
a synthetic AND a real Source, pre-merge Post+ImageProvenance
collisions on the canonical, bulk-repoint Posts/ImageProvenance/
DownloadEvent.source_id onto the real Source, and delete the
synthetic. Lone synthetics (no real twin) are left intact — they
anchor real imported content the operator may still want; the
list-filter hides them so they no longer surface as phantoms.

fix(download): tier-gated = warning, race subprocess timeout, install yt-dlp ... 66f19d67f5

Three coupled operator-reported pains from the 2026-05-31 download
event audit:

1. `[patreon][warning] Not allowed to view post N` was bubbling up as
   an error event, bumping consecutive_failures and parking the source
   in "needs attention." The classifier's tier-gated branch was gated
   on `return_code in (1, 4)`. Gallery-dl returns a different exit
   code for mixed-failure runs (e.g. paywall warnings + a missing
   yt-dlp dep flipping the exit bits), so the branch never fired and
   the path fell through to UNKNOWN_ERROR. Widen the gate: when no
   source-level error fired AND tier-gated warnings are present,
   classify as TIER_LIMITED regardless of return code.

2. Knuxy event #38275 (2026-05-31) ran 30 min and finalized with
   "stranded by recovery sweep (no terminal status after time_limit)"
   + empty stdout/stderr. Root cause: subprocess.run timeout (900s)
   and Celery soft_time_limit (900s) raced; when Celery won, SIGKILL
   wiped the in-memory captured output and the DownloadEvent ended up
   empty-logged 18 minutes later when the sweep finalized it. Drop
   gallery-dl's default subprocess timeout to 870s — a 30s margin
   shy of Celery's soft limit — so subprocess.TimeoutExpired always
   wins the race and captures the partial stdout/stderr via the
   existing handler.

3. `[downloader.ytdl][error] Cannot import yt-dlp or youtube-dl` was
   firing on every video attachment, causing per-item download
   failures that masked legitimate tier-gated classification.
   Add yt-dlp>=2025.1 to requirements.txt. Once it's in the image,
   video posts download normally and the per-item failure noise
   disappears.

Tests added:
- pure tier-gated stderr with exit code 128 → TIER_LIMITED + success
- mixed tier-gated + yt-dlp + per-item failures → still TIER_LIMITED

bvandeusen merged commit 856e9104b4 into main 2026-06-01 00:16:58 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#39