From 59c40ccdb97731f4a2bcb55e4d7434d291277010 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 18 Apr 2026 12:46:00 -0400 Subject: [PATCH] docs: add patreon campaign-ID resolver design spec Auto-heal "Failed to extract campaign ID" failures by resolving via Patreon's campaigns API, caching the ID in Source.metadata_, and retrying once within the same task run. Co-Authored-By: Claude Opus 4.7 --- ...-04-18-patreon-campaign-resolver-design.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-18-patreon-campaign-resolver-design.md diff --git a/docs/superpowers/specs/2026-04-18-patreon-campaign-resolver-design.md b/docs/superpowers/specs/2026-04-18-patreon-campaign-resolver-design.md new file mode 100644 index 0000000..a58f0db --- /dev/null +++ b/docs/superpowers/specs/2026-04-18-patreon-campaign-resolver-design.md @@ -0,0 +1,236 @@ +# Patreon Campaign-ID Auto-Resolver — Design Spec + +**Date:** 2026-04-18 +**Scope:** Automatically recover from gallery-dl's "Failed to extract campaign ID" errors on Patreon sources by resolving the campaign ID via Patreon's campaigns API, caching it on the source, and rewriting the URL at download time. + +--- + +## Motivation + +Gallery-dl 1.31.10's Patreon extractor fails for some creators with: + +``` +[patreon][error] Failed to extract campaign ID +``` + +The failure happens during the vanity-URL → campaign-ID lookup, when Patreon serves an HTML page whose bootstrap JSON is empty or missing. Both gallery-dl's primary JSON parse and its Next.js regex fallback miss, and the extraction aborts. + +The upstream-recommended workaround is to use `https://www.patreon.com/id:` as the source URL, which bypasses the vanity-resolution path entirely. Campaign IDs are retrievable from Patreon's `/api/campaigns?filter[vanity]=` endpoint. + +This spec automates that workaround: on failure, look up the campaign ID, cache it on the source, and retry the download once in the same task run. Subsequent downloads use the cached ID-URL directly. The user's original URL is preserved for display. + +--- + +## Design + +### Components + +#### 1. `backend/app/services/patreon_resolver.py` — new + +Single async function: + +```python +async def resolve_campaign_id(vanity: str, cookies_path: Optional[str]) -> Optional[str]: + """Resolve a Patreon vanity (creator slug) to a campaign ID. + + Returns the campaign ID as a string, or None on any failure (network + error, auth error, empty response, unexpected shape). Never raises. + """ +``` + +Implementation: + +- Load cookies with `http.cookiejar.MozillaCookieJar(cookies_path).load(ignore_discard=True, ignore_expires=True)`, then convert to an `aiohttp.CookieJar` by iterating and calling `update_cookies`. (aiohttp doesn't read Netscape format natively.) +- `GET https://www.patreon.com/api/campaigns?filter[vanity]={vanity}&fields[campaign]=name` with header `User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36` (a stable desktop UA — Patreon rejects requests with suspicious UAs like `python-requests/*`). +- Parse the JSON response. Success path: `data[0].id` (string). Log at `logging.INFO`. +- Any failure — non-200 status, JSON parse error, empty `data`, missing `id` field, `aiohttp.ClientError`, `asyncio.TimeoutError` — returns `None` and logs at `logging.WARNING` with the vanity and the specific reason (HTTP status, exception class, or shape description) for debuggability. +- 10-second total timeout. +- If `cookies_path` is `None` or the file doesn't exist, proceed without cookies (the endpoint may still work; the resolver logs a debug note). + +#### 2. `backend/app/tasks/downloads.py` — retry hook + URL rewrite + +**URL rewrite (outbound path, before every download):** + +Before calling `gallery_dl_service.download()`, compute the effective URL: + +```python +def _effective_patreon_url(source: Source) -> str: + if source.platform == "patreon": + campaign_id = (source.metadata_ or {}).get("patreon_campaign_id") + if campaign_id: + return f"https://www.patreon.com/id:{campaign_id}" + return source.url +``` + +Called in `_run_single_download` (or whichever function assembles the `download()` call arguments). All Patreon downloads with a cached ID use the id-URL. Non-Patreon platforms and Patreon sources without a cached ID use `source.url` unchanged. + +**Retry hook (on first failure, inline):** + +After `dl_result = await gallery_dl_service.download(...)`, before the DB-write block: + +```python +if ( + not dl_result.success + and source.platform == "patreon" + and _looks_like_campaign_id_failure(dl_result) + and "patreon_campaign_id" not in (source.metadata_ or {}) +): + vanity = _extract_vanity_from_url(source.url) + if vanity: + logger.info(f"Attempting campaign-ID resolution for {subscription_name}/patreon ({vanity})") + campaign_id = await resolve_campaign_id(vanity, cookies_path) + if campaign_id: + source.metadata_ = { + **(source.metadata_ or {}), + "patreon_campaign_id": campaign_id, + } + # Retry once with the id URL; overwrite dl_result + dl_result = await gallery_dl_service.download( + url=f"https://www.patreon.com/id:{campaign_id}", + subscription_name=subscription_name, + platform="patreon", + source_config=source_config, + cookies_path=cookies_path, + auth_token=auth_token, + ) +``` + +Helper functions in the same module: + +```python +def _looks_like_campaign_id_failure(dl_result) -> bool: + haystack = f"{dl_result.stdout or ''}\n{dl_result.stderr or ''}".lower() + return "failed to extract campaign id" in haystack + +def _extract_vanity_from_url(url: str) -> Optional[str]: + # Matches patreon.com/ and patreon.com/c//... and patreon.com/c/ + # Returns None for id: URLs and for anything that doesn't match. + m = re.match( + r"https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)", + url, + ) + return m.group(1) if m else None +``` + +The retry's `dl_result` flows into the existing DB-write block unchanged. `error_count` resets to 0 on retry success, or increments once (not twice) on retry failure — the original failure is discarded. + +#### 3. `backend/app/services/gallery_dl.py` — no change + +The caller (`downloads.py`) supplies the already-effective URL. The service remains platform-agnostic. + +#### 4. Data model — no change + +`Source.metadata_` is already a JSONB dict column. We write one new string key, `patreon_campaign_id`. No migration. + +--- + +## Data flow + +### First-time failure → auto-heal + +1. Scheduled check calls `_run_single_download` for source `{platform: patreon, url: https://www.patreon.com/mstsu, metadata_: {}}`. +2. `_effective_patreon_url(source)` returns `source.url` (no cache yet). +3. `gallery_dl_service.download(url=source.url, ...)` returns `DownloadResult(success=False, stderr="... Failed to extract campaign ID ...")`. +4. Retry hook fires: matches `patreon` platform + error pattern + empty cache. +5. `_extract_vanity_from_url(source.url)` returns `"mstsu"`. +6. `resolve_campaign_id("mstsu", cookies_path)` hits the campaigns API, returns `"12345"`. +7. Task writes `source.metadata_['patreon_campaign_id'] = "12345"` into the in-memory ORM instance (commit happens in the normal DB-write block). +8. `gallery_dl_service.download(url="https://www.patreon.com/id:12345", ...)` runs; succeeds (typical case). +9. DB-write block sees `dl_result.success=True`, resets `error_count=0`, commits the metadata update along with it, publishes `download.completed`. + +### Subsequent runs + +1. `_effective_patreon_url(source)` sees `patreon_campaign_id=12345`, returns `https://www.patreon.com/id:12345`. +2. Gallery-dl gets the id-URL directly; no vanity lookup, no resolver involvement. +3. Normal download path. + +### Resolver failure + +- `resolve_campaign_id` returns `None` (API 401, network error, empty `data[]`, etc.). +- No metadata written, no retry. +- Original `dl_result` flows through the normal error path: `error_count += 1`, `download.failed` event, etc. +- A warning log entry records the vanity and the reason. The user sees the same "Failed to extract campaign ID" error they would have seen without the feature. + +### Resolver succeeds but retry still fails + +- Metadata is still written (the ID is valid; the resolver was not the problem). +- The retry's failure is recorded normally. Next scheduled run uses the id-URL and hits whatever the real issue is (likely auth). +- No further resolver runs for this source. + +--- + +## Error handling + +| Situation | Behavior | +|---|---| +| Cookies file missing | Resolver proceeds without cookies; may still succeed on public endpoint | +| Cookies expired (API 401/403) | Resolver returns None, logs warning, original failure surfaced | +| Vanity regex doesn't match | Retry hook skipped, original failure surfaced | +| `data[]` empty in response | Resolver returns None, logs warning | +| Unexpected JSON shape | Resolver returns None, logs shape at warning | +| Network error / timeout | Resolver returns None, logs exception class | +| Resolver + retry both succeed | Normal success path, metadata cached | +| Resolver succeeds, retry fails | Metadata cached (ID is valid), retry's failure recorded | +| Source already has `patreon_campaign_id` but retry fails | No second resolver run (guard on metadata presence) | +| Non-Patreon source fails with any error | Retry hook skipped (platform gate) | +| Patreon source fails with a different error (e.g., auth) | Retry hook skipped (pattern gate) | + +--- + +## Testing + +### Unit — `patreon_resolver.py` + +- `resolve_campaign_id` happy path: mock `aiohttp` response with `{"data": [{"id": "12345", "type": "campaign"}]}` → returns `"12345"`. +- Empty `data[]` → returns `None`, warning logged. +- Missing `data` key → returns `None`, warning logged. +- HTTP 401 → returns `None`, warning logged with status. +- `aiohttp.ClientError` raised → returns `None`, warning logged. +- `asyncio.TimeoutError` → returns `None`, warning logged. +- `cookies_path=None` → call proceeds, no cookie jar loaded. + +Use `aioresponses` or `aiohttp.test_utils` to mock; no real network calls. + +### Unit — `downloads.py` helpers + +- `_looks_like_campaign_id_failure`: `DownloadResult` with the pattern in stderr → True; pattern in stdout → True; neither → False; case-insensitive match. +- `_extract_vanity_from_url`: `patreon.com/mstsu` → `"mstsu"`; `patreon.com/c/mstsu/posts` → `"mstsu"`; `patreon.com/id:12345` → `None`; `patreon.com/` → `None`; unrelated URL → `None`. + +### Integration — retry hook + +Patch `app.tasks.downloads.gallery_dl_service.download` and `app.tasks.downloads.resolve_campaign_id` (after importing the symbol into the module, patching the local binding). + +- **Happy-path heal:** First `download()` returns campaign-ID failure, resolver returns `"12345"`, second `download()` returns success. Verify: resolver called with correct vanity; `source.metadata_['patreon_campaign_id'] == "12345"` after commit; second `download()` called with `patreon.com/id:12345` URL; `source.error_count == 0`; `download.completed` event published once (for the retry). +- **Resolver returns None:** First `download()` returns campaign-ID failure, resolver returns `None`. Verify: second `download()` not called; `source.metadata_` unchanged (no `patreon_campaign_id` key); `source.error_count == 1`; `download.failed` event published with the original error. +- **Already cached:** Source has `metadata_['patreon_campaign_id'] = "12345"` before run. Verify: `_effective_patreon_url` returns id-URL; first `download()` called with id-URL; no resolver call regardless of result. +- **Non-patreon:** Source is `subscribestar` with similar-looking error text. Verify: retry hook skipped entirely. + +### Integration — URL-rewrite outbound + +- Source with `patreon_campaign_id` cached → `download()` called with id-URL. +- Source without cached ID → `download()` called with `source.url`. +- Non-Patreon source with `patreon_campaign_id` somehow set (data corruption) → `download()` called with `source.url` (platform gate wins). + +--- + +## Files changed + +| File | Change | +|---|---| +| `backend/app/services/patreon_resolver.py` | Create: single `resolve_campaign_id(vanity, cookies_path)` async function | +| `backend/app/tasks/downloads.py` | Add `_effective_patreon_url`, `_looks_like_campaign_id_failure`, `_extract_vanity_from_url` helpers; insert retry hook before DB-write block in `_run_single_download`; use effective URL when calling `gallery_dl_service.download()` | +| `tests/services/test_patreon_resolver.py` | Create: unit tests for resolver | +| `tests/tasks/test_downloads_patreon_retry.py` | Create: integration tests for retry hook and URL-rewrite | + +No frontend changes. No schema migration. No new API endpoints. No settings changes. + +--- + +## Non-goals + +- No one-time migration to pre-resolve all existing Patreon sources. Sources auto-heal on their next scheduled check (≤ `check_interval`, default 8h). +- No UI surface for the campaign ID. It lives in `Source.metadata_` and is visible only via the existing metadata inspection path (JSON blob in the DB). +- No retry on the id-URL if the retry itself fails. One resolve + one retry per task run, total. +- No resolver re-run if cached campaign ID ever becomes stale. Manual fix: delete the key from `source.metadata_`. +- No change to `gallery_dl.py` — it stays platform-agnostic. The Patreon URL quirk is contained in `downloads.py`. +- No change to the error-categorization logic in `_categorize_error`. The "Failed to extract campaign ID" failure is still categorized as `UNKNOWN_ERROR` if the resolver can't heal it; this is fine because the primary recovery path runs before categorization matters.