docs: add patreon campaign-ID resolver design spec

Auto-heal "Failed to extract campaign ID" failures by resolving via Patreon's campaigns API, caching the ID in Source.metadata_, and retrying once within the same task run. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-18 12:46:00 -04:00
parent f3412dea57
commit 59c40ccdb9
1 changed files with 236 additions and 0 deletions
@@ -0,0 +1,236 @@
+# Patreon Campaign-ID Auto-Resolver — Design Spec
+
+**Date:** 2026-04-18
+**Scope:** Automatically recover from gallery-dl's "Failed to extract campaign ID" errors on Patreon sources by resolving the campaign ID via Patreon's campaigns API, caching it on the source, and rewriting the URL at download time.
+
+---
+
+## Motivation
+
+Gallery-dl 1.31.10's Patreon extractor fails for some creators with:
+
+```
+[patreon][error] Failed to extract campaign ID
+```
+
+The failure happens during the vanity-URL → campaign-ID lookup, when Patreon serves an HTML page whose bootstrap JSON is empty or missing. Both gallery-dl's primary JSON parse and its Next.js regex fallback miss, and the extraction aborts.
+
+The upstream-recommended workaround is to use `https://www.patreon.com/id:<campaign_id>` as the source URL, which bypasses the vanity-resolution path entirely. Campaign IDs are retrievable from Patreon's `/api/campaigns?filter[vanity]=<vanity>` endpoint.
+
+This spec automates that workaround: on failure, look up the campaign ID, cache it on the source, and retry the download once in the same task run. Subsequent downloads use the cached ID-URL directly. The user's original URL is preserved for display.
+
+---
+
+## Design
+
+### Components
+
+#### 1. `backend/app/services/patreon_resolver.py` — new
+
+Single async function:
+
+```python
+async def resolve_campaign_id(vanity: str, cookies_path: Optional[str]) -> Optional[str]:
+    """Resolve a Patreon vanity (creator slug) to a campaign ID.
+
+    Returns the campaign ID as a string, or None on any failure (network
+    error, auth error, empty response, unexpected shape). Never raises.
+    """
+```
+
+Implementation:
+
+- Load cookies with `http.cookiejar.MozillaCookieJar(cookies_path).load(ignore_discard=True, ignore_expires=True)`, then convert to an `aiohttp.CookieJar` by iterating and calling `update_cookies`. (aiohttp doesn't read Netscape format natively.)
+- `GET https://www.patreon.com/api/campaigns?filter[vanity]={vanity}&fields[campaign]=name` with header `User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36` (a stable desktop UA — Patreon rejects requests with suspicious UAs like `python-requests/*`).
+- Parse the JSON response. Success path: `data[0].id` (string). Log at `logging.INFO`.
+- Any failure — non-200 status, JSON parse error, empty `data`, missing `id` field, `aiohttp.ClientError`, `asyncio.TimeoutError` — returns `None` and logs at `logging.WARNING` with the vanity and the specific reason (HTTP status, exception class, or shape description) for debuggability.
+- 10-second total timeout.
+- If `cookies_path` is `None` or the file doesn't exist, proceed without cookies (the endpoint may still work; the resolver logs a debug note).
+
+#### 2. `backend/app/tasks/downloads.py` — retry hook + URL rewrite
+
+**URL rewrite (outbound path, before every download):**
+
+Before calling `gallery_dl_service.download()`, compute the effective URL:
+
+```python
+def _effective_patreon_url(source: Source) -> str:
+    if source.platform == "patreon":
+        campaign_id = (source.metadata_ or {}).get("patreon_campaign_id")
+        if campaign_id:
+            return f"https://www.patreon.com/id:{campaign_id}"
+    return source.url
+```
+
+Called in `_run_single_download` (or whichever function assembles the `download()` call arguments). All Patreon downloads with a cached ID use the id-URL. Non-Patreon platforms and Patreon sources without a cached ID use `source.url` unchanged.
+
+**Retry hook (on first failure, inline):**
+
+After `dl_result = await gallery_dl_service.download(...)`, before the DB-write block:
+
+```python
+if (
+    not dl_result.success
+    and source.platform == "patreon"
+    and _looks_like_campaign_id_failure(dl_result)
+    and "patreon_campaign_id" not in (source.metadata_ or {})
+):
+    vanity = _extract_vanity_from_url(source.url)
+    if vanity:
+        logger.info(f"Attempting campaign-ID resolution for {subscription_name}/patreon ({vanity})")
+        campaign_id = await resolve_campaign_id(vanity, cookies_path)
+        if campaign_id:
+            source.metadata_ = {
+                **(source.metadata_ or {}),
+                "patreon_campaign_id": campaign_id,
+            }
+            # Retry once with the id URL; overwrite dl_result
+            dl_result = await gallery_dl_service.download(
+                url=f"https://www.patreon.com/id:{campaign_id}",
+                subscription_name=subscription_name,
+                platform="patreon",
+                source_config=source_config,
+                cookies_path=cookies_path,
+                auth_token=auth_token,
+            )
+```
+
+Helper functions in the same module:
+
+```python
+def _looks_like_campaign_id_failure(dl_result) -> bool:
+    haystack = f"{dl_result.stdout or ''}\n{dl_result.stderr or ''}".lower()
+    return "failed to extract campaign id" in haystack
+
+def _extract_vanity_from_url(url: str) -> Optional[str]:
+    # Matches patreon.com/<vanity> and patreon.com/c/<vanity>/... and patreon.com/c/<vanity>
+    # Returns None for id:<N> URLs and for anything that doesn't match.
+    m = re.match(
+        r"https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
+        url,
+    )
+    return m.group(1) if m else None
+```
+
+The retry's `dl_result` flows into the existing DB-write block unchanged. `error_count` resets to 0 on retry success, or increments once (not twice) on retry failure — the original failure is discarded.
+
+#### 3. `backend/app/services/gallery_dl.py` — no change
+
+The caller (`downloads.py`) supplies the already-effective URL. The service remains platform-agnostic.
+
+#### 4. Data model — no change
+
+`Source.metadata_` is already a JSONB dict column. We write one new string key, `patreon_campaign_id`. No migration.
+
+---
+
+## Data flow
+
+### First-time failure → auto-heal
+
+1. Scheduled check calls `_run_single_download` for source `{platform: patreon, url: https://www.patreon.com/mstsu, metadata_: {}}`.
+2. `_effective_patreon_url(source)` returns `source.url` (no cache yet).
+3. `gallery_dl_service.download(url=source.url, ...)` returns `DownloadResult(success=False, stderr="... Failed to extract campaign ID ...")`.
+4. Retry hook fires: matches `patreon` platform + error pattern + empty cache.
+5. `_extract_vanity_from_url(source.url)` returns `"mstsu"`.
+6. `resolve_campaign_id("mstsu", cookies_path)` hits the campaigns API, returns `"12345"`.
+7. Task writes `source.metadata_['patreon_campaign_id'] = "12345"` into the in-memory ORM instance (commit happens in the normal DB-write block).
+8. `gallery_dl_service.download(url="https://www.patreon.com/id:12345", ...)` runs; succeeds (typical case).
+9. DB-write block sees `dl_result.success=True`, resets `error_count=0`, commits the metadata update along with it, publishes `download.completed`.
+
+### Subsequent runs
+
+1. `_effective_patreon_url(source)` sees `patreon_campaign_id=12345`, returns `https://www.patreon.com/id:12345`.
+2. Gallery-dl gets the id-URL directly; no vanity lookup, no resolver involvement.
+3. Normal download path.
+
+### Resolver failure
+
+- `resolve_campaign_id` returns `None` (API 401, network error, empty `data[]`, etc.).
+- No metadata written, no retry.
+- Original `dl_result` flows through the normal error path: `error_count += 1`, `download.failed` event, etc.
+- A warning log entry records the vanity and the reason. The user sees the same "Failed to extract campaign ID" error they would have seen without the feature.
+
+### Resolver succeeds but retry still fails
+
+- Metadata is still written (the ID is valid; the resolver was not the problem).
+- The retry's failure is recorded normally. Next scheduled run uses the id-URL and hits whatever the real issue is (likely auth).
+- No further resolver runs for this source.
+
+---
+
+## Error handling
+
+| Situation | Behavior |
+|---|---|
+| Cookies file missing | Resolver proceeds without cookies; may still succeed on public endpoint |
+| Cookies expired (API 401/403) | Resolver returns None, logs warning, original failure surfaced |
+| Vanity regex doesn't match | Retry hook skipped, original failure surfaced |
+| `data[]` empty in response | Resolver returns None, logs warning |
+| Unexpected JSON shape | Resolver returns None, logs shape at warning |
+| Network error / timeout | Resolver returns None, logs exception class |
+| Resolver + retry both succeed | Normal success path, metadata cached |
+| Resolver succeeds, retry fails | Metadata cached (ID is valid), retry's failure recorded |
+| Source already has `patreon_campaign_id` but retry fails | No second resolver run (guard on metadata presence) |
+| Non-Patreon source fails with any error | Retry hook skipped (platform gate) |
+| Patreon source fails with a different error (e.g., auth) | Retry hook skipped (pattern gate) |
+
+---
+
+## Testing
+
+### Unit — `patreon_resolver.py`
+
+- `resolve_campaign_id` happy path: mock `aiohttp` response with `{"data": [{"id": "12345", "type": "campaign"}]}` → returns `"12345"`.
+- Empty `data[]` → returns `None`, warning logged.
+- Missing `data` key → returns `None`, warning logged.
+- HTTP 401 → returns `None`, warning logged with status.
+- `aiohttp.ClientError` raised → returns `None`, warning logged.
+- `asyncio.TimeoutError` → returns `None`, warning logged.
+- `cookies_path=None` → call proceeds, no cookie jar loaded.
+
+Use `aioresponses` or `aiohttp.test_utils` to mock; no real network calls.
+
+### Unit — `downloads.py` helpers
+
+- `_looks_like_campaign_id_failure`: `DownloadResult` with the pattern in stderr → True; pattern in stdout → True; neither → False; case-insensitive match.
+- `_extract_vanity_from_url`: `patreon.com/mstsu` → `"mstsu"`; `patreon.com/c/mstsu/posts` → `"mstsu"`; `patreon.com/id:12345` → `None`; `patreon.com/` → `None`; unrelated URL → `None`.
+
+### Integration — retry hook
+
+Patch `app.tasks.downloads.gallery_dl_service.download` and `app.tasks.downloads.resolve_campaign_id` (after importing the symbol into the module, patching the local binding).
+
+- **Happy-path heal:** First `download()` returns campaign-ID failure, resolver returns `"12345"`, second `download()` returns success. Verify: resolver called with correct vanity; `source.metadata_['patreon_campaign_id'] == "12345"` after commit; second `download()` called with `patreon.com/id:12345` URL; `source.error_count == 0`; `download.completed` event published once (for the retry).
+- **Resolver returns None:** First `download()` returns campaign-ID failure, resolver returns `None`. Verify: second `download()` not called; `source.metadata_` unchanged (no `patreon_campaign_id` key); `source.error_count == 1`; `download.failed` event published with the original error.
+- **Already cached:** Source has `metadata_['patreon_campaign_id'] = "12345"` before run. Verify: `_effective_patreon_url` returns id-URL; first `download()` called with id-URL; no resolver call regardless of result.
+- **Non-patreon:** Source is `subscribestar` with similar-looking error text. Verify: retry hook skipped entirely.
+
+### Integration — URL-rewrite outbound
+
+- Source with `patreon_campaign_id` cached → `download()` called with id-URL.
+- Source without cached ID → `download()` called with `source.url`.
+- Non-Patreon source with `patreon_campaign_id` somehow set (data corruption) → `download()` called with `source.url` (platform gate wins).
+
+---
+
+## Files changed
+
+| File | Change |
+|---|---|
+| `backend/app/services/patreon_resolver.py` | Create: single `resolve_campaign_id(vanity, cookies_path)` async function |
+| `backend/app/tasks/downloads.py` | Add `_effective_patreon_url`, `_looks_like_campaign_id_failure`, `_extract_vanity_from_url` helpers; insert retry hook before DB-write block in `_run_single_download`; use effective URL when calling `gallery_dl_service.download()` |
+| `tests/services/test_patreon_resolver.py` | Create: unit tests for resolver |
+| `tests/tasks/test_downloads_patreon_retry.py` | Create: integration tests for retry hook and URL-rewrite |
+
+No frontend changes. No schema migration. No new API endpoints. No settings changes.
+
+---
+
+## Non-goals
+
+- No one-time migration to pre-resolve all existing Patreon sources. Sources auto-heal on their next scheduled check (≤ `check_interval`, default 8h).
+- No UI surface for the campaign ID. It lives in `Source.metadata_` and is visible only via the existing metadata inspection path (JSON blob in the DB).
+- No retry on the id-URL if the retry itself fails. One resolve + one retry per task run, total.
+- No resolver re-run if cached campaign ID ever becomes stale. Manual fix: delete the key from `source.metadata_`.
+- No change to `gallery_dl.py` — it stays platform-agnostic. The Patreon URL quirk is contained in `downloads.py`.
+- No change to the error-categorization logic in `_categorize_error`. The "Failed to extract campaign ID" failure is still categorized as `UNKNOWN_ERROR` if the resolver can't heal it; this is fine because the primary recovery path runs before categorization matters.