GallerySubscriber/docs/superpowers/specs/2026-04-18-patreon-campaign-resolver-design.md

Patreon Campaign-ID Auto-Resolver — Design Spec

Date: 2026-04-18 Scope: Automatically recover from gallery-dl's "Failed to extract campaign ID" errors on Patreon sources by resolving the campaign ID via Patreon's campaigns API, caching it on the source, and rewriting the URL at download time.

---

Motivation

Gallery-dl 1.31.10's Patreon extractor fails for some creators with:

[patreon][error] Failed to extract campaign ID

The failure happens during the vanity-URL → campaign-ID lookup, when Patreon serves an HTML page whose bootstrap JSON is empty or missing. Both gallery-dl's primary JSON parse and its Next.js regex fallback miss, and the extraction aborts.

The upstream-recommended workaround is to use https://www.patreon.com/id:<campaign_id> as the source URL, which bypasses the vanity-resolution path entirely. Campaign IDs are retrievable from Patreon's /api/campaigns?filter[vanity]=<vanity> endpoint.

This spec automates that workaround: on failure, look up the campaign ID, cache it on the source, and retry the download once in the same task run. Subsequent downloads use the cached ID-URL directly. The user's original URL is preserved for display.

---

Design

Components

1. backend/app/services/patreon_resolver.py — new

Single async function:

async def resolve_campaign_id(vanity: str, cookies_path: Optional[str]) -> Optional[str]:
    """Resolve a Patreon vanity (creator slug) to a campaign ID.

    Returns the campaign ID as a string, or None on any failure (network
    error, auth error, empty response, unexpected shape). Never raises.
    """

Implementation:

• Load cookies with http.cookiejar.MozillaCookieJar(cookies_path).load(ignore_discard=True, ignore_expires=True), then convert to an aiohttp.CookieJar by iterating and calling update_cookies. (aiohttp doesn't read Netscape format natively.)
• GET https://www.patreon.com/api/campaigns?filter[vanity]={vanity}&fields[campaign]=name with header User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 (a stable desktop UA — Patreon rejects requests with suspicious UAs like python-requests/*).
• Parse the JSON response. Success path: data[0].id (string). Log at logging.INFO.
• Any failure — non-200 status, JSON parse error, empty data, missing id field, aiohttp.ClientError, asyncio.TimeoutError — returns None and logs at logging.WARNING with the vanity and the specific reason (HTTP status, exception class, or shape description) for debuggability.
• 10-second total timeout.
• If cookies_path is None or the file doesn't exist, proceed without cookies (the endpoint may still work; the resolver logs a debug note).

2. backend/app/tasks/downloads.py — retry hook + URL rewrite

URL rewrite (outbound path, before every download):

Before calling gallery_dl_service.download(), compute the effective URL:

def _effective_patreon_url(source: Source) -> str:
    if source.platform == "patreon":
        campaign_id = (source.metadata_ or {}).get("patreon_campaign_id")
        if campaign_id:
            return f"https://www.patreon.com/id:{campaign_id}"
    return source.url

Called in _run_single_download (or whichever function assembles the download() call arguments). All Patreon downloads with a cached ID use the id-URL. Non-Patreon platforms and Patreon sources without a cached ID use source.url unchanged.

Retry hook (on first failure, inline):

After dl_result = await gallery_dl_service.download(...), before the DB-write block:

if (
    not dl_result.success
    and source.platform == "patreon"
    and _looks_like_campaign_id_failure(dl_result)
    and "patreon_campaign_id" not in (source.metadata_ or {})
):
    vanity = _extract_vanity_from_url(source.url)
    if vanity:
        logger.info(f"Attempting campaign-ID resolution for {subscription_name}/patreon ({vanity})")
        campaign_id = await resolve_campaign_id(vanity, cookies_path)
        if campaign_id:
            source.metadata_ = {
                **(source.metadata_ or {}),
                "patreon_campaign_id": campaign_id,
            }
            # Retry once with the id URL; overwrite dl_result
            dl_result = await gallery_dl_service.download(
                url=f"https://www.patreon.com/id:{campaign_id}",
                subscription_name=subscription_name,
                platform="patreon",
                source_config=source_config,
                cookies_path=cookies_path,
                auth_token=auth_token,
            )

Helper functions in the same module:

def _looks_like_campaign_id_failure(dl_result) -> bool:
    haystack = f"{dl_result.stdout or ''}\n{dl_result.stderr or ''}".lower()
    return "failed to extract campaign id" in haystack

def _extract_vanity_from_url(url: str) -> Optional[str]:
    # Matches patreon.com/<vanity> and patreon.com/c/<vanity>/... and patreon.com/c/<vanity>
    # Returns None for id:<N> URLs and for anything that doesn't match.
    m = re.match(
        r"https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
        url,
    )
    return m.group(1) if m else None

The retry's dl_result flows into the existing DB-write block unchanged. error_count resets to 0 on retry success, or increments once (not twice) on retry failure — the original failure is discarded.

3. backend/app/services/gallery_dl.py — no change

The caller (downloads.py) supplies the already-effective URL. The service remains platform-agnostic.

4. Data model — no change

Source.metadata_ is already a JSONB dict column. We write one new string key, patreon_campaign_id. No migration.

---

Data flow

First-time failure → auto-heal

1. Scheduled check calls _run_single_download for source {platform: patreon, url: https://www.patreon.com/mstsu, metadata_: {}}.
2. _effective_patreon_url(source) returns source.url (no cache yet).
3. gallery_dl_service.download(url=source.url, ...) returns DownloadResult(success=False, stderr="... Failed to extract campaign ID ...").
4. Retry hook fires: matches patreon platform + error pattern + empty cache.
5. _extract_vanity_from_url(source.url) returns "mstsu".
6. resolve_campaign_id("mstsu", cookies_path) hits the campaigns API, returns "12345".
7. Task writes source.metadata_['patreon_campaign_id'] = "12345" into the in-memory ORM instance (commit happens in the normal DB-write block).
8. gallery_dl_service.download(url="https://www.patreon.com/id:12345", ...) runs; succeeds (typical case).
9. DB-write block sees dl_result.success=True, resets error_count=0, commits the metadata update along with it, publishes download.completed.

Subsequent runs

1. _effective_patreon_url(source) sees patreon_campaign_id=12345, returns https://www.patreon.com/id:12345.
2. Gallery-dl gets the id-URL directly; no vanity lookup, no resolver involvement.
3. Normal download path.

Resolver failure

• resolve_campaign_id returns None (API 401, network error, empty data[], etc.).
• No metadata written, no retry.
• Original dl_result flows through the normal error path: error_count += 1, download.failed event, etc.
• A warning log entry records the vanity and the reason. The user sees the same "Failed to extract campaign ID" error they would have seen without the feature.

Resolver succeeds but retry still fails

• Metadata is still written (the ID is valid; the resolver was not the problem).
• The retry's failure is recorded normally. Next scheduled run uses the id-URL and hits whatever the real issue is (likely auth).
• No further resolver runs for this source.

---

Error handling

Situation	Behavior
Cookies file missing	Resolver proceeds without cookies; may still succeed on public endpoint
Cookies expired (API 401/403)	Resolver returns None, logs warning, original failure surfaced
Vanity regex doesn't match	Retry hook skipped, original failure surfaced
data[] empty in response	Resolver returns None, logs warning
Unexpected JSON shape	Resolver returns None, logs shape at warning
Network error / timeout	Resolver returns None, logs exception class
Resolver + retry both succeed	Normal success path, metadata cached
Resolver succeeds, retry fails	Metadata cached (ID is valid), retry's failure recorded
Source already has patreon_campaign_id but retry fails	No second resolver run (guard on metadata presence)
Non-Patreon source fails with any error	Retry hook skipped (platform gate)
Patreon source fails with a different error (e.g., auth)	Retry hook skipped (pattern gate)

---

Testing

Unit — patreon_resolver.py

• resolve_campaign_id happy path: mock aiohttp response with {"data": [{"id": "12345", "type": "campaign"}]} → returns "12345".
• Empty data[] → returns None, warning logged.
• Missing data key → returns None, warning logged.
• HTTP 401 → returns None, warning logged with status.
• aiohttp.ClientError raised → returns None, warning logged.
• asyncio.TimeoutError → returns None, warning logged.
• cookies_path=None → call proceeds, no cookie jar loaded.

Use aioresponses or aiohttp.test_utils to mock; no real network calls.

Unit — downloads.py helpers

• _looks_like_campaign_id_failure: DownloadResult with the pattern in stderr → True; pattern in stdout → True; neither → False; case-insensitive match.
• _extract_vanity_from_url: patreon.com/mstsu → "mstsu"; patreon.com/c/mstsu/posts → "mstsu"; patreon.com/id:12345 → None; patreon.com/ → None; unrelated URL → None.

Integration — retry hook

Patch app.tasks.downloads.gallery_dl_service.download and app.tasks.downloads.resolve_campaign_id (after importing the symbol into the module, patching the local binding).

• Happy-path heal: First download() returns campaign-ID failure, resolver returns "12345", second download() returns success. Verify: resolver called with correct vanity; source.metadata_['patreon_campaign_id'] == "12345" after commit; second download() called with patreon.com/id:12345 URL; source.error_count == 0; download.completed event published once (for the retry).
• Resolver returns None: First download() returns campaign-ID failure, resolver returns None. Verify: second download() not called; source.metadata_ unchanged (no patreon_campaign_id key); source.error_count == 1; download.failed event published with the original error.
• Already cached: Source has metadata_['patreon_campaign_id'] = "12345" before run. Verify: _effective_patreon_url returns id-URL; first download() called with id-URL; no resolver call regardless of result.
• Non-patreon: Source is subscribestar with similar-looking error text. Verify: retry hook skipped entirely.

Integration — URL-rewrite outbound

• Source with patreon_campaign_id cached → download() called with id-URL.
• Source without cached ID → download() called with source.url.
• Non-Patreon source with patreon_campaign_id somehow set (data corruption) → download() called with source.url (platform gate wins).

---

Files changed

File	Change
backend/app/services/patreon_resolver.py	Create: single resolve_campaign_id(vanity, cookies_path) async function
backend/app/tasks/downloads.py	Add _effective_patreon_url, _looks_like_campaign_id_failure, _extract_vanity_from_url helpers; insert retry hook before DB-write block in _run_single_download; use effective URL when calling gallery_dl_service.download()
tests/services/test_patreon_resolver.py	Create: unit tests for resolver
tests/tasks/test_downloads_patreon_retry.py	Create: integration tests for retry hook and URL-rewrite

No frontend changes. No schema migration. No new API endpoints. No settings changes.

---

Non-goals

• No one-time migration to pre-resolve all existing Patreon sources. Sources auto-heal on their next scheduled check (≤ check_interval, default 8h).
• No UI surface for the campaign ID. It lives in Source.metadata_ and is visible only via the existing metadata inspection path (JSON blob in the DB).
• No retry on the id-URL if the retry itself fails. One resolve + one retry per task run, total.
• No resolver re-run if cached campaign ID ever becomes stale. Manual fix: delete the key from source.metadata_.
• No change to gallery_dl.py — it stays platform-agnostic. The Patreon URL quirk is contained in downloads.py.
• No change to the error-categorization logic in _categorize_error. The "Failed to extract campaign ID" failure is still categorized as UNKNOWN_ERROR if the resolver can't heal it; this is fine because the primary recovery path runs before categorization matters.