# Patreon Campaign-ID Auto-Resolver — Design Spec

**Date:** 2026-04-18
**Scope:** Automatically recover from gallery-dl's "Failed to extract campaign ID" errors on Patreon sources by resolving the campaign ID via Patreon's campaigns API, caching it on the source, and rewriting the URL at download time.

---

## Motivation

Gallery-dl 1.31.10's Patreon extractor fails for some creators with:

```
[patreon][error] Failed to extract campaign ID
```

The failure happens during the vanity-URL → campaign-ID lookup, when Patreon serves an HTML page whose bootstrap JSON is empty or missing. Both gallery-dl's primary JSON parse and its Next.js regex fallback miss, and the extraction aborts.

The upstream-recommended workaround is to use `https://www.patreon.com/id:<campaign_id>` as the source URL, which bypasses the vanity-resolution path entirely. Campaign IDs are retrievable from Patreon's `/api/campaigns?filter[vanity]=<vanity>` endpoint.

This spec automates that workaround: on failure, look up the campaign ID, cache it on the source, and retry the download once in the same task run. Subsequent downloads use the cached ID-URL directly. The user's original URL is preserved for display.

---

## Design

### Components

#### 1. `backend/app/services/patreon_resolver.py` — new

Single async function:

```python
async def resolve_campaign_id(vanity: str, cookies_path: Optional[str]) -> Optional[str]:
    """Resolve a Patreon vanity (creator slug) to a campaign ID.

    Returns the campaign ID as a string, or None on any failure (network
    error, auth error, empty response, unexpected shape). Never raises.
    """
```

Implementation:

- Load cookies with `http.cookiejar.MozillaCookieJar(cookies_path).load(ignore_discard=True, ignore_expires=True)`, then convert to an `aiohttp.CookieJar` by iterating and calling `update_cookies`. (aiohttp doesn't read Netscape format natively.)
- `GET https://www.patreon.com/api/campaigns?filter[vanity]={vanity}&fields[campaign]=name` with header `User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36` (a stable desktop UA — Patreon rejects requests with suspicious UAs like `python-requests/*`).
- Parse the JSON response. Success path: `data[0].id` (string). Log at `logging.INFO`.
- Any failure — non-200 status, JSON parse error, empty `data`, missing `id` field, `aiohttp.ClientError`, `asyncio.TimeoutError` — returns `None` and logs at `logging.WARNING` with the vanity and the specific reason (HTTP status, exception class, or shape description) for debuggability.
- 10-second total timeout.
- If `cookies_path` is `None` or the file doesn't exist, proceed without cookies (the endpoint may still work; the resolver logs a debug note).

#### 2. `backend/app/tasks/downloads.py` — retry hook + URL rewrite

**URL rewrite (outbound path, before every download):**

Before calling `gallery_dl_service.download()`, compute the effective URL:

```python
def _effective_patreon_url(source: Source) -> str:
    if source.platform == "patreon":
        campaign_id = (source.metadata_ or {}).get("patreon_campaign_id")
        if campaign_id:
            return f"https://www.patreon.com/id:{campaign_id}"
    return source.url
```

Called in `_run_single_download` (or whichever function assembles the `download()` call arguments). All Patreon downloads with a cached ID use the id-URL. Non-Patreon platforms and Patreon sources without a cached ID use `source.url` unchanged.

**Retry hook (on first failure, inline):**

After `dl_result = await gallery_dl_service.download(...)`, before the DB-write block:

```python
if (
    not dl_result.success
    and source.platform == "patreon"
    and _looks_like_campaign_id_failure(dl_result)
    and "patreon_campaign_id" not in (source.metadata_ or {})
):
    vanity = _extract_vanity_from_url(source.url)
    if vanity:
        logger.info(f"Attempting campaign-ID resolution for {subscription_name}/patreon ({vanity})")
        campaign_id = await resolve_campaign_id(vanity, cookies_path)
        if campaign_id:
            source.metadata_ = {
                **(source.metadata_ or {}),
                "patreon_campaign_id": campaign_id,
            }
            # Retry once with the id URL; overwrite dl_result
            dl_result = await gallery_dl_service.download(
                url=f"https://www.patreon.com/id:{campaign_id}",
                subscription_name=subscription_name,
                platform="patreon",
                source_config=source_config,
                cookies_path=cookies_path,
                auth_token=auth_token,
            )
```

Helper functions in the same module:

```python
def _looks_like_campaign_id_failure(dl_result) -> bool:
    haystack = f"{dl_result.stdout or ''}\n{dl_result.stderr or ''}".lower()
    return "failed to extract campaign id" in haystack

def _extract_vanity_from_url(url: str) -> Optional[str]:
    # Matches patreon.com/<vanity> and patreon.com/c/<vanity>/... and patreon.com/c/<vanity>
    # Returns None for id:<N> URLs and for anything that doesn't match.
    m = re.match(
        r"https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
        url,
    )
    return m.group(1) if m else None
```

The retry's `dl_result` flows into the existing DB-write block unchanged. `error_count` resets to 0 on retry success, or increments once (not twice) on retry failure — the original failure is discarded.

#### 3. `backend/app/services/gallery_dl.py` — no change

The caller (`downloads.py`) supplies the already-effective URL. The service remains platform-agnostic.

#### 4. Data model — no change

`Source.metadata_` is already a JSONB dict column. We write one new string key, `patreon_campaign_id`. No migration.

---

## Data flow

### First-time failure → auto-heal

1. Scheduled check calls `_run_single_download` for source `{platform: patreon, url: https://www.patreon.com/mstsu, metadata_: {}}`.
2. `_effective_patreon_url(source)` returns `source.url` (no cache yet).
3. `gallery_dl_service.download(url=source.url, ...)` returns `DownloadResult(success=False, stderr="... Failed to extract campaign ID ...")`.
4. Retry hook fires: matches `patreon` platform + error pattern + empty cache.
5. `_extract_vanity_from_url(source.url)` returns `"mstsu"`.
6. `resolve_campaign_id("mstsu", cookies_path)` hits the campaigns API, returns `"12345"`.
7. Task writes `source.metadata_['patreon_campaign_id'] = "12345"` into the in-memory ORM instance (commit happens in the normal DB-write block).
8. `gallery_dl_service.download(url="https://www.patreon.com/id:12345", ...)` runs; succeeds (typical case).
9. DB-write block sees `dl_result.success=True`, resets `error_count=0`, commits the metadata update along with it, publishes `download.completed`.

### Subsequent runs

1. `_effective_patreon_url(source)` sees `patreon_campaign_id=12345`, returns `https://www.patreon.com/id:12345`.
2. Gallery-dl gets the id-URL directly; no vanity lookup, no resolver involvement.
3. Normal download path.

### Resolver failure

- `resolve_campaign_id` returns `None` (API 401, network error, empty `data[]`, etc.).
- No metadata written, no retry.
- Original `dl_result` flows through the normal error path: `error_count += 1`, `download.failed` event, etc.
- A warning log entry records the vanity and the reason. The user sees the same "Failed to extract campaign ID" error they would have seen without the feature.

### Resolver succeeds but retry still fails

- Metadata is still written (the ID is valid; the resolver was not the problem).
- The retry's failure is recorded normally. Next scheduled run uses the id-URL and hits whatever the real issue is (likely auth).
- No further resolver runs for this source.

---

## Error handling

| Situation | Behavior |
|---|---|
| Cookies file missing | Resolver proceeds without cookies; may still succeed on public endpoint |
| Cookies expired (API 401/403) | Resolver returns None, logs warning, original failure surfaced |
| Vanity regex doesn't match | Retry hook skipped, original failure surfaced |
| `data[]` empty in response | Resolver returns None, logs warning |
| Unexpected JSON shape | Resolver returns None, logs shape at warning |
| Network error / timeout | Resolver returns None, logs exception class |
| Resolver + retry both succeed | Normal success path, metadata cached |
| Resolver succeeds, retry fails | Metadata cached (ID is valid), retry's failure recorded |
| Source already has `patreon_campaign_id` but retry fails | No second resolver run (guard on metadata presence) |
| Non-Patreon source fails with any error | Retry hook skipped (platform gate) |
| Patreon source fails with a different error (e.g., auth) | Retry hook skipped (pattern gate) |

---

## Testing

### Unit — `patreon_resolver.py`

- `resolve_campaign_id` happy path: mock `aiohttp` response with `{"data": [{"id": "12345", "type": "campaign"}]}` → returns `"12345"`.
- Empty `data[]` → returns `None`, warning logged.
- Missing `data` key → returns `None`, warning logged.
- HTTP 401 → returns `None`, warning logged with status.
- `aiohttp.ClientError` raised → returns `None`, warning logged.
- `asyncio.TimeoutError` → returns `None`, warning logged.
- `cookies_path=None` → call proceeds, no cookie jar loaded.

Use `aioresponses` or `aiohttp.test_utils` to mock; no real network calls.

### Unit — `downloads.py` helpers

- `_looks_like_campaign_id_failure`: `DownloadResult` with the pattern in stderr → True; pattern in stdout → True; neither → False; case-insensitive match.
- `_extract_vanity_from_url`: `patreon.com/mstsu` → `"mstsu"`; `patreon.com/c/mstsu/posts` → `"mstsu"`; `patreon.com/id:12345` → `None`; `patreon.com/` → `None`; unrelated URL → `None`.

### Integration — retry hook

Patch `app.tasks.downloads.gallery_dl_service.download` and `app.tasks.downloads.resolve_campaign_id` (after importing the symbol into the module, patching the local binding).

- **Happy-path heal:** First `download()` returns campaign-ID failure, resolver returns `"12345"`, second `download()` returns success. Verify: resolver called with correct vanity; `source.metadata_['patreon_campaign_id'] == "12345"` after commit; second `download()` called with `patreon.com/id:12345` URL; `source.error_count == 0`; `download.completed` event published once (for the retry).
- **Resolver returns None:** First `download()` returns campaign-ID failure, resolver returns `None`. Verify: second `download()` not called; `source.metadata_` unchanged (no `patreon_campaign_id` key); `source.error_count == 1`; `download.failed` event published with the original error.
- **Already cached:** Source has `metadata_['patreon_campaign_id'] = "12345"` before run. Verify: `_effective_patreon_url` returns id-URL; first `download()` called with id-URL; no resolver call regardless of result.
- **Non-patreon:** Source is `subscribestar` with similar-looking error text. Verify: retry hook skipped entirely.

### Integration — URL-rewrite outbound

- Source with `patreon_campaign_id` cached → `download()` called with id-URL.
- Source without cached ID → `download()` called with `source.url`.
- Non-Patreon source with `patreon_campaign_id` somehow set (data corruption) → `download()` called with `source.url` (platform gate wins).

---

## Files changed

| File | Change |
|---|---|
| `backend/app/services/patreon_resolver.py` | Create: single `resolve_campaign_id(vanity, cookies_path)` async function |
| `backend/app/tasks/downloads.py` | Add `_effective_patreon_url`, `_looks_like_campaign_id_failure`, `_extract_vanity_from_url` helpers; insert retry hook before DB-write block in `_run_single_download`; use effective URL when calling `gallery_dl_service.download()` |
| `tests/services/test_patreon_resolver.py` | Create: unit tests for resolver |
| `tests/tasks/test_downloads_patreon_retry.py` | Create: integration tests for retry hook and URL-rewrite |

No frontend changes. No schema migration. No new API endpoints. No settings changes.

---

## Non-goals

- No one-time migration to pre-resolve all existing Patreon sources. Sources auto-heal on their next scheduled check (≤ `check_interval`, default 8h).
- No UI surface for the campaign ID. It lives in `Source.metadata_` and is visible only via the existing metadata inspection path (JSON blob in the DB).
- No retry on the id-URL if the retry itself fails. One resolve + one retry per task run, total.
- No resolver re-run if cached campaign ID ever becomes stale. Manual fix: delete the key from `source.metadata_`.
- No change to `gallery_dl.py` — it stays platform-agnostic. The Patreon URL quirk is contained in `downloads.py`.
- No change to the error-categorization logic in `_categorize_error`. The "Failed to extract campaign ID" failure is still categorized as `UNKNOWN_ERROR` if the resolver can't heal it; this is fine because the primary recovery path runs before categorization matters.