feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)

Final step of the native Patreon ingester: a first-class Recovery action,
and removal of the now-dead gallery-dl Patreon path.

Recovery (rules #23/#24/#27 — full product, with UI):
- source_service.start_recovery arms the #693 backfill state machine PLUS
  `_backfill_bypass_seen`, flipping download mode to recovery (bypass the
  seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate
  under the current pHash threshold). Stop via the shared stop_backfill.
- SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill
  gains action="recover".
- Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow
  + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling
  (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource +
  SubscriptionsTab onRecover.

Cutover (rule #22 — no legacy):
- gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon
  files/cursor branch in _build_config_for_source, and the patreon/Mux
  yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the
  other platforms' yt-dlp fetches; native ingester owns it now).
- download_service: removed the dead campaign-id-retry helpers
  (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and
  _effective_url. Vanity→campaign resolution + resume_cursor + the
  cursor/PARTIAL lifecycle stay — they serve the native ingester.

Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp
Referer, resume-cursor); repointed the generic skip-value config tests to
subscribestar; added start_recovery + recover-endpoint coverage
(backfill_bypass_seen). gallery-dl stays for the other 5 platforms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

backend/app/api/sources.py

@@ -122,23 +122,30 @@ async def delete_source(source_id: int):
 
 @sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
 async def set_backfill(source_id: int):
-    """Plan #693: start or stop a run-until-done backfill. Body:
-    `{"action": "start" | "stop"}` (default "start"). 'start' walks the full
-    post history in time-boxed chunks until it reaches the bottom (then the
-    source shows 'complete'); 'stop' cancels back to tick mode. Returns the
-    updated source dict (incl. backfill_state / backfill_chunks)."""
+    """Plan #693/#697: start/stop a run-until-done backfill, or start a recovery.
+    Body: `{"action": "start" | "stop" | "recover"}` (default "start"). 'start'
+    walks the full post history in time-boxed chunks until it reaches the bottom
+    (then the source shows 'complete'); 'recover' is the same walk but bypasses
+    the Patreon seen-ledger to re-fetch dropped-and-deleted near-dups under the
+    current pHash threshold; 'stop' cancels either back to tick mode. Returns the
+    updated source dict (incl. backfill_state / backfill_chunks /
+    backfill_bypass_seen)."""
     payload = await request.get_json(silent=True) or {}
     action = payload.get("action", "start")
-    if action not in ("start", "stop"):
-        return _bad("invalid_action", detail="action must be 'start' or 'stop'")
+    if action not in ("start", "stop", "recover"):
+        return _bad(
+            "invalid_action",
+            detail="action must be 'start', 'stop', or 'recover'",
+        )
     async with get_session() as session:
         try:
             svc = SourceService(session)
-            record = (
-                await svc.start_backfill(source_id)
-                if action == "start"
-                else await svc.stop_backfill(source_id)
-            )
+            if action == "start":
+                record = await svc.start_backfill(source_id)
+            elif action == "recover":
+                record = await svc.start_recovery(source_id)
+            else:
+                record = await svc.stop_backfill(source_id)
         except LookupError:
             return _bad("not_found", status=404)
     return jsonify(record.to_dict())

backend/app/services/download_service.py

@@ -44,11 +44,14 @@ from .scheduler_service import set_platform_cooldown
 log = logging.getLogger(__name__)
 
 
+# Vanity → campaign-id resolution is still needed by the native Patreon
+# ingester (phase 2 resolves the campaign id before the walk). gallery-dl's
+# reactive campaign-id retry + the `id:` effective-URL rewrite were removed at
+# the #697 cutover (Patreon no longer flows through gallery-dl).
 _PATREON_VANITY_RE = re.compile(
     r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
     re.IGNORECASE,
 )
-_CAMPAIGN_ID_FAILURE_PATTERN = "failed to extract campaign id"
 
 
 def _extract_patreon_vanity(url: str) -> str | None:
@@ -56,16 +59,6 @@ def _extract_patreon_vanity(url: str) -> str | None:
     return m.group(1) if m else None
 
 
-def _looks_like_campaign_id_failure(stdout: str, stderr: str) -> bool:
-    return _CAMPAIGN_ID_FAILURE_PATTERN in f"{stdout}\n{stderr}".lower()
-
-
-def _effective_url(platform: str, source_url: str, overrides: dict) -> str:
-    if platform == "patreon" and overrides.get("patreon_campaign_id"):
-        return f"https://www.patreon.com/id:{overrides['patreon_campaign_id']}"
-    return source_url
-
-
 class DownloadService:
     """Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
 
@@ -160,11 +153,8 @@ class DownloadService:
                 ctx, source_config, mode,
             )
         else:
-            effective_url = _effective_url(
-                ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
-            )
             dl_result = await self.gdl.download(
-                url=effective_url,
+                url=ctx["url"],
                 artist_slug=ctx["artist_slug"],
                 platform=ctx["platform"],
                 source_config=source_config,

backend/app/services/gallery_dl.py

@@ -107,12 +107,14 @@ class SourceConfig:
 
     `resume_cursor` is RUNTIME state, not persisted operator config: the
     cursor-paged backfill checkpoint (see parse_last_cursor + the
-    download_service backfill lifecycle). When set on a patreon backfill
-    run, gallery-dl resumes its newest→oldest walk from that pagination
-    cursor instead of restarting from the top. It is threaded by
-    download_service from config_overrides["_backfill_cursor"]; SourceConfig
-    deliberately does NOT read it in from_dict (config_overrides also holds
-    operator config, and resume_cursor must only apply in backfill mode).
+    download_service backfill lifecycle). It is consumed only by the native
+    Patreon ingester (the sole cursor-paged platform; gallery-dl's Patreon path
+    was removed at the #697 cutover): on a backfill/recovery run the ingester
+    resumes its newest→oldest walk from that pagination cursor instead of
+    restarting from the top. download_service threads it from
+    config_overrides["_backfill_cursor"]; SourceConfig deliberately does NOT read
+    it in from_dict (config_overrides also holds operator config, and
+    resume_cursor must only apply in backfill/recovery mode).
     """
     content_types: list[str] = field(default_factory=lambda: ["all"])
     sleep: float | None = None
@@ -170,13 +172,14 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
     return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
 
 
-# gallery-dl logs its pagination cursor (when extractor.*.cursor is truthy)
-# as `Cursor: <token>` — once per fetched page, at debug verbosity. The LAST
-# such line is the furthest-progressed page, i.e. the resume point. We scan
-# both streams (the debug log lands on stderr, but be defensive) and take the
-# final match; passing it back as extractor.patreon.cursor resumes the walk.
-# Survives a timed-out run too: the TimeoutExpired path returns the partial
-# stdout/stderr, so an interrupted backfill still yields its last cursor.
+# The native Patreon ingester emits its pagination cursor as `Cursor: <token>`
+# — one line per fetched page — into the DownloadResult stdout (mirroring the
+# convention gallery-dl used before the #697 cutover, so the backfill lifecycle
+# stayed unchanged). The LAST such line is the furthest-progressed page, i.e.
+# the resume point. We scan both streams (be defensive) and take the final
+# match; download_service checkpoints it as the next chunk's resume_cursor.
+# Survives a time-boxed chunk too: the ingester emits the cursor for a page when
+# it STARTS it, so an interrupted walk still yields its last cursor.
 _CURSOR_RE = re.compile(r"Cursor:\s*(\S+)")
 
 
@@ -218,16 +221,10 @@ class GalleryDLService:
         "permission denied", "tier required", "pledge required",
     ]
 
-    # Per-platform defaults. Lifted from GS — same six platforms FC supports.
+    # Per-platform defaults for the gallery-dl-backed platforms. Patreon was
+    # removed at the plan-#697 cutover — it now uses the native ingester
+    # (services/patreon_ingester.py), not gallery-dl.
     PLATFORM_DEFAULTS = {
-        "patreon": {
-            "content_types": ["images", "image_large", "attachments", "postfile", "content"],
-            "directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
-            "filename": "{num:>02}_{filename}.{extension}",
-            "videos": True,
-            "embeds": True,
-            "cursor": True,
-        },
         "subscribestar": {
             "content_types": ["all"],
             "directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
@@ -312,25 +309,11 @@ class GalleryDLService:
                 # transfer cap — large GIFs that keep streaming are unaffected.
                 "retries": 2,
                 "timeout": 60.0,
-                # Forward Patreon as Referer/Origin to yt-dlp when it
-                # fetches video manifests. Operator-flagged 2026-06-01
-                # (DaferQ patreon): video posts hosted on Mux carry a JWT
-                # playback restriction that checks Referer/Origin on every
-                # request — not just the token signature. gallery-dl's
-                # HEAD probe to stream.mux.com returns 200 (the token is
-                # valid), but yt-dlp's actual GET-with-Range to fetch the
-                # m3u8 manifest 403s because yt-dlp sends its own default
-                # Referer, which Mux's policy rejects. Forcing the right
-                # headers fixes the headers-only case; Mux IP-range
-                # restrictions are unfixable from here.
-                "ytdl": {
-                    "raw-options": {
-                        "http_headers": {
-                            "Referer": "https://www.patreon.com/",
-                            "Origin": "https://www.patreon.com",
-                        },
-                    },
-                },
+                # NOTE: the Patreon/Mux yt-dlp Referer/Origin forwarding lived
+                # here until the plan-#697 cutover. It was Patreon-specific (and
+                # would have wrongly tagged the other platforms' yt-dlp fetches);
+                # Patreon video is now handled by the native ingester's
+                # downloader (patreon_downloader._VIDEO_HEADERS), so it's gone.
             },
             "output": {"progress": True},
         }
@@ -382,23 +365,7 @@ class GalleryDLService:
 
         platform_section = config["extractor"].setdefault(platform, {})
 
-        if platform == "patreon":
-            if "all" in source_config.content_types:
-                platform_section["files"] = [
-                    "images", "image_large", "attachments", "postfile", "content",
-                ]
-            else:
-                platform_section["files"] = source_config.content_types
-            # Cursor-paged backfill: resume the newest→oldest walk from the
-            # checkpoint saved by the previous run instead of restarting from
-            # the top. Without this, a large catalog (Anduo's weekly
-            # image-dense Reports) never reaches its frontier inside the
-            # subprocess budget — the HEAD walk alone times out, 0 files
-            # written, no progress (event #40411). The PLATFORM_DEFAULTS leave
-            # `cursor: True` (log-only) for the first/fresh run.
-            if source_config.resume_cursor:
-                platform_section["cursor"] = source_config.resume_cursor
-        elif platform == "hentaifoundry":
+        if platform == "hentaifoundry":
             if "pictures" in source_config.content_types or "all" in source_config.content_types:
                 platform_section["include"] = "all"
 

backend/app/services/patreon_downloader.py

@@ -50,10 +50,11 @@ _TITLE_MAX = 40
 _TIMEOUT_SECONDS = 120.0
 _CHUNK = 1 << 16
 
-# Same Referer/Origin gallery-dl forwards to yt-dlp for Mux-hosted Patreon
-# video (gallery_dl.py _get_default_config -> downloader.ytdl.raw-options).
-# Mux's JWT playback policy checks Referer/Origin on every request, so yt-dlp
-# must send Patreon's, not its own default.
+# Referer/Origin yt-dlp must send for Mux-hosted Patreon video. Mux's JWT
+# playback policy checks Referer/Origin on every request, so yt-dlp must send
+# Patreon's, not its own default. (gallery-dl forwarded the same headers before
+# the #697 cutover removed its Patreon path; this is now the only place they
+# live.)
 _VIDEO_HEADERS = {
     "Referer": "https://www.patreon.com/",
     "Origin": "https://www.patreon.com",

backend/app/services/source_service.py

@@ -67,6 +67,9 @@ class SourceRecord:
     # plan #693: derived from config_overrides for the UI badge.
     backfill_state: str | None  # "running" | "complete" | "stalled" | None (idle)
     backfill_chunks: int
+    # plan #697: a running deep-walk that bypasses the Patreon seen-ledger
+    # (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
+    backfill_bypass_seen: bool
 
     def to_dict(self) -> dict:
         return {
@@ -87,6 +90,7 @@ class SourceRecord:
             "backfill_runs_remaining": self.backfill_runs_remaining,
             "backfill_state": self.backfill_state,
             "backfill_chunks": self.backfill_chunks,
+            "backfill_bypass_seen": self.backfill_bypass_seen,
         }
 
 
@@ -162,6 +166,7 @@ class SourceService:
             backfill_runs_remaining=source.backfill_runs_remaining or 0,
             backfill_state=co.get("_backfill_state"),
             backfill_chunks=int(co.get("_backfill_chunks", 0)),
+            backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
         )
 
     async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -319,6 +324,34 @@ class SourceService:
         await self.session.commit()
         return await self._row_to_record(source)
 
+    async def start_recovery(self, source_id: int) -> SourceRecord:
+        """Plan #697: arm a RECOVERY walk — a backfill that bypasses the Patreon
+        seen-ledger so deliberately-dropped-and-deleted near-dups get re-fetched
+        and re-evaluated under the CURRENT pHash threshold (tier-2 disk still
+        spares files we kept). Reuses the entire #693 backfill state machine
+        (time-boxed chunks, cursor checkpoint, complete/stall lifecycle) plus the
+        `_backfill_bypass_seen` flag that flips download mode to recovery. Clears
+        any prior cursor/chunk/stall state so it walks fresh from the top. The
+        flag is cleared on completion (download_service) and on stop.
+
+        Recovery is Patreon-only (the seen-ledger is Patreon's); for other
+        platforms the flag is inert (download_service ignores it) and the walk
+        runs as a plain backfill. The UI gates the action to Patreon sources."""
+        source = (await self.session.execute(
+            select(Source).where(Source.id == source_id)
+        )).scalar_one_or_none()
+        if source is None:
+            raise LookupError(f"source id={source_id} not found")
+        co = dict(source.config_overrides or {})
+        co["_backfill_state"] = "running"
+        co["_backfill_bypass_seen"] = True
+        for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks"):
+            co.pop(k, None)
+        source.config_overrides = co
+        source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
+        await self.session.commit()
+        return await self._row_to_record(source)
+
     async def stop_backfill(self, source_id: int) -> SourceRecord:
         """Plan #693: cancel an in-progress backfill — back to idle/tick mode.
         Clears the running state + cursor/chunk/stall bookkeeping."""