feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -44,11 +44,14 @@ from .scheduler_service import set_platform_cooldown
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Vanity → campaign-id resolution is still needed by the native Patreon
|
||||
# ingester (phase 2 resolves the campaign id before the walk). gallery-dl's
|
||||
# reactive campaign-id retry + the `id:` effective-URL rewrite were removed at
|
||||
# the #697 cutover (Patreon no longer flows through gallery-dl).
|
||||
_PATREON_VANITY_RE = re.compile(
|
||||
r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_CAMPAIGN_ID_FAILURE_PATTERN = "failed to extract campaign id"
|
||||
|
||||
|
||||
def _extract_patreon_vanity(url: str) -> str | None:
|
||||
@@ -56,16 +59,6 @@ def _extract_patreon_vanity(url: str) -> str | None:
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def _looks_like_campaign_id_failure(stdout: str, stderr: str) -> bool:
|
||||
return _CAMPAIGN_ID_FAILURE_PATTERN in f"{stdout}\n{stderr}".lower()
|
||||
|
||||
|
||||
def _effective_url(platform: str, source_url: str, overrides: dict) -> str:
|
||||
if platform == "patreon" and overrides.get("patreon_campaign_id"):
|
||||
return f"https://www.patreon.com/id:{overrides['patreon_campaign_id']}"
|
||||
return source_url
|
||||
|
||||
|
||||
class DownloadService:
|
||||
"""Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
|
||||
|
||||
@@ -160,11 +153,8 @@ class DownloadService:
|
||||
ctx, source_config, mode,
|
||||
)
|
||||
else:
|
||||
effective_url = _effective_url(
|
||||
ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
|
||||
)
|
||||
dl_result = await self.gdl.download(
|
||||
url=effective_url,
|
||||
url=ctx["url"],
|
||||
artist_slug=ctx["artist_slug"],
|
||||
platform=ctx["platform"],
|
||||
source_config=source_config,
|
||||
|
||||
Reference in New Issue
Block a user