feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -260,7 +260,7 @@ def test_build_config_emits_tick_skip_value(gdl):
|
||||
scans once 20 contiguous archived items are seen."""
|
||||
from backend.app.services.gallery_dl import TICK_SKIP_VALUE
|
||||
cfg = gdl._build_config_for_source(
|
||||
platform="patreon",
|
||||
platform="subscribestar",
|
||||
source_config=SourceConfig(),
|
||||
artist_slug="alice",
|
||||
skip_value=TICK_SKIP_VALUE,
|
||||
@@ -273,7 +273,7 @@ def test_build_config_emits_backfill_skip_value(gdl):
|
||||
full post history."""
|
||||
from backend.app.services.gallery_dl import BACKFILL_SKIP_VALUE
|
||||
cfg = gdl._build_config_for_source(
|
||||
platform="patreon",
|
||||
platform="subscribestar",
|
||||
source_config=SourceConfig(),
|
||||
artist_slug="alice",
|
||||
skip_value=BACKFILL_SKIP_VALUE,
|
||||
@@ -318,23 +318,17 @@ def test_categorize_tier_limited_wins_over_partial(gdl):
|
||||
assert etype == ErrorType.TIER_LIMITED
|
||||
|
||||
|
||||
def test_default_config_forwards_patreon_referer_to_ytdl(gdl):
|
||||
"""Operator-flagged 2026-06-01: Mux video playback restrictions reject
|
||||
yt-dlp's manifest fetch when Referer/Origin don't match Patreon. The
|
||||
fix is a static `downloader.ytdl.raw-options.http_headers` block, so
|
||||
it's enough to assert the keys land in the default config."""
|
||||
cfg = gdl._get_default_config()
|
||||
headers = cfg["downloader"]["ytdl"]["raw-options"]["http_headers"]
|
||||
assert headers["Referer"] == "https://www.patreon.com/"
|
||||
assert headers["Origin"] == "https://www.patreon.com"
|
||||
|
||||
|
||||
# --- Cursor-paged backfill (plan #689) -------------------------------------
|
||||
# --- Cursor parsing (plan #689 / native ingester #697) ---------------------
|
||||
# parse_last_cursor is retained post-cutover: the native Patreon ingester emits
|
||||
# `Cursor: <token>` per page into DownloadResult.stdout and download_service
|
||||
# checkpoints the last one. (The gallery-dl Patreon `files`/`cursor` config and
|
||||
# the Mux yt-dlp Referer block were removed at the #697 cutover, along with the
|
||||
# tests that pinned them.)
|
||||
|
||||
|
||||
def test_parse_last_cursor_returns_last_match_across_streams():
|
||||
# gallery-dl logs `Cursor: <token>` per page (debug → stderr); the last
|
||||
# is the furthest-progressed resume point.
|
||||
# One `Cursor: <token>` line per page; the last is the furthest-progressed
|
||||
# resume point.
|
||||
stderr = (
|
||||
"[patreon][debug] Cursor: 03:AAAA:bbb\n"
|
||||
"[urllib3] GET ...\n"
|
||||
@@ -347,18 +341,3 @@ def test_parse_last_cursor_scans_stdout_too_and_none_when_absent():
|
||||
assert parse_last_cursor("Cursor: 99:ZZZ", "") == "99:ZZZ"
|
||||
assert parse_last_cursor("no marker here", "still nothing") is None
|
||||
assert parse_last_cursor("", "") is None
|
||||
|
||||
|
||||
def test_build_config_patreon_resume_cursor_overrides_default(gdl):
|
||||
# No resume cursor → PLATFORM_DEFAULTS leaves the log-only `cursor: True`.
|
||||
fresh = gdl._build_config_for_source(
|
||||
"patreon", SourceConfig(), "alice", skip_value=True,
|
||||
)
|
||||
assert fresh["extractor"]["patreon"]["cursor"] is True
|
||||
|
||||
# Resume cursor set → gallery-dl restarts the walk from that page.
|
||||
resumed = gdl._build_config_for_source(
|
||||
"patreon", SourceConfig(resume_cursor="03:CCCC:ddd"), "alice",
|
||||
skip_value=True,
|
||||
)
|
||||
assert resumed["extractor"]["patreon"]["cursor"] == "03:CCCC:ddd"
|
||||
|
||||
Reference in New Issue
Block a user