feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
Final step of the native Patreon ingester: a first-class Recovery action, and removal of the now-dead gallery-dl Patreon path. Recovery (rules #23/#24/#27 — full product, with UI): - source_service.start_recovery arms the #693 backfill state machine PLUS `_backfill_bypass_seen`, flipping download mode to recovery (bypass the seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate under the current pHash threshold). Stop via the shared stop_backfill. - SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill gains action="recover". - Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource + SubscriptionsTab onRecover. Cutover (rule #22 — no legacy): - gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon files/cursor branch in _build_config_for_source, and the patreon/Mux yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the other platforms' yt-dlp fetches; native ingester owns it now). - download_service: removed the dead campaign-id-retry helpers (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and _effective_url. Vanity→campaign resolution + resume_cursor + the cursor/PARTIAL lifecycle stay — they serve the native ingester. Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp Referer, resume-cursor); repointed the generic skip-value config tests to subscribestar; added start_recovery + recover-endpoint coverage (backfill_bypass_seen). gallery-dl stays for the other 5 platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -107,12 +107,14 @@ class SourceConfig:
|
||||
|
||||
`resume_cursor` is RUNTIME state, not persisted operator config: the
|
||||
cursor-paged backfill checkpoint (see parse_last_cursor + the
|
||||
download_service backfill lifecycle). When set on a patreon backfill
|
||||
run, gallery-dl resumes its newest→oldest walk from that pagination
|
||||
cursor instead of restarting from the top. It is threaded by
|
||||
download_service from config_overrides["_backfill_cursor"]; SourceConfig
|
||||
deliberately does NOT read it in from_dict (config_overrides also holds
|
||||
operator config, and resume_cursor must only apply in backfill mode).
|
||||
download_service backfill lifecycle). It is consumed only by the native
|
||||
Patreon ingester (the sole cursor-paged platform; gallery-dl's Patreon path
|
||||
was removed at the #697 cutover): on a backfill/recovery run the ingester
|
||||
resumes its newest→oldest walk from that pagination cursor instead of
|
||||
restarting from the top. download_service threads it from
|
||||
config_overrides["_backfill_cursor"]; SourceConfig deliberately does NOT read
|
||||
it in from_dict (config_overrides also holds operator config, and
|
||||
resume_cursor must only apply in backfill/recovery mode).
|
||||
"""
|
||||
content_types: list[str] = field(default_factory=lambda: ["all"])
|
||||
sleep: float | None = None
|
||||
@@ -170,13 +172,14 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
|
||||
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
|
||||
|
||||
|
||||
# gallery-dl logs its pagination cursor (when extractor.*.cursor is truthy)
|
||||
# as `Cursor: <token>` — once per fetched page, at debug verbosity. The LAST
|
||||
# such line is the furthest-progressed page, i.e. the resume point. We scan
|
||||
# both streams (the debug log lands on stderr, but be defensive) and take the
|
||||
# final match; passing it back as extractor.patreon.cursor resumes the walk.
|
||||
# Survives a timed-out run too: the TimeoutExpired path returns the partial
|
||||
# stdout/stderr, so an interrupted backfill still yields its last cursor.
|
||||
# The native Patreon ingester emits its pagination cursor as `Cursor: <token>`
|
||||
# — one line per fetched page — into the DownloadResult stdout (mirroring the
|
||||
# convention gallery-dl used before the #697 cutover, so the backfill lifecycle
|
||||
# stayed unchanged). The LAST such line is the furthest-progressed page, i.e.
|
||||
# the resume point. We scan both streams (be defensive) and take the final
|
||||
# match; download_service checkpoints it as the next chunk's resume_cursor.
|
||||
# Survives a time-boxed chunk too: the ingester emits the cursor for a page when
|
||||
# it STARTS it, so an interrupted walk still yields its last cursor.
|
||||
_CURSOR_RE = re.compile(r"Cursor:\s*(\S+)")
|
||||
|
||||
|
||||
@@ -218,16 +221,10 @@ class GalleryDLService:
|
||||
"permission denied", "tier required", "pledge required",
|
||||
]
|
||||
|
||||
# Per-platform defaults. Lifted from GS — same six platforms FC supports.
|
||||
# Per-platform defaults for the gallery-dl-backed platforms. Patreon was
|
||||
# removed at the plan-#697 cutover — it now uses the native ingester
|
||||
# (services/patreon_ingester.py), not gallery-dl.
|
||||
PLATFORM_DEFAULTS = {
|
||||
"patreon": {
|
||||
"content_types": ["images", "image_large", "attachments", "postfile", "content"],
|
||||
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
|
||||
"filename": "{num:>02}_{filename}.{extension}",
|
||||
"videos": True,
|
||||
"embeds": True,
|
||||
"cursor": True,
|
||||
},
|
||||
"subscribestar": {
|
||||
"content_types": ["all"],
|
||||
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
|
||||
@@ -312,25 +309,11 @@ class GalleryDLService:
|
||||
# transfer cap — large GIFs that keep streaming are unaffected.
|
||||
"retries": 2,
|
||||
"timeout": 60.0,
|
||||
# Forward Patreon as Referer/Origin to yt-dlp when it
|
||||
# fetches video manifests. Operator-flagged 2026-06-01
|
||||
# (DaferQ patreon): video posts hosted on Mux carry a JWT
|
||||
# playback restriction that checks Referer/Origin on every
|
||||
# request — not just the token signature. gallery-dl's
|
||||
# HEAD probe to stream.mux.com returns 200 (the token is
|
||||
# valid), but yt-dlp's actual GET-with-Range to fetch the
|
||||
# m3u8 manifest 403s because yt-dlp sends its own default
|
||||
# Referer, which Mux's policy rejects. Forcing the right
|
||||
# headers fixes the headers-only case; Mux IP-range
|
||||
# restrictions are unfixable from here.
|
||||
"ytdl": {
|
||||
"raw-options": {
|
||||
"http_headers": {
|
||||
"Referer": "https://www.patreon.com/",
|
||||
"Origin": "https://www.patreon.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
# NOTE: the Patreon/Mux yt-dlp Referer/Origin forwarding lived
|
||||
# here until the plan-#697 cutover. It was Patreon-specific (and
|
||||
# would have wrongly tagged the other platforms' yt-dlp fetches);
|
||||
# Patreon video is now handled by the native ingester's
|
||||
# downloader (patreon_downloader._VIDEO_HEADERS), so it's gone.
|
||||
},
|
||||
"output": {"progress": True},
|
||||
}
|
||||
@@ -382,23 +365,7 @@ class GalleryDLService:
|
||||
|
||||
platform_section = config["extractor"].setdefault(platform, {})
|
||||
|
||||
if platform == "patreon":
|
||||
if "all" in source_config.content_types:
|
||||
platform_section["files"] = [
|
||||
"images", "image_large", "attachments", "postfile", "content",
|
||||
]
|
||||
else:
|
||||
platform_section["files"] = source_config.content_types
|
||||
# Cursor-paged backfill: resume the newest→oldest walk from the
|
||||
# checkpoint saved by the previous run instead of restarting from
|
||||
# the top. Without this, a large catalog (Anduo's weekly
|
||||
# image-dense Reports) never reaches its frontier inside the
|
||||
# subprocess budget — the HEAD walk alone times out, 0 files
|
||||
# written, no progress (event #40411). The PLATFORM_DEFAULTS leave
|
||||
# `cursor: True` (log-only) for the first/fresh run.
|
||||
if source_config.resume_cursor:
|
||||
platform_section["cursor"] = source_config.resume_cursor
|
||||
elif platform == "hentaifoundry":
|
||||
if platform == "hentaifoundry":
|
||||
if "pictures" in source_config.content_types or "all" in source_config.content_types:
|
||||
platform_section["include"] = "all"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user