feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
CI / frontend-build (push) Successful in 23s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 10s
CI / integration (push) Successful in 2m59s

Final step of the native Patreon ingester: a first-class Recovery action,
and removal of the now-dead gallery-dl Patreon path.

Recovery (rules #23/#24/#27 — full product, with UI):
- source_service.start_recovery arms the #693 backfill state machine PLUS
  `_backfill_bypass_seen`, flipping download mode to recovery (bypass the
  seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate
  under the current pHash threshold). Stop via the shared stop_backfill.
- SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill
  gains action="recover".
- Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow
  + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling
  (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource +
  SubscriptionsTab onRecover.

Cutover (rule #22 — no legacy):
- gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon
  files/cursor branch in _build_config_for_source, and the patreon/Mux
  yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the
  other platforms' yt-dlp fetches; native ingester owns it now).
- download_service: removed the dead campaign-id-retry helpers
  (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and
  _effective_url. Vanity→campaign resolution + resume_cursor + the
  cursor/PARTIAL lifecycle stay — they serve the native ingester.

Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp
Referer, resume-cursor); repointed the generic skip-value config tests to
subscribestar; added start_recovery + recover-endpoint coverage
(backfill_bypass_seen). gallery-dl stays for the other 5 platforms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 22:18:03 -04:00
parent 682beafbc5
commit ec43e823e1
12 changed files with 211 additions and 126 deletions
+33
View File
@@ -67,6 +67,9 @@ class SourceRecord:
# plan #693: derived from config_overrides for the UI badge.
backfill_state: str | None # "running" | "complete" | "stalled" | None (idle)
backfill_chunks: int
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
backfill_bypass_seen: bool
def to_dict(self) -> dict:
return {
@@ -87,6 +90,7 @@ class SourceRecord:
"backfill_runs_remaining": self.backfill_runs_remaining,
"backfill_state": self.backfill_state,
"backfill_chunks": self.backfill_chunks,
"backfill_bypass_seen": self.backfill_bypass_seen,
}
@@ -162,6 +166,7 @@ class SourceService:
backfill_runs_remaining=source.backfill_runs_remaining or 0,
backfill_state=co.get("_backfill_state"),
backfill_chunks=int(co.get("_backfill_chunks", 0)),
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
)
async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -319,6 +324,34 @@ class SourceService:
await self.session.commit()
return await self._row_to_record(source)
async def start_recovery(self, source_id: int) -> SourceRecord:
"""Plan #697: arm a RECOVERY walk — a backfill that bypasses the Patreon
seen-ledger so deliberately-dropped-and-deleted near-dups get re-fetched
and re-evaluated under the CURRENT pHash threshold (tier-2 disk still
spares files we kept). Reuses the entire #693 backfill state machine
(time-boxed chunks, cursor checkpoint, complete/stall lifecycle) plus the
`_backfill_bypass_seen` flag that flips download mode to recovery. Clears
any prior cursor/chunk/stall state so it walks fresh from the top. The
flag is cleared on completion (download_service) and on stop.
Recovery is Patreon-only (the seen-ledger is Patreon's); for other
platforms the flag is inert (download_service ignores it) and the walk
runs as a plain backfill. The UI gates the action to Patreon sources."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
co["_backfill_state"] = "running"
co["_backfill_bypass_seen"] = True
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def stop_backfill(self, source_id: int) -> SourceRecord:
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
Clears the running state + cursor/chunk/stall bookkeeping."""