feat(patreon): phase-2 ingester integration — build step 3 (plan #697)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 31s
CI / integration (push) Successful in 2m59s

Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.

Three modes wired from config_overrides state:
  - tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
    contiguous already-have-it items.
  - backfill: full-history time-boxed chunk, cursor checkpoint via
    gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
    lifecycle + parse_last_cursor verbatim).
  - recovery: backfill that BYPASSES the tier-1 seen-ledger so
    dropped-and-deleted near-dups get re-fetched and re-evaluated under
    the current pHash threshold. Rides the #693 state machine via a
    _backfill_bypass_seen flag, cleared on completion / stop.

The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.

Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 21:38:42 -04:00
parent 2ec7d86a3b
commit 96c30eba13
6 changed files with 984 additions and 109 deletions
+5
View File
@@ -143,6 +143,11 @@ def download_source(self, source_id: int) -> int:
gdl=gdl,
importer=importer,
cred_service=cred_service,
# The native Patreon ingester opens its own short-lived
# sync sessions for the seen-ledger (never held across
# the walk). Same factory the importer's sync session
# comes from — a different DB connection per checkout.
sync_session_factory=SyncFactory,
)
return await svc.download_source(source_id)
finally: