feat(patreon): phase-2 ingester integration — build step 3 (plan #697)
Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.
Three modes wired from config_overrides state:
- tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
contiguous already-have-it items.
- backfill: full-history time-boxed chunk, cursor checkpoint via
gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
lifecycle + parse_last_cursor verbatim).
- recovery: backfill that BYPASSES the tier-1 seen-ledger so
dropped-and-deleted near-dups get re-fetched and re-evaluated under
the current pHash threshold. Rides the #693 state machine via a
_backfill_bypass_seen flag, cleared on completion / stop.
The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.
Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -329,7 +329,7 @@ class SourceService:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
co = dict(source.config_overrides or {})
|
||||
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
|
||||
"_backfill_chunks"):
|
||||
"_backfill_chunks", "_backfill_bypass_seen"):
|
||||
co.pop(k, None)
|
||||
source.config_overrides = co
|
||||
source.backfill_runs_remaining = 0
|
||||
|
||||
Reference in New Issue
Block a user