Native Patreon ingester + download-engine ownership (plans #697, #703–#708) #72
Assignees
bvandeusen
Clear assignees
bvandeusen
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: bvandeusen/FabledCurator#72
Reference in New Issue
Block a user
Delete Branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Replaces the gallery-dl subprocess for Patreon with a native, automation-friendly download engine, then evolves it into a platform-agnostic core with a registry seam. 26 commits.
Highlights
Native Patreon ingester (plan #697) — JSON-API client + media downloader + seen-ledger; zero per-file HEADs, native cursor/resume, loud drift detection. Three modes (tick / backfill / recovery). gallery-dl Patreon path removed at cutover; the other five platforms unchanged.
Expansion roadmap #1–#9 (plans #703–#706)
DownloadResult, dropped the regex-over-stdout scrape); quarantine surfacing; live within-chunk posts progress.patreon_failed_media); per-media transient/permanent retry (GET + video).ingest_core.Ingesterengine + thinPatreonIngester(Ingester)adapter — adding a platform is now a small adapter.DRY pass (plan #707) — shared
quarantine_file(+ native quarantine sidecar parity fix),make_run_statsfactory, onesafe_ext, promoted log helpers, anddownload_backends.run_downloadas the single phase-2 dispatch sodownload_backendsis the one registry of how each platform downloads + verifies.Ingester-ownership wins (plan #708) — platform cooldown honors the server's 429 Retry-After; partial media downloads resume via HTTP Range; Stop halts a live backfill chunk; dry-run Preview (count what a backfill would download, with a Patreon-only Preview button + dialog).
Deploy notes
patreon_seen_media, 0038patreon_failed_media) — deploy must run migrations.session_idcookies): an Anduo backfill reaches bottom without timing out, Recovery re-fetches dropped near-dups, Preview returns a sensible count, and Stop halts a running chunk promptly.🤖 Generated with Claude Code
Branch download_service phase 2 by platform: Patreon now routes to the native PatreonIngester (zero per-file HEADs, native cursor/resume, loud drift detection) instead of gallery-dl; the other 5 platforms are unchanged. The ingester returns a DownloadResult-shaped object so phase 1 (DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched. Three modes wired from config_overrides state: - tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N contiguous already-have-it items. - backfill: full-history time-boxed chunk, cursor checkpoint via gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693 lifecycle + parse_last_cursor verbatim). - recovery: backfill that BYPASSES the tier-1 seen-ledger so dropped-and-deleted near-dups get re-fetched and re-evaluated under the current pHash threshold. Rides the #693 state machine via a _backfill_bypass_seen flag, cleared on completion / stop. The seen-ledger uses short-lived sync sessions (injected sessionmaker), never held across the walk (avoids the connection-reaping trap). Campaign id resolves from override, an id: URL, or a vanity lookup; unresolvable = loud NOT_FOUND, never a silent empty success. Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency, budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented download_service tests now drive the ingester branch via a stub; the gallery-dl campaign-retry test is replaced by resolution/caching coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Typed, loud failure mapping for the native Patreon ingester so a changed API shape or expired auth never silently zero-downloads as "success". - New ErrorType.API_DRIFT (free varchar error_type col → no migration): distinct from auth so the operator knows the fix is updating the ingester, not rotating cookies. - patreon_client: PatreonAPIError carries status_code; new PatreonAuthError for 401/403 + HTML-login/non-JSON bodies (reclassified from drift — expired-session is auth, actionable as "rotate cookies"). - patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR, PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR, transport→NETWORK_ERROR. (429 thus drives the platform cooldown.) - FailingSourcesCard: api_drift chip (red) + hint. Contract test (new test_patreon_contract.py): the recorded /api/posts fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the request params must still carry every field the parser depends on (file_name, image_urls/download_url, images/attachments_media/media includes, content/post_file/image post fields) — a trim of either trips a red build. Plus client HTTP-status classification tests and ingester error-type mapping tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide cooldown → every Patreon source dark ("testing dead"). The native path also ignored the operator's existing politeness setting. Fixed both: - Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's `rate_limit`, read off self.gdl) as a pre-download sleep on real media downloads only (skips don't pace); pace /api/posts page fetches with the per-source sleep_request override, defaulting to max(0.5, rate_limit/4) — the same API-pacing default gallery-dl used for `sleep-request`. - 429 backoff (ride out transient limits): PatreonClient._fetch retries a 429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped 30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal RATE_LIMITED. Light 2-retry on a media-GET 429 too. Threaded via PatreonIngester(rate_limit=, request_sleep=) → PatreonClient/PatreonDownloader; download_service sources them. Injected test client/downloader are unaffected (carry their own pacing). Tests mock time.sleep (no real sleeping): retry-then-success, persistent 429 raises after N, Retry-After honored, request_sleep paces, media pacing per real download, skips don't pace, media 429 retried. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Before arming a deep walk on a native-ingester platform (Patreon — where verify is one cheap API page), POST /sources/{id}/backfill {start|recover} runs the shared verify_source_credential first and REFUSES (409 + reason) only on a definitive rejection (verify→False, e.g. expired cookies). It proceeds on valid (True) or inconclusive (None — a network blip must not block). Gated to native platforms: gallery-dl verify is a slow --simulate subprocess, too heavy for an arm action. The credential read happens in a session that's CLOSED before the verify network call (no held conn). Frontend: onBackfill/onRecover now read e.body.detail (ApiError carries the reason in .body, not .detail) so the rejection text surfaces in the toast. Tests: arm blocked on rejection (409, source not armed), proceeds on inconclusive, stop never pre-flights, gallery-dl platform skips pre-flight. An autouse fixture stubs verify to 'valid' for the existing backfill endpoint tests so they stay network-free. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>A worker SIGKILL (hard-time-limit backstop) mid-chunk lost the whole chunk's walk — the cursor was only persisted at chunk boundaries by phase 3, so the next tick re-walked from the chunk start. Now the ingester checkpoints _backfill_cursor to the DB at each page boundary (backfill/ recovery only) via an ATOMIC single-key UPDATE (config_overrides::jsonb → jsonb_set('{_backfill_cursor}') → ::json), so it never clobbers operator config or other backfill keys. On a crash the last mid-walk cursor survives → the next chunk resumes near the crash, not the chunk start. phase 3 still writes the final cursor (same value); this is the safety net. Tests: a backfill walk leaves the last page's cursor in the DB (written by the ingester, before any phase 3); a tick never checkpoints. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Owning the walk lets an operator gauge "is this source worth a backfill?" before arming one. ingest_core.Ingester.preview walks the first few feed pages and counts media NOT already in the seen/dead ledgers, downloading nothing (read-only). download_backends.preview_source resolves the campaign id + runs it (native-only, mirrors verify_source_credential / run_download); POST /api/sources/{id}/preview returns {total_new, posts_scanned, has_more, sample[]} (409 on auth/drift/unresolvable, 400 for gallery-dl platforms). PatreonClient gains post_meta(post) for the sample's title/date. UI: a Patreon-only Preview button (mdi-eye-outline) on SourceRow + SourceCard opens PreviewDialog — self-fetches with loading / error / empty / result states and a "Start backfill" shortcut. Store action previewSource. Tests: preview counts new media without downloading + samples only posts with new items; page_limit caps the walk + flags has_more. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>