SubscribeStar → native core ingester + native-ingest DRY pass (milestone #71) #117

Merged

bvandeusen merged 7 commits from dev into main 2026-06-17 12:48:29 -04:00

Conversation 0 Commits 7 Files Changed 24

+1759 -455

bvandeusen commented 2026-06-17 12:48:20 -04:00

Owner

Copy Link

Copy Source

Migrates SubscribeStar off gallery-dl onto the native core ingester, plus a DRY/observability pass over the native-ingest subsystem. All CI-green on dev (head d526447).

SubscribeStar native (milestone #71, steps 0–6)

• HTML-scrape client (no JSON:API): page-based feed walk, data-gallery media manifest, post-first capture, best-effort gating. Parser validated against a real sample.
• Ledger tables + migration 0054; downloader (gallery-dl on-disk layout for cutover dedup); ingester adapter.
• Dispatch flip: subscribestar now in NATIVE_INGESTER_PLATFORMS + SERIALIZED_PLATFORMS; campaign_id = creator URL (no resolver); gallery-dl path removed (rule 22). Gets backfill/recovery/recapture/preview/live-progress for free.

DRY pass (#899, process #594)

• New native_ingest_common.py: make_session, retry helper, sanitize/basename/post_dir_name, MediaOutcome/PostRecordOutcome, BaseNativeDownloader (shared streaming GET + validation), shared exception trio + base _failure_result. Patreon + SubscribeStar both consume it — kills the divergence-bug risk and makes HentaiFoundry/Discord cheap.
• Logging (L1/L2/L3): run start / per-page breadcrumb / final summary / stop now go through the real logger with source_id, so a worker SIGKILL/OOM/hard-limit no longer erases the run trace; quarantines logged.

Deploy

Re-pull download-worker + web images (:latest tracks main). Then test a SubscribeStar source (Preview → tick → backfill resume); existing gallery-dl files should NOT re-download (content/on-disk dedup). post_is_gated is best-effort.

🤖 Generated with Claude Code

Migrates SubscribeStar off gallery-dl onto the native core ingester, plus a DRY/observability pass over the native-ingest subsystem. All CI-green on dev (head d526447). ## SubscribeStar native (milestone #71, steps 0–6) - HTML-scrape client (no JSON:API): page-based feed walk, data-gallery media manifest, post-first capture, best-effort gating. Parser validated against a real sample. - Ledger tables + migration 0054; downloader (gallery-dl on-disk layout for cutover dedup); ingester adapter. - **Dispatch flip**: subscribestar now in NATIVE_INGESTER_PLATFORMS + SERIALIZED_PLATFORMS; campaign_id = creator URL (no resolver); gallery-dl path removed (rule 22). Gets backfill/recovery/recapture/preview/live-progress for free. ## DRY pass (#899, process #594) - New native_ingest_common.py: make_session, retry helper, sanitize/basename/post_dir_name, MediaOutcome/PostRecordOutcome, BaseNativeDownloader (shared streaming GET + validation), shared exception trio + base _failure_result. Patreon + SubscribeStar both consume it — kills the divergence-bug risk and makes HentaiFoundry/Discord cheap. - **Logging (L1/L2/L3)**: run start / per-page breadcrumb / final summary / stop now go through the real logger with source_id, so a worker SIGKILL/OOM/hard-limit no longer erases the run trace; quarantines logged. ## Deploy Re-pull download-worker + web images (:latest tracks main). Then test a SubscribeStar source (Preview → tick → backfill resume); existing gallery-dl files should NOT re-download (content/on-disk dedup). post_is_gated is best-effort. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 7 commits 2026-06-17 12:48:20 -04:00

feat(subscribestar): seen/failed ledger models + migration 0054 (#889) ... f678819093

Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core
ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger:
SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery
bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing
media stops re-burning backfill chunks). Per operator decision, dedicated
per-platform tables (not a generalized shared ledger).

filehash is String(128): a CDN content hash when the URL carries one, else a
synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key.
Registered in models/__init__; migration 0054 creates both tables (down 0053).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892) ... 817a002c2b

Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.

- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
  creator page → infinite_scroll-next_page href → JSON {html} fragments
  (campaign_id = creator URL; no resolver). extract_media reads the per-post
  data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
  post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
  locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
  Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
  branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
  on-disk layout so existing downloads dedup on disk at cutover. Post-first:
  _post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
  SubscribeStar ledgers into the core; ledger_key = filehash else
  post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
  layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.

Not yet wired into dispatch (Step 5) — these modules are inert until then.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(native-ingest): extract native_ingest_common + BaseNativeDownloader (#899 DRY 1/3) ... 7ac5c7e522

DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing
the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was
importing patreon privates — wrong owner). New backend/app/services/
native_ingest_common.py is the neutral home for:
- make_session (was _load_session ×2), retry_after_seconds + 429 constants,
  sanitize_segment, basename_from_url, post_dir_name, MediaOutcome /
  PostRecordOutcome.
- BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume)
  and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it;
  each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch
  enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the
  divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone.
- Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only).

post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO).
Tests repointed to the single source at every consumer (rule 93 / §8b parity):
patreon_client/downloader, subscribestar_native. Exception-trio consolidation +
base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(native-ingest): shared exception trio + base _failure_result (#899 DRY 2/3) ... ebe6ab9741

DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).

- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
  (status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
  SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
  keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
  NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
  free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
  ("Patreon API" / "SubscribeStar markup"), preserving the existing message
  (test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
  -unused DownloadResult/ErrorType/*Auth/*Drift imports.

Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(native-ingest): durable run logging that survives a worker kill (#899 L1/L3, DRY 3/3) ... 82551a89d1

DRY pass commit 3 — the observability half. ingest_core accumulated ALL
human-readable progress in log_lines → DownloadResult.stdout, persisted to the
DownloadEvent ONLY at phase 3; the real logger was used almost nowhere. So a
worker SIGKILL/OOM/hard-time-limit mid-walk left NO trace (the "task died,
no trace" mode from the recovery-sweep work).

Route run milestones through the container log too, each carrying source_id (L3
context):
- run START (platform/mode/source/campaign/resume_cursor)
- per-PAGE breadcrumb (posts/downloaded/skipped/errors/quarantined/gated/cursor)
  at each page boundary — pages are minutes apart on big backfills, so this shows
  how far a since-died walk got
- final SUMMARY (same string as the stdout summary)
- operator STOP

(L2 — quarantines log.warning'd — already landed in commit 1's base
_validate_path; failures log.warning via the base _failure_result; the #862 body
canary already log.error's.) log_lines/stdout content is unchanged (summary just
captured in a var), so existing assertions hold.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(subscribestar): flip dispatch to the native ingester (#893, Step 5) ... d8d8ecd78f

SubscribeStar now downloads + verifies through the native core ingester instead
of gallery-dl — the go-live switch for milestone #71.

- download_backends: subscribestar added to NATIVE_INGESTER_PLATFORMS; a
  _NATIVE_INGESTERS registry + _resolve_native_campaign_id make _run_native_ingester
  / preview_source / verify_source_credential platform-aware. SubscribeStar's
  campaign_id IS the creator URL (no resolver); Patreon still resolves the vanity.
  preview now catches the shared NativeIngestError (covers both platforms).
- platform_lock: subscribestar serialized (one paced walk at a time).
- gallery_dl: subscribestar entry removed from PLATFORM_DEFAULTS (rule 22 — no
  fallback once native works).
- frontend SourceActions: isPatreon → isNative (patreon|subscribestar) so the
  recover/recapture actions show for subscribestar; download_service's
  cursor/mode/post_first + the preview endpoint already key on
  uses_native_ingester, so backfill/recovery/recapture/preview light up for free.
- tests: download_backends (subscribestar native), platform_lock (serialized),
  and three gallery-dl-sample tests repointed to hentaifoundry (api_credentials
  verify, gallery_dl_service skip-value, api_sources arm-no-preflight).

post_is_gated stays best-effort (can't cause junk downloads); not gating this.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(dispatch): resolve native ingester class at call time (test monkeypatch) ... d526447496

The _NATIVE_INGESTERS dict captured PatreonIngester/SubscribeStarIngester at
import, so test_download_service's monkeypatch.setattr(db_mod, "PatreonIngester",
_FakeIngester) no longer affected dispatch → the fake's run() never ran →
KeyError 'campaign_id' on empty run_kwargs (integration run 1215). Replace the
dict with a _native_ingester_cls() call-time lookup that reads the module globals,
so monkeypatching the class names works again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit 9b1b0369cc into main 2026-06-17 12:48:29 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#117