Commit Graph
4 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 da98195ada refactor: the native downloaders share one download loop, file record and seen-ledger key; a clean walk marks its credential verified
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
Patreon, SubscribeStar and Discord each carried their own copy of the same
per-post download loop, file record and seen-ledger key, and every copy was
another place a fix could be missed.

- BaseNativeDownloader runs one post's files: the seen and on-disk skips,
  the mid-post time-box, pacing, validation and the per-file sidecar. Each
  platform overrides only where its layout differs: Discord's channel folder,
  file names and message-id sidecar, and Patreon's Mux video through yt-dlp.
- One MediaItem and one ledger_key in native_ingest_common replace the three
  per-client copies. Keys are unchanged, except that a SubscribeStar file with
  no gallery id is now keyed by its filename instead of an empty id.
- DownloadResult, ErrorType and the run-stats and tier-gated helpers move
  from gallery_dl.py, which now serves only HentaiFoundry, to
  download_result.py. The unused delegator methods on GalleryDLService go.
- The download lock's platform list is the native-ingester list, not a copy.
- A walk that used a credential and finished with no error type stamps that
  credential's last-verified time, so the credentials page shows when the
  account last worked. A tier-limited walk does not count: an expired session
  hides paid posts the same way.
- The extension no longer clears a retired pixiv token on startup.
- Two comments that still described every download as gallery-dl now name
  both backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:03:32 -05:00
bvandeusenandClaude Fable 5 0bbcdee3bd feat(pixiv): flip dispatch to the native ingester (#129 step 4)
CI / integration (push) Failing after 3m37s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the
recover/recapture UI actions now route through PixivIngester. Campaign id is
parsed straight from the source URL (numeric user id — no network resolver),
with a platform-aware resolution-failure message. auth_token now rides the
uniform adapter construction (token platforms use it, cookie platforms
accept-and-ignore), and the preview endpoint fetches/threads it. The legacy
gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the
refresh-token config branches in download/verify) per no-legacy policy;
gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-03 09:54:18 -04:00
bvandeusenandClaude Opus 4.8 ebe6ab9741 refactor(native-ingest): shared exception trio + base _failure_result (#899 DRY 2/3)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m18s
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).

- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
  (status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
  SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
  keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
  NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
  free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
  ("Patreon API" / "SubscribeStar markup"), preserving the existing message
  (test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
  -unused DownloadResult/ErrorType/*Auth/*Drift imports.

Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:40:36 -04:00
bvandeusenandClaude Opus 4.8 817a002c2b feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892)
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 38s
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 3m23s
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.

- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
  creator page → infinite_scroll-next_page href → JSON {html} fragments
  (campaign_id = creator URL; no resolver). extract_media reads the per-post
  data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
  post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
  locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
  Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
  branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
  on-disk layout so existing downloads dedup on disk at cutover. Post-first:
  _post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
  SubscribeStar ledgers into the core; ledger_key = filehash else
  post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
  layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.

Not yet wired into dispatch (Step 5) — these modules are inert until then.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:06:02 -04:00