6e5db94b800cdd2771cc6647e8c764c7889fb06c
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
da98195ada |
refactor: the native downloaders share one download loop, file record and seen-ledger key; a clean walk marks its credential verified
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
Patreon, SubscribeStar and Discord each carried their own copy of the same per-post download loop, file record and seen-ledger key, and every copy was another place a fix could be missed. - BaseNativeDownloader runs one post's files: the seen and on-disk skips, the mid-post time-box, pacing, validation and the per-file sidecar. Each platform overrides only where its layout differs: Discord's channel folder, file names and message-id sidecar, and Patreon's Mux video through yt-dlp. - One MediaItem and one ledger_key in native_ingest_common replace the three per-client copies. Keys are unchanged, except that a SubscribeStar file with no gallery id is now keyed by its filename instead of an empty id. - DownloadResult, ErrorType and the run-stats and tier-gated helpers move from gallery_dl.py, which now serves only HentaiFoundry, to download_result.py. The unused delegator methods on GalleryDLService go. - The download lock's platform list is the native-ingester list, not a copy. - A walk that used a credential and finished with no error type stamps that credential's last-verified time, so the credentials page shows when the account last worked. A tier-limited walk does not count: an expired session hides paid posts the same way. - The extension no longer clears a retired pixiv token on startup. - Two comments that still described every download as gallery-dl now name both backends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bbcdee3bd |
feat(pixiv): flip dispatch to the native ingester (#129 step 4)
pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the recover/recapture UI actions now route through PixivIngester. Campaign id is parsed straight from the source URL (numeric user id — no network resolver), with a platform-aware resolution-failure message. auth_token now rides the uniform adapter construction (token platforms use it, cookie platforms accept-and-ignore), and the preview endpoint fetches/threads it. The legacy gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the refresh-token config branches in download/verify) per no-legacy policy; gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
|
ebe6ab9741 |
refactor(native-ingest): shared exception trio + base _failure_result (#899 DRY 2/3)
DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404
→http→network mapping in _failure_result; only the exception classes + drift
phrasing differed (divergence-bug risk: a new error_type handled in one and not
the other).
- native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError
(status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and
SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance,
keeping their isinstance-distinct platform names.
- Ingester._failure_result (base) does the whole mapping via the shared
NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it
free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing
("Patreon API" / "SubscribeStar markup"), preserving the existing message
(test asserts "Patreon API changed").
- Both adapters drop their near-identical _failure_result overrides and their now
-unused DownloadResult/ErrorType/*Auth/*Drift imports.
Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/
404/network) and test_subscribestar_native (_failure_result mapping) both exercise
the base method now. Remaining: ingest_core L1/L3 logging (3/3).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
817a002c2b |
feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892)
Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core
ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the
platform-agnostic core (ingest_core) is unchanged.
- subscribestar_client.py: HTML-scrape read path. iter_posts pages via the
creator page → infinite_scroll-next_page href → JSON {html} fragments
(campaign_id = creator URL; no resolver). extract_media reads the per-post
data-gallery JSON manifest (id/original_filename/type/url). post_record_key,
post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live
locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error).
Parser validated against the real Step-0 fixtures.
- subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp
branch (SubscribeStar serves files directly via /post_uploads). gallery-dl
on-disk layout so existing downloads dedup on disk at cutover. Post-first:
_post.json owns the body/links; per-media sidecar carries image identity only.
- subscribestar_ingester.py: thin adapter wiring client/downloader/the
SubscribeStar ledgers into the core; ledger_key = filehash else
post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential.
- tests: client parsing/pagination/media/gating/record-key/dates, downloader
layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping.
Not yet wired into dispatch (Step 5) — these modules are inert until then.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|