Pixiv native ingester (#129) + SubscribeStar .art fix, disable-clears-failure, system-tag suggestion floor #191

Merged

bvandeusen merged 9 commits from dev into main 2026-07-03 16:36:56 -04:00

Conversation 0 Commits 9 Files Changed 27

+2434 -50

bvandeusen commented 2026-07-03 16:36:49 -04:00

Owner

Copy Link

Copy Source

Merges dev → main. All CI-green on dev head 6c6e8bd (run 1919: lint, backend-unit, frontend, integration).

Migrations on this merge: 0075 (system-tag seeds wip/banner/editor — carried from the previous still-undeployed main state) + 0076 (pixiv seen/failed ledgers).

Pixiv → native post-first ingester (milestone #129)

Third native platform alongside patreon/subscribestar. gallery-dl 1.32.5-parity app-API client (iOS headers, OAuth refresh, /v1/user/illusts, next_url cursor), layout-parity downloader (byte-exact gallery-dl filename via gdl_clean_filename → tier-2 disk-skip recognizes pre-cutover files, no re-download), enriched _post_<id>.json (JP+EN tags, rating, series, view/bookmark/comment counts, AI flag, dims, ugoira frame delays → Post.raw_metadata + title/body), pixiv ledgers (0076), dispatch flipped, legacy gallery-dl pixiv path removed.

Pre-merge fixes

• SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on .art; _normalize_ss_host routes .art → .adult at request time (walk + verify), logged.
• Disable clears source failure: SourceService.update wipes last_error/error_type/consecutive_failures on explicit disable, so paused subs stop showing as failing.
• System-tag suggestion floor: wip/banner/editor heads surface at a flat 0.65 confidence floor (not their high auto threshold) so false positives appear for reject-to-train; content tags keep their own thresholds.

Post-deploy operator action: label ~8+ each of wip/banner/editor to arm the system-tag heads (training hygiene + the 0.65 reject-to-train surfacing both need ≥8 labels).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

Merges dev → main. All CI-green on dev head `6c6e8bd` (run 1919: lint, backend-unit, frontend, integration). **Migrations on this merge:** 0075 (system-tag seeds wip/banner/editor — carried from the previous still-undeployed main state) + **0076 (pixiv seen/failed ledgers)**. ### Pixiv → native post-first ingester (milestone #129) Third native platform alongside patreon/subscribestar. gallery-dl 1.32.5-parity app-API client (iOS headers, OAuth refresh, /v1/user/illusts, next_url cursor), layout-parity downloader (byte-exact gallery-dl filename via gdl_clean_filename → tier-2 disk-skip recognizes pre-cutover files, no re-download), enriched `_post_<id>.json` (JP+EN tags, rating, series, view/bookmark/comment counts, AI flag, dims, ugoira frame delays → Post.raw_metadata + title/body), pixiv ledgers (0076), dispatch flipped, legacy gallery-dl pixiv path removed. ### Pre-merge fixes - **SubscribeStar .art age wall**: the 18+ cookie doesn't clear the age gate on `.art`; `_normalize_ss_host` routes `.art` → `.adult` at request time (walk + verify), logged. - **Disable clears source failure**: `SourceService.update` wipes last_error/error_type/consecutive_failures on explicit disable, so paused subs stop showing as failing. - **System-tag suggestion floor**: wip/banner/editor heads surface at a flat 0.65 confidence floor (not their high auto threshold) so false positives appear for reject-to-train; content tags keep their own thresholds. Post-deploy operator action: label ~8+ each of wip/banner/editor to arm the system-tag heads (training hygiene + the 0.65 reject-to-train surfacing both need ≥8 labels). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

bvandeusen added 9 commits 2026-07-03 16:36:50 -04:00

feat(pixiv): native app-API client — gallery-dl-parity profile, post-first seams (#129 step 1) ... 86ae396914

PixivClient mirrors gallery-dl 1.32.5's PixivAppAPI request profile exactly
(iOS app headers, OAuth refresh with X-Client-Time/X-Client-Hash,
/v1/user/illusts pagination via next_url — whose query string doubles as the
resumable page cursor). Post-first seams (post_record_key / post_is_gated /
post_meta) + extract_media covering multi-page, single-page, ugoira zip
(600x600→1920x1080 swap, frame delays memoized for the post record), and
limit_* placeholder gating. No PHPSESSID web fallback: FC holds only the
refresh token, same effective coverage as the gallery-dl path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(pixiv): native downloader — gallery-dl layout parity + enriched post record (#129 step 2) ... 7ef2ecd82f

PixivDownloader writes originals to the exact pre-cutover gallery-dl layout
(<artist_slug>/pixiv/pixiv/{id}_{title[:50]}_{NN}.{ext} — flat, double
platform segment) so tier-2 disk-skip recognizes existing files. Post-first:
per-media sidecar is identity-only; the post record (_post_<id>.json — id
suffix because the flat layout would collide a bare _post.json) carries the
enrichment: tags + EN translations, rating from x_restrict, series,
view/bookmark/comment counts, AI flag, dimensions, author, and ugoira frame
delays (the zip has no timings). i.pximg.net media GETs ride the app-header
profile (403 without the app-api Referer).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(pixiv): ledger models + migration 0076 + PixivIngester adapter (#129 step 3) ... 0563b2d750

pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar
ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira
— pximg URLs carry no content hash). PixivIngester wires client/downloader/
ledgers into ingest_core with drift label 'Pixiv app API' and the new
body_canary=False opt-out: caption-less pixiv artists are common, so the
zero-bodies #862 alarm would false-positive here — the client's
response-shape drift checks cover that failure class instead. auth_token
joins the uniform adapter constructor (pixiv is the first token-auth native
platform). verify_pixiv_credential = one OAuth refresh, no feed walk.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

test(pixiv): fix downloader tests — skip validation on stub bytes, no media extraction in post-record tests ... 4068a97392

The stub payload is PNG bytes regardless of target extension, so the real
validator quarantined the .jpg cases; and extracting the ugoira work hit the
API seam of a fake session with no .post. Validation/quarantine plumbing
stays covered by the Patreon downloader tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(pixiv): flip dispatch to the native ingester (#129 step 4) ... 0bbcdee3bd

pixiv joins NATIVE_INGESTER_PLATFORMS: download/verify/preview and the
recover/recapture UI actions now route through PixivIngester. Campaign id is
parsed straight from the source URL (numeric user id — no network resolver),
with a platform-aware resolution-failure message. auth_token now rides the
uniform adapter construction (token platforms use it, cookie platforms
accept-and-ignore), and the preview endpoint fetches/threads it. The legacy
gallery-dl pixiv path is fully removed (PLATFORM_DEFAULTS entry + the
refresh-token config branches in download/verify) per no-legacy policy;
gallery-dl keeps hentaifoundry/discord/deviantart until they migrate/retire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

test: stub ctx gains the auth_token key run_download now reads ... a7f715ec43

The real phase-1 ctx has always carried auth_token; the native branch now
threads it into the adapter constructors, so the stub ctx must match the
contract (kept the strict ctx[...] read — it catches exactly this drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

fix(pixiv): match gallery-dl's exact on-disk filename to avoid re-download at cutover ... 544e30bfb9

The native downloader used the Windows-safe sanitize_segment, but gallery-dl on
Linux (path-restrict auto→'/', path-remove default control chars, path-strip
auto→'') replaces ONLY '/' and deletes control chars — the Windows-forbidden set
(<>:"|?*) and trailing dots/spaces stay RAW in on-disk titles. Any pixiv title
with those chars would therefore miss the tier-2 disk-skip and re-download the
whole work at cutover (seen-ledger starts empty). Replace sanitize_segment with
gdl_clean_filename, a byte-exact mirror of gallery-dl 1.32.5 build_filename
(verified against path.py). Directory + template already matched; this closes the
last parity gap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

fix(subscribestar): route .art creators to .adult; clear source failure on disable ... c9696a2faf

Two pre-merge fixes:

1. SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on
   the .art domain (keeps 302'ing to /age_confirmation_warning even with the
   cookie — Elasid #54116), but the same creator is reachable on .adult where
   the cookie works. _normalize_ss_host rewrites subscribestar.art →
   subscribestar.adult at request time (stored Source.url untouched), logged so
   it's visible in walk logs. .com/.adult pass through.

2. Disabling a source now clears its failure state (last_error, error_type,
   consecutive_failures) so subs you pause (not paying for) stop lingering as
   'failing'. Only the explicit disable clears — an unrelated edit to an
   already-disabled source leaves state alone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(heads): surface system-tag suggestions at a flat 0.65 confidence floor ... 6c6e8bdb6d

System tags (wip/banner/editor) already get heads (kind=general) and aren't
filtered from suggestions, but they surfaced only at each head's precision-tuned
suggest_threshold — high enough to hide the borderline/false-positive guesses the
operator wants to SEE and REJECT (hard-negative mining: 'negatively reinforce
what isn't a system tag'). score_image now uses a flat _SYSTEM_TAG_SUGGEST_FLOOR
(0.65, operator-set) for system-tag heads instead of their auto threshold;
content-tag heads keep their own, and the typed-dropdown threshold_override still
overrides everything. _current_heads carries Tag.is_system into the head meta to
drive it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

bvandeusen merged commit d2acec61ae into main 2026-07-03 16:36:56 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#191