feat: retire pixiv entirely — delete its code, its ledgers, its credential (3977, 3978, 3979)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 34s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m54s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m19s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 34s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m54s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m19s
Milestone #406 phase 2, with issue #3980 folded in. Phase 1 (2026-09-13) unregistered pixiv so nothing could reach it; the code has sat in the tree uncalled since. DeviantArt is why the second half is not left for later — #3069 retired it in code on 2026-08-27 and its stored session was still in the database seven weeks on. Step 5 — the code. Deletes pixiv_client, pixiv_downloader, pixiv_ingester, platforms/pixiv and their three test modules and fixture, then edits out every remaining reference: the dispatch entry, the campaign-id and verify branches in download_backends, the display-name branch in extension_service, and the comments that still described pixiv as live. The consolidation check the step asked for comes back negative: native_ingest_common has seven non-pixiv callers (patreon, subscribestar, membership_reconcile, membership_roster, ingest_core), so nothing there drops to a single user. Step 6 — the data, alembic 0102. Drops pixiv_seen_media and pixiv_failed_media, and deletes credential rows whose platform is not registered. Written as "not registered" rather than "pixiv" at the step's explicit ask, which is what makes one migration cover two retirements: the pixiv OAuth refresh token and DeviantArt's leftover session (#3980). It is also the only way either row can go — the credentials UI renders one card per platform from /api/platforms and looks the credential up by key, so an unregistered platform's row has no card and no Remove button. Pixiv's Source rows are KEPT, changing the milestone's original data table on the operator's call. `platform` is stored only on Source; neither Post nor ImageRecord carries it. Both FKs are ON DELETE SET NULL, so a delete would not lose the art — but it would drop every pixiv image into the gallery's __unsourced__ bucket and strip the platform chip off every pixiv post. The rows stay disabled (0097) and unregistered, so nothing schedules or downloads through them. Keeping them costs nothing and keeps the attribution that "the art already downloaded from pixiv stays" is about. Step 7 — the guard. test_pixiv_code_and_tables_are_gone asserts absence from the module table and from Base.metadata, not from prose (snippet #3352's trap). The extension and registry negative assertions were already in place from phase 1. The final sweep found one real residue step 4 missed: extension/README.md still advertised pixiv support and carried a "Pixiv OAuth" manual-test item. Also replaces the two deleted dispatch tests with one over the whole NATIVE_INGESTER_PLATFORMS set, so adding a platform and forgetting its ingester class now fails at unit level rather than as a mid-download KeyError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
Claude Opus 5
@@ -109,8 +109,8 @@ async def quick_add_source():
|
||||
if not await _ext_key_required(session):
|
||||
return _bad("unauthorized", status=401)
|
||||
try:
|
||||
# crypto lets a pixiv add resolve the artist's display name via the
|
||||
# stored OAuth token (else it falls back to the numeric id). #130.
|
||||
# crypto lets an add resolve the artist's display name via the
|
||||
# stored credential (else it falls back to the URL handle). #130.
|
||||
result = await ExtensionService(session, _get_crypto()).quick_add_source(url)
|
||||
except UnknownPlatformError as exc:
|
||||
return _bad(
|
||||
|
||||
@@ -26,8 +26,6 @@ from .membership_sync import MembershipSync
|
||||
from .ml_settings import MLSettings
|
||||
from .patreon_failed_media import PatreonFailedMedia
|
||||
from .patreon_seen_media import PatreonSeenMedia
|
||||
from .pixiv_failed_media import PixivFailedMedia
|
||||
from .pixiv_seen_media import PixivSeenMedia
|
||||
from .platform_membership import PlatformMembership
|
||||
from .post import Post
|
||||
from .post_association import PostAssociation
|
||||
@@ -58,8 +56,6 @@ __all__ = [
|
||||
"Credential",
|
||||
"PatreonFailedMedia",
|
||||
"PatreonSeenMedia",
|
||||
"PixivFailedMedia",
|
||||
"PixivSeenMedia",
|
||||
"SubscribeStarFailedMedia",
|
||||
"SubscribeStarSeenMedia",
|
||||
"Post",
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
"""PixivFailedMedia — per-source dead-letter ledger of Pixiv media that keeps
|
||||
failing to download/validate.
|
||||
|
||||
Mirror of PatreonFailedMedia/SubscribeStarFailedMedia. Media that fails every
|
||||
walk (404'd pximg URL, deleted work, persistently-corrupt bytes) would
|
||||
otherwise re-error forever and re-burn backfill chunks. After ``attempts``
|
||||
reaches the dead-letter threshold the ingester skips it on routine
|
||||
tick/backfill walks (recovery still re-attempts). A later clean download
|
||||
clears the row.
|
||||
|
||||
`filehash` is the same synthesized ``<illust_id>:p<num>`` /
|
||||
``<illust_id>:ugoira`` key the seen-ledger uses. UNIQUE (source_id, filehash)
|
||||
is the upsert key.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from sqlalchemy.types import DateTime
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class PixivFailedMedia(Base):
|
||||
__tablename__ = "pixiv_failed_media"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"source_id", "filehash", name="uq_pixiv_failed_media_source_id"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
|
||||
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
first_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
last_failed_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
@@ -1,42 +0,0 @@
|
||||
"""PixivSeenMedia — per-source ledger of Pixiv media already
|
||||
downloaded+processed.
|
||||
|
||||
Mirror of PatreonSeenMedia/SubscribeStarSeenMedia for the Pixiv native
|
||||
ingester (replacing gallery-dl). One queryable row per (source, media) so
|
||||
routine walks skip media we've already ingested; recovery mode bypasses the
|
||||
ledger to re-walk.
|
||||
|
||||
Pixiv original URLs carry no content hash, so `filehash` is always the
|
||||
synthesized ``<illust_id>:p<num>`` (page) / ``<illust_id>:ugoira`` (frame
|
||||
zip) key — stable across any URL-shape drift. String(128) matches the sibling
|
||||
ledgers.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from sqlalchemy.types import DateTime
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class PixivSeenMedia(Base):
|
||||
__tablename__ = "pixiv_seen_media"
|
||||
__table_args__ = (
|
||||
# Dedup key the downloader upserts against: one ledger row per
|
||||
# (source, media). A second sighting of the same media is a no-op.
|
||||
UniqueConstraint(
|
||||
"source_id", "filehash", name="uq_pixiv_seen_media_source_id"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
seen_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
@@ -26,14 +26,12 @@ from pathlib import Path
|
||||
from .gallery_dl import DownloadResult, ErrorType
|
||||
from .patreon_ingester import PatreonIngester
|
||||
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
|
||||
from .pixiv_client import user_id_from_url
|
||||
from .pixiv_ingester import PixivIngester
|
||||
from .platforms import known_platform_keys
|
||||
from .subscribestar_ingester import SubscribeStarIngester
|
||||
|
||||
# Platforms whose download + verify go through the native ingester rather than
|
||||
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
|
||||
# they migrate too. pixiv left this set when it was retired (milestone #406).
|
||||
# they migrate too.
|
||||
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
|
||||
|
||||
|
||||
@@ -67,7 +65,6 @@ def _native_ingester_cls(platform: str):
|
||||
dispatch pick up the replacement."""
|
||||
return {
|
||||
"patreon": PatreonIngester,
|
||||
"pixiv": PixivIngester,
|
||||
"subscribestar": SubscribeStarIngester,
|
||||
}[platform]
|
||||
|
||||
@@ -127,25 +124,17 @@ async def _resolve_native_campaign_id(
|
||||
platform: str, url: str, cookies_path: str | None, overrides: dict,
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
|
||||
feed id IS the creator URL; Pixiv's is the numeric user id parsed straight
|
||||
from it (no lookup → resolved None either way). Patreon resolves the
|
||||
feed id IS the creator URL (no lookup → resolved None). Patreon resolves the
|
||||
campaign id from the vanity URL (resolved non-None when a lookup actually ran,
|
||||
so phase 3 caches it)."""
|
||||
if platform == "subscribestar":
|
||||
return url, None
|
||||
if platform == "pixiv":
|
||||
return user_id_from_url(url), None
|
||||
return await resolve_campaign_id_for_source(url, cookies_path, overrides)
|
||||
|
||||
|
||||
def _campaign_resolution_error(platform: str, url: str) -> str:
|
||||
"""Operator-facing message for a native source whose campaign id could not
|
||||
be resolved — names the platform's own lookup mechanism."""
|
||||
if platform == "pixiv":
|
||||
return (
|
||||
f"Could not extract a pixiv user id. source_url={url!r} — expected "
|
||||
"a URL like https://www.pixiv.net/users/<id>."
|
||||
)
|
||||
vanity = extract_vanity(url)
|
||||
return (
|
||||
f"Could not resolve Patreon campaign id. source_url={url!r}; "
|
||||
@@ -172,8 +161,8 @@ async def _run_native_ingester(
|
||||
platform, ctx["url"], ctx["cookies_path"], overrides
|
||||
)
|
||||
if not campaign_id:
|
||||
# Patreon: vanity lookup failed. Pixiv: no numeric user id in the URL.
|
||||
# (SubscribeStar's campaign id is the URL itself — never lands here.)
|
||||
# Patreon: vanity lookup failed. (SubscribeStar's campaign id is the
|
||||
# URL itself — never lands here.)
|
||||
url = ctx["url"]
|
||||
return (
|
||||
DownloadResult(
|
||||
@@ -205,7 +194,7 @@ async def _run_native_ingester(
|
||||
validate=gdl._validate_files,
|
||||
rate_limit=rate_limit,
|
||||
request_sleep=request_sleep,
|
||||
# Uniform across adapters: token platforms (pixiv) authenticate with
|
||||
# Uniform across adapters: a token platform would authenticate with
|
||||
# it, cookie platforms accept-and-ignore — so this construction stays
|
||||
# platform-agnostic.
|
||||
auth_token=ctx["auth_token"],
|
||||
@@ -252,16 +241,11 @@ async def verify_source_credential(
|
||||
if uses_native_ingester(platform):
|
||||
# Native ingester platforms verify via their own lightweight auth probe.
|
||||
# SubscribeStar's probe takes the creator URL directly; Patreon's
|
||||
# resolves the campaign id first; Pixiv's is one OAuth refresh (the
|
||||
# exact call that fails when the token is bad — no feed walk).
|
||||
# resolves the campaign id first.
|
||||
if platform == "subscribestar":
|
||||
from .subscribestar_ingester import verify_subscribestar_credential
|
||||
|
||||
return await verify_subscribestar_credential(url, cookies_path, config_overrides)
|
||||
if platform == "pixiv":
|
||||
from .pixiv_ingester import verify_pixiv_credential
|
||||
|
||||
return await verify_pixiv_credential(auth_token)
|
||||
from .patreon_ingester import verify_patreon_credential
|
||||
|
||||
return await verify_patreon_credential(url, cookies_path, config_overrides)
|
||||
|
||||
@@ -61,8 +61,8 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
|
||||
class ExtensionService:
|
||||
def __init__(self, session: AsyncSession, crypto=None) -> None:
|
||||
self.session = session
|
||||
# Optional decryptor for resolving a token-auth platform's display name
|
||||
# (pixiv) at add-time. None → skip resolution, fall back to the handle.
|
||||
# Optional decryptor for resolving a platform's display name at
|
||||
# add-time. None → skip resolution, fall back to the handle.
|
||||
self._crypto = crypto
|
||||
|
||||
async def quick_add_source(self, url: str) -> dict:
|
||||
@@ -112,12 +112,12 @@ class ExtensionService:
|
||||
self, platform: str, raw_slug: str, url: str
|
||||
) -> str:
|
||||
"""The real display name for a new artist, resolved from the platform at
|
||||
add-time (#130). Our native platforms each have a name source — pixiv the
|
||||
app API (token), patreon the campaigns API, subscribestar the profile
|
||||
page (both cookies). Other platforms (and any failure — no credential,
|
||||
network error) fall back to the URL handle, which is already readable.
|
||||
add-time (#130). Our native platforms each have a name source — patreon
|
||||
the campaigns API, subscribestar the profile page (both cookies). Other
|
||||
platforms (and any failure — no credential, network error) fall back to
|
||||
the URL handle, which is already readable.
|
||||
The resolvers are sync, so they run in an executor."""
|
||||
if self._crypto is None or platform not in ("pixiv", "patreon", "subscribestar"):
|
||||
if self._crypto is None or platform not in ("patreon", "subscribestar"):
|
||||
return raw_slug
|
||||
import asyncio
|
||||
|
||||
@@ -125,15 +125,7 @@ class ExtensionService:
|
||||
cred = CredentialService(self.session, self._crypto)
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
if platform == "pixiv":
|
||||
token = await cred.get_token("pixiv")
|
||||
if not token:
|
||||
return raw_slug
|
||||
from .pixiv_client import PixivClient
|
||||
name = await loop.run_in_executor(
|
||||
None, PixivClient(token).resolve_display_name, raw_slug
|
||||
)
|
||||
elif platform == "patreon":
|
||||
if platform == "patreon":
|
||||
cookies = await cred.get_cookies_path("patreon")
|
||||
from .patreon_resolver import resolve_display_name
|
||||
name = await loop.run_in_executor(
|
||||
|
||||
@@ -45,7 +45,7 @@ from .tag_query import (
|
||||
# provenance (filesystem imports). Returned by facets() as a null-valued
|
||||
# bucket; the frontend maps that null back to this sentinel in the URL so the
|
||||
# bucket is selectable. Underscore-wrapped so it can't collide with a real
|
||||
# gallery-dl platform name (patreon/pixiv/...).
|
||||
# gallery-dl platform name (patreon/hentaifoundry/...).
|
||||
UNSOURCED_PLATFORM = "__unsourced__"
|
||||
|
||||
|
||||
|
||||
@@ -113,9 +113,9 @@ class Ingester:
|
||||
# (e.g. "Patreon API", "SubscribeStar markup").
|
||||
self._drift_label = drift_label or platform
|
||||
# #862 canary opt-out: platforms whose posts legitimately have empty
|
||||
# bodies across large samples (pixiv — caption-less artists are common)
|
||||
# would false-positive the zero-bodies-means-drift alarm; their clients
|
||||
# catch drift structurally (response-shape checks) instead. The
|
||||
# bodies across large samples would false-positive the
|
||||
# zero-bodies-means-drift alarm; their clients catch drift structurally
|
||||
# (response-shape checks) instead. The
|
||||
# "bodies X/N" summary line still surfaces the ratio either way.
|
||||
self._body_canary = body_canary
|
||||
|
||||
|
||||
@@ -1,579 +0,0 @@
|
||||
"""Native Pixiv client — the Pixiv adapter's read path.
|
||||
|
||||
Pixiv has a real (if unofficial) API: the mobile app API gallery-dl drives
|
||||
(`PixivAppAPI`). Per the downloader ground rule — gallery-dl is the
|
||||
known-working base — this client mirrors gallery-dl 1.32.5's request profile
|
||||
EXACTLY: the same iOS app headers on every request, the same OAuth
|
||||
refresh-token dance against oauth.secure.pixiv.net (X-Client-Time +
|
||||
X-Client-Hash), and the same `/v1/user/illusts` walk paginated by `next_url`.
|
||||
Deviating from that profile is how the SubscribeStar/Patreon spikes broke, so
|
||||
any change here should be diffed against gallery-dl's extractor first.
|
||||
|
||||
Feed shape (characterized from gallery-dl 1.32.5, extractor/pixiv.py):
|
||||
- `GET /v1/user/illusts?user_id=<id>` returns `{"illusts": [work...],
|
||||
"next_url": "https://app-api...?user_id=..&offset=30" | null}`.
|
||||
- Pagination: re-issue the SAME endpoint with `next_url`'s query params. The
|
||||
query string doubles as our resumable page cursor (re-fetching it re-serves
|
||||
the same page — the ingest-core resume contract).
|
||||
- A work carries id/title/type(illust|manga|ugoira)/caption(HTML)/
|
||||
create_date(ISO+09:00)/tags[{name,translated_name}]/user/page_count/
|
||||
x_restrict/series/total_view/total_bookmarks/meta_single_page/meta_pages.
|
||||
- Files: multi-page → meta_pages[].image_urls.original; single page →
|
||||
meta_single_page.original_image_url; ugoira → `/v1/ugoira/metadata` zip
|
||||
(600x600 → 1920x1080 URL swap, gallery-dl's default non-original mode).
|
||||
|
||||
`campaign_id` for Pixiv is the numeric user id (extracted from the source URL
|
||||
by `user_id_from_url` — no network resolver needed).
|
||||
|
||||
Gated works: pixiv serves a `https://s.pximg.net/common/images/limit_*.png`
|
||||
placeholder as the "original" when a work is blocked for this account
|
||||
(sanity-level filter, my-pixiv lock, deleted). gallery-dl's fallback for those
|
||||
is a web-AJAX scrape that needs PHPSESSID browser cookies — FC stores only the
|
||||
OAuth refresh token, so (exactly like our previous gallery-dl configuration,
|
||||
which warned "No PHPSESSID cookie set") those works are skipped, via the
|
||||
post_is_gated seam. Auth failures are loud (rotate the refresh token); a
|
||||
response missing the fields we depend on is DRIFT (update this client).
|
||||
|
||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from urllib.parse import parse_qsl, urlsplit
|
||||
|
||||
import requests
|
||||
|
||||
from ..utils.paths import safe_ext
|
||||
from .native_ingest_common import (
|
||||
_MAX_429_RETRIES,
|
||||
NativeAuthError,
|
||||
NativeDriftError,
|
||||
NativeIngestError,
|
||||
make_session,
|
||||
retry_after_seconds,
|
||||
)
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_TIMEOUT_SECONDS = 30.0
|
||||
_API_ROOT = "https://app-api.pixiv.net"
|
||||
_OAUTH_URL = "https://oauth.secure.pixiv.net/auth/token"
|
||||
|
||||
# gallery-dl's public Pixiv-app credentials (PixivAppAPI, also pixivpy's) —
|
||||
# these identify the official iOS app to the API, NOT the operator; the
|
||||
# operator's identity is the OAuth refresh token.
|
||||
_CLIENT_ID = "MOBrBDS8blbauoSck0ZfDbtuzpyT"
|
||||
_CLIENT_SECRET = "lsACyCD94FhDUtGTXi3QzcFE2uU1hqtDaKeqrdwj"
|
||||
_HASH_SECRET = (
|
||||
"28c1fdd170a5204386cb1313c7077b34"
|
||||
"f83e4aaf4aa829ce78c231e05b0bae2c"
|
||||
)
|
||||
|
||||
# The exact header set gallery-dl 1.32.5 installs on its session — the proven
|
||||
# app-API request profile. The Referer also unlocks i.pximg.net media GETs
|
||||
# (403 without it), so the downloader reuses this constant.
|
||||
PIXIV_APP_HEADERS = {
|
||||
"App-OS": "ios",
|
||||
"App-OS-Version": "16.7.2",
|
||||
"App-Version": "7.19.1",
|
||||
"User-Agent": "PixivIOSApp/7.19.1 (iOS 16.7.2; iPhone12,8)",
|
||||
"Referer": "https://app-api.pixiv.net/",
|
||||
}
|
||||
|
||||
# Placeholder image prefix pixiv serves instead of a blocked work's original
|
||||
# (limit_sanity_level / limit_mypixiv / limit_unknown variants).
|
||||
_LIMIT_URL = "https://s.pximg.net/common/images/limit_"
|
||||
|
||||
# The app API reports rate-limiting as an error MESSAGE (often on HTTP 403),
|
||||
# not only as HTTP 429. gallery-dl sleeps 300s in-walk; sleeping that long
|
||||
# inside our time-boxed chunk would eat the whole budget, so we surface it as
|
||||
# a typed 429 and let download_service's cooldown machinery honor the wait.
|
||||
_RATE_LIMIT_RETRY_AFTER = 300.0
|
||||
|
||||
_TITLE_MAX = 50 # gallery-dl pixiv filename template: {title[:50]}
|
||||
|
||||
_RATINGS = {0: "General", 1: "R-18", 2: "R-18G"}
|
||||
|
||||
|
||||
class PixivAPIError(NativeIngestError):
|
||||
"""Base for native Pixiv client failures. status_code / retry_after are
|
||||
inherited from NativeIngestError."""
|
||||
|
||||
|
||||
class PixivAuthError(PixivAPIError, NativeAuthError):
|
||||
"""Auth failure — missing/expired/revoked OAuth refresh token. Fix =
|
||||
rotate the credential (Settings → Credentials → Pixiv), not update the
|
||||
client. Maps to error_type 'auth_error'."""
|
||||
|
||||
|
||||
class PixivDriftError(PixivAPIError, NativeDriftError):
|
||||
"""A response did not match the shape this client depends on (missing
|
||||
`illusts`, un-parseable JSON where JSON was promised). Fail loud so the
|
||||
run flags 'the Pixiv app API changed' instead of silently importing
|
||||
nothing. Maps to API_DRIFT."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class MediaItem:
|
||||
"""One resolved downloadable file belonging to a Pixiv work.
|
||||
|
||||
Fields mirror the other native clients' MediaItem so the downloader and
|
||||
ledger are structurally the same. Pixiv original URLs carry no content
|
||||
hash, so `filehash` is always None and the ledger keys on
|
||||
`<post_id>:<media_id>` where media_id is `p<num>` (page) or `ugoira`
|
||||
(the frame zip) — stable across URL-shape drift.
|
||||
"""
|
||||
|
||||
url: str
|
||||
filename: str
|
||||
kind: str
|
||||
filehash: str | None
|
||||
post_id: str
|
||||
media_id: str
|
||||
|
||||
|
||||
def user_id_from_url(url: str) -> str | None:
|
||||
"""The numeric pixiv user id from a source URL, or None.
|
||||
|
||||
Handles the modern forms FC accepts as sources
|
||||
(https://www.pixiv.net/users/<id>, /en/users/<id>) plus the legacy
|
||||
member.php?id=<id>. This IS the campaign id — no network resolver.
|
||||
"""
|
||||
parts = urlsplit(url or "")
|
||||
if "pixiv.net" not in parts.netloc:
|
||||
return None
|
||||
segs = [s for s in parts.path.split("/") if s]
|
||||
if segs and segs[0] == "en":
|
||||
segs = segs[1:]
|
||||
if len(segs) >= 2 and segs[0] == "users" and segs[1].isdigit():
|
||||
return segs[1]
|
||||
if segs and segs[0] == "member.php":
|
||||
qid = dict(parse_qsl(parts.query)).get("id", "")
|
||||
if qid.isdigit():
|
||||
return qid
|
||||
return None
|
||||
|
||||
|
||||
def _work_filename(work: dict, num: int, url: str) -> str:
|
||||
"""gallery-dl layout parity: `{id}_{title[:50]}_{num:>02}.{extension}`
|
||||
(the downloader sanitizes the final segment)."""
|
||||
title = work.get("title")
|
||||
title50 = (title if isinstance(title, str) else "")[:_TITLE_MAX]
|
||||
ext = safe_ext(urlsplit(url).path.rsplit("/", 1)[-1])
|
||||
return f"{work.get('id')}_{title50}_{num:02d}{ext}"
|
||||
|
||||
|
||||
class PixivClient:
|
||||
"""Synchronous Pixiv app-API read client. Construct with the operator's
|
||||
OAuth refresh token (the same token-type Credential the gallery-dl path
|
||||
consumed as `extractor.pixiv.refresh-token`)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
refresh_token: str | None,
|
||||
*,
|
||||
request_sleep: float = 0.0,
|
||||
max_retries: int = _MAX_429_RETRIES,
|
||||
session: requests.Session | None = None,
|
||||
):
|
||||
self.refresh_token = refresh_token
|
||||
self._request_sleep = request_sleep or 0.0
|
||||
self._max_retries = max_retries
|
||||
# No cookies — the app API authenticates via the Bearer token _login
|
||||
# installs. make_session still supplies the retry/UA plumbing; the
|
||||
# extra_headers overwrite its browser UA with the app profile.
|
||||
self._session = (
|
||||
session if session is not None
|
||||
else make_session(None, extra_headers=PIXIV_APP_HEADERS)
|
||||
)
|
||||
self._authed_user: dict = {}
|
||||
# Monotonic deadline after which the access token must be refreshed;
|
||||
# 0 forces a refresh on first use.
|
||||
self._token_deadline = 0.0
|
||||
|
||||
# -- auth ----------------------------------------------------------------
|
||||
|
||||
def _login(self) -> None:
|
||||
"""Exchange the refresh token for a Bearer access token (gallery-dl's
|
||||
`_login_impl`, including the X-Client-Time/X-Client-Hash pair the
|
||||
endpoint validates). No-op while the current token is still fresh."""
|
||||
if time.monotonic() < self._token_deadline:
|
||||
return
|
||||
if not self.refresh_token:
|
||||
raise PixivAuthError(
|
||||
"No Pixiv refresh token configured — add the OAuth refresh "
|
||||
"token as the Pixiv credential (token type)."
|
||||
)
|
||||
# gallery-dl stamps naive-UTC with a literal +00:00 suffix.
|
||||
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S+00:00")
|
||||
headers = {
|
||||
"X-Client-Time": now,
|
||||
"X-Client-Hash": hashlib.md5(
|
||||
(now + _HASH_SECRET).encode()
|
||||
).hexdigest(),
|
||||
}
|
||||
data = {
|
||||
"client_id": _CLIENT_ID,
|
||||
"client_secret": _CLIENT_SECRET,
|
||||
"grant_type": "refresh_token",
|
||||
"refresh_token": self.refresh_token,
|
||||
"get_secure_url": "1",
|
||||
}
|
||||
try:
|
||||
resp = self._session.post(
|
||||
_OAUTH_URL, data=data, headers=headers,
|
||||
timeout=_TIMEOUT_SECONDS,
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
raise PixivAPIError(f"Pixiv OAuth request failed: {exc}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise PixivAuthError(
|
||||
"Pixiv rejected the refresh token (HTTP "
|
||||
f"{resp.status_code}) — rotate the Pixiv credential.",
|
||||
status_code=resp.status_code,
|
||||
)
|
||||
try:
|
||||
payload = resp.json()["response"]
|
||||
access = payload["access_token"]
|
||||
except (ValueError, KeyError, TypeError) as exc:
|
||||
raise PixivDriftError(
|
||||
f"Pixiv OAuth response shape changed: {exc}"
|
||||
) from exc
|
||||
self._authed_user = payload.get("user") or {}
|
||||
self._session.headers["Authorization"] = f"Bearer {access}"
|
||||
# expires_in is 3600 today; refresh 60s early so a long walk never
|
||||
# rides an expiring token into a spurious 400.
|
||||
expires_in = payload.get("expires_in")
|
||||
lifetime = float(expires_in) if isinstance(expires_in, (int, float)) else 3600.0
|
||||
self._token_deadline = time.monotonic() + max(60.0, lifetime - 60.0)
|
||||
|
||||
# -- request -------------------------------------------------------------
|
||||
|
||||
def _call(self, endpoint: str, params: dict) -> dict:
|
||||
"""Authenticated app-API GET → parsed JSON body, with the shared 429
|
||||
backoff and the loud auth/drift/rate-limit mapping."""
|
||||
self._login()
|
||||
if self._request_sleep > 0:
|
||||
time.sleep(self._request_sleep)
|
||||
url = _API_ROOT + endpoint
|
||||
attempt = 0
|
||||
while True:
|
||||
try:
|
||||
resp = self._session.get(
|
||||
url, params=params, timeout=_TIMEOUT_SECONDS
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
raise PixivAPIError(
|
||||
f"Pixiv request failed ({endpoint}): {exc}"
|
||||
) from exc
|
||||
if resp.status_code == 429 and attempt < self._max_retries:
|
||||
attempt += 1
|
||||
delay = retry_after_seconds(resp, attempt)
|
||||
log.warning(
|
||||
"Pixiv 429 (%s) — backing off %.1fs (retry %d/%d)",
|
||||
endpoint, delay, attempt, self._max_retries,
|
||||
)
|
||||
time.sleep(delay)
|
||||
continue
|
||||
break
|
||||
|
||||
try:
|
||||
body = resp.json()
|
||||
except ValueError as exc:
|
||||
raise PixivDriftError(
|
||||
f"Pixiv returned non-JSON for {endpoint} "
|
||||
f"(HTTP {resp.status_code})"
|
||||
) from exc
|
||||
|
||||
error = body.get("error") if isinstance(body, dict) else None
|
||||
message = ""
|
||||
if isinstance(error, dict):
|
||||
message = str(
|
||||
error.get("user_message") or error.get("message") or ""
|
||||
)
|
||||
# Rate limiting first: the app API reports it as an error MESSAGE
|
||||
# (often on HTTP 403), which must not be mistaken for an auth failure.
|
||||
if resp.status_code == 429 or "rate limit" in message.lower():
|
||||
raise PixivAPIError(
|
||||
f"Pixiv rate limit hit ({endpoint}): {message or 'HTTP 429'}",
|
||||
status_code=429,
|
||||
retry_after=_RATE_LIMIT_RETRY_AFTER,
|
||||
)
|
||||
if resp.status_code in (400, 401, 403):
|
||||
# Invalid/expired access token surfaces as 400 invalid_grant-style
|
||||
# errors on the app API; 401/403 are straight auth rejections.
|
||||
raise PixivAuthError(
|
||||
f"Pixiv rejected the request ({endpoint}, HTTP "
|
||||
f"{resp.status_code}): {message or 'auth rejected'} — "
|
||||
"rotate the Pixiv refresh token.",
|
||||
status_code=resp.status_code,
|
||||
)
|
||||
if resp.status_code >= 400:
|
||||
raise PixivAPIError(
|
||||
f"Pixiv API error ({endpoint}, HTTP {resp.status_code}): "
|
||||
f"{message or 'unknown error'}",
|
||||
status_code=resp.status_code,
|
||||
)
|
||||
if error:
|
||||
# HTTP 200 carrying an error object — unexpected, but never
|
||||
# silently treat it as data.
|
||||
raise PixivAPIError(
|
||||
f"Pixiv API error ({endpoint}): {message or error}",
|
||||
status_code=resp.status_code,
|
||||
)
|
||||
return body
|
||||
|
||||
# -- normalization -------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _normalize(work: dict) -> dict:
|
||||
"""Wrap an app-API work in the `{"id", "attributes", ...}` post shape
|
||||
the platform-agnostic core and shared helpers read. The raw work rides
|
||||
along under `_work` for extract_media / the post record."""
|
||||
title = work.get("title")
|
||||
caption = work.get("caption")
|
||||
wtype = work.get("type")
|
||||
return {
|
||||
"id": work.get("id"),
|
||||
"attributes": {
|
||||
"title": title if isinstance(title, str) else "",
|
||||
"content": caption if isinstance(caption, str) else "",
|
||||
"published_at": work.get("create_date"),
|
||||
"post_type": wtype if isinstance(wtype, str) else "illust",
|
||||
},
|
||||
"_work": work,
|
||||
}
|
||||
|
||||
# -- post-first seams ----------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def post_record_key(post: dict) -> tuple[str, str] | None:
|
||||
"""`(ledger_key, post_id)` gating post-record capture through the seen
|
||||
ledger (`post:<id>` synthetic key), or None when the work has no id."""
|
||||
pid = post.get("id")
|
||||
pid = str(pid) if pid is not None else ""
|
||||
if not pid:
|
||||
return None
|
||||
return (f"post:{pid}", pid)
|
||||
|
||||
@staticmethod
|
||||
def post_meta(post: dict) -> dict:
|
||||
attrs = post.get("attributes") or {}
|
||||
return {"title": attrs.get("title") or None, "date": attrs.get("published_at")}
|
||||
|
||||
@staticmethod
|
||||
def post_is_gated(post: dict) -> bool:
|
||||
"""True when this account cannot fetch the work's real files: pixiv
|
||||
substitutes a `limit_*` placeholder for the original (sanity-level
|
||||
filter / my-pixiv lock / deleted), or zeroes the author (deleted
|
||||
account). Mirrors #874 semantics: gated content leaves NO trace — a
|
||||
placeholder thumbnail and an empty stub would only pollute the
|
||||
archive. (gallery-dl's PHPSESSID web-scrape fallback for these is out
|
||||
of scope: FC holds no pixiv browser cookies — module docstring.)"""
|
||||
work = post.get("_work") or {}
|
||||
user = work.get("user") or {}
|
||||
if not user.get("id"):
|
||||
return True
|
||||
if work.get("meta_pages"):
|
||||
return False
|
||||
single = work.get("meta_single_page") or {}
|
||||
original = single.get("original_image_url")
|
||||
return isinstance(original, str) and original.startswith(_LIMIT_URL)
|
||||
|
||||
# -- media ---------------------------------------------------------------
|
||||
|
||||
def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
|
||||
"""Resolve a work's downloadable files (gallery-dl's `_extract_files`):
|
||||
multi-page originals, the single-page original, or the ugoira frame
|
||||
zip. `included_index` is unused (pixiv works are self-contained)."""
|
||||
work = post.get("_work") or {}
|
||||
pid = str(post.get("id") or "")
|
||||
if not pid or self.post_is_gated(post):
|
||||
return []
|
||||
|
||||
if work.get("type") == "ugoira":
|
||||
return self._ugoira_media(work, pid)
|
||||
|
||||
meta_pages = work.get("meta_pages") or []
|
||||
if meta_pages:
|
||||
items = []
|
||||
for num, page in enumerate(meta_pages):
|
||||
urls = page.get("image_urls") or {}
|
||||
url = urls.get("original")
|
||||
if not isinstance(url, str) or not url:
|
||||
continue
|
||||
items.append(
|
||||
MediaItem(
|
||||
url=url,
|
||||
filename=_work_filename(work, num, url),
|
||||
kind="image",
|
||||
filehash=None,
|
||||
post_id=pid,
|
||||
media_id=f"p{num}",
|
||||
)
|
||||
)
|
||||
return items
|
||||
|
||||
single = work.get("meta_single_page") or {}
|
||||
url = single.get("original_image_url")
|
||||
if not isinstance(url, str) or not url or url.startswith(_LIMIT_URL):
|
||||
return []
|
||||
return [
|
||||
MediaItem(
|
||||
url=url,
|
||||
filename=_work_filename(work, 0, url),
|
||||
kind="image",
|
||||
filehash=None,
|
||||
post_id=pid,
|
||||
media_id="p0",
|
||||
)
|
||||
]
|
||||
|
||||
def _ugoira_meta(self, work: dict, pid: str) -> dict | None:
|
||||
"""Fetch + memoize the ugoira metadata (frames + zip urls) for a work.
|
||||
|
||||
Idempotent and cached on the work dict, so the post record and the
|
||||
media extraction share ONE `/v1/ugoira/metadata` call regardless of
|
||||
which runs first (the core writes the post record BEFORE it extracts
|
||||
media). Returns None — and caches the miss — on a non-auth failure
|
||||
(matching gallery-dl's downgrade); auth failures stay loud."""
|
||||
if "_ugoira_meta" in work:
|
||||
return work["_ugoira_meta"]
|
||||
try:
|
||||
body = self._call("/v1/ugoira/metadata", {"illust_id": pid})
|
||||
meta = body["ugoira_metadata"]
|
||||
except PixivAuthError:
|
||||
raise
|
||||
except (PixivAPIError, KeyError, TypeError) as exc:
|
||||
log.warning("Pixiv ugoira metadata failed for %s: %s", pid, exc)
|
||||
work["_ugoira_meta"] = None
|
||||
return None
|
||||
work["_ugoira_meta"] = meta
|
||||
# Frame delays: a future ugoira→video conversion needs the timings (the
|
||||
# zip alone has none), so the post record captures them.
|
||||
work["_ugoira_frames"] = meta.get("frames") or []
|
||||
return meta
|
||||
|
||||
def fetch_ugoira_frames(self, post: dict) -> None:
|
||||
"""Populate `post['_work']['_ugoira_frames']` for an ugoira post (no-op
|
||||
otherwise). The core writes the post record BEFORE extract_media, so
|
||||
without this the frame timings would never reach the record; this
|
||||
fetches (and memoizes, so extract_media reuses it) the metadata. Injected
|
||||
into the downloader by the ingester, mirroring Patreon's content_fetcher.
|
||||
Auth errors propagate; other failures leave frames unset."""
|
||||
work = post.get("_work") or {}
|
||||
if work.get("type") != "ugoira":
|
||||
return
|
||||
pid = str(post.get("id") or "")
|
||||
if pid:
|
||||
self._ugoira_meta(work, pid)
|
||||
|
||||
def _ugoira_media(self, work: dict, pid: str) -> list[MediaItem]:
|
||||
"""The ugoira frame zip (gallery-dl's default non-original mode):
|
||||
`/v1/ugoira/metadata` → zip_urls.medium with the 600x600→1920x1080
|
||||
swap. A metadata failure downgrades to 'no media' with a warning
|
||||
(matching gallery-dl) instead of failing the walk — except auth
|
||||
failures, which stay loud."""
|
||||
meta = self._ugoira_meta(work, pid)
|
||||
if meta is None:
|
||||
return []
|
||||
try:
|
||||
zip_url = meta["zip_urls"]["medium"]
|
||||
except (KeyError, TypeError) as exc:
|
||||
log.warning("Pixiv ugoira zip url missing for %s: %s", pid, exc)
|
||||
return []
|
||||
url = zip_url.replace("_ugoira600x600", "_ugoira1920x1080", 1)
|
||||
return [
|
||||
MediaItem(
|
||||
url=url,
|
||||
filename=_work_filename(work, 0, url),
|
||||
kind="ugoira",
|
||||
filehash=None,
|
||||
post_id=pid,
|
||||
media_id="ugoira",
|
||||
)
|
||||
]
|
||||
|
||||
# -- iteration -----------------------------------------------------------
|
||||
|
||||
def iter_posts(
|
||||
self, campaign_id: str, cursor: str | None = None
|
||||
) -> Iterator[tuple[dict, dict, str | None]]:
|
||||
"""Yield (post, {}, page_cursor) for every work in the user's feed.
|
||||
|
||||
`campaign_id` is the numeric pixiv user id. `cursor` is the query
|
||||
string of the app API's `next_url` (offset pagination); None fetches
|
||||
page 1. The yielded `page_cursor` is the cursor that FETCHED this
|
||||
work's page, so the core checkpoints a value that re-serves the same
|
||||
page on resume (the shared cursor contract)."""
|
||||
if not str(campaign_id or "").isdigit():
|
||||
raise PixivDriftError(
|
||||
f"Pixiv campaign id must be a numeric user id, got "
|
||||
f"{campaign_id!r}"
|
||||
)
|
||||
current = cursor
|
||||
while True:
|
||||
page_cursor = current
|
||||
if current is None:
|
||||
params: dict = {"user_id": campaign_id}
|
||||
else:
|
||||
params = dict(parse_qsl(current))
|
||||
data = self._call("/v1/user/illusts", params)
|
||||
works = data.get("illusts")
|
||||
if not isinstance(works, list):
|
||||
raise PixivDriftError(
|
||||
"Pixiv user-illusts response had no 'illusts' list "
|
||||
f"(keys: {sorted(data)[:8]})"
|
||||
)
|
||||
for work in works:
|
||||
if not isinstance(work, dict):
|
||||
continue
|
||||
yield self._normalize(work), {}, page_cursor
|
||||
next_url = data.get("next_url")
|
||||
if not next_url:
|
||||
return
|
||||
current = str(next_url).rpartition("?")[2]
|
||||
|
||||
# -- user detail ---------------------------------------------------------
|
||||
|
||||
def resolve_display_name(self, user_id: str) -> str | None:
|
||||
"""The pixiv user's display name via `/v1/user/detail` (gallery-dl's
|
||||
user_detail) — used to name the Artist when a source is added by numeric
|
||||
id. None on any failure (the caller falls back to the id)."""
|
||||
try:
|
||||
body = self._call("/v1/user/detail", {"user_id": str(user_id)})
|
||||
except PixivAPIError:
|
||||
return None
|
||||
name = (body.get("user") or {}).get("name") if isinstance(body, dict) else None
|
||||
return name if isinstance(name, str) and name.strip() else None
|
||||
|
||||
# -- verify --------------------------------------------------------------
|
||||
|
||||
def verify_auth(self) -> tuple[bool | None, str]:
|
||||
"""Cheap credential probe: run the OAuth refresh (the thing that fails
|
||||
when the token is bad) without walking any feed."""
|
||||
try:
|
||||
self._token_deadline = 0.0 # force a real refresh
|
||||
self._login()
|
||||
except PixivAuthError as exc:
|
||||
return False, f"Pixiv rejected the credential — {exc}"
|
||||
except PixivAPIError as exc:
|
||||
return None, f"Couldn't verify (network/HTTP issue): {exc}"
|
||||
account = self._authed_user.get("account") or self._authed_user.get("name")
|
||||
suffix = f" as {account}" if account else ""
|
||||
return True, f"Credentials valid — Pixiv OAuth refresh succeeded{suffix}."
|
||||
|
||||
|
||||
def rating_label(x_restrict) -> str | None:
|
||||
"""Human rating from pixiv's x_restrict (0/1/2) — written into the post
|
||||
record so the archive keeps the R-18 flag without the reader needing to
|
||||
know pixiv's numeric scheme."""
|
||||
if isinstance(x_restrict, bool) or not isinstance(x_restrict, int):
|
||||
return None
|
||||
return _RATINGS.get(x_restrict)
|
||||
@@ -1,276 +0,0 @@
|
||||
"""Native Pixiv media downloader — the Pixiv counterpart to
|
||||
patreon_downloader / subscribestar_downloader.
|
||||
|
||||
Given a normalized Pixiv work and its resolved `MediaItem`s
|
||||
(pixiv_client.extract_media), download the originals to gallery-dl's on-disk
|
||||
layout (so pre-cutover gallery-dl downloads are recognized on disk and not
|
||||
re-fetched), write the post-first sidecars the importer consumes, and report
|
||||
per-media outcomes.
|
||||
|
||||
On-disk layout (matches FC's gallery-dl pixiv config, PLATFORM_DEFAULTS:
|
||||
base-directory `<images_root>/<artist_slug>/pixiv` + `directory:
|
||||
["{category}"]` + filename `{id}_{title[:50]}_{num:>02}.{extension}`):
|
||||
|
||||
<images_root>/<artist_slug>/pixiv/pixiv/<id>_<title50>_<NN>.<ext>
|
||||
|
||||
— note the intentional DOUBLE `pixiv` segment: gallery-dl appended
|
||||
`{category}` under a base-directory that already ended in the platform name,
|
||||
and tier-2 disk-skip parity requires reproducing that exactly. The layout is
|
||||
FLAT (no per-post directory), so the post-first record is `_post_<id>.json`
|
||||
in the same directory (the id suffix prevents the collisions a bare
|
||||
`_post.json` would have here; phase 3 receives explicit post_record_paths, so
|
||||
the name is a convention, not a discovery key).
|
||||
|
||||
Simpler than Patreon (no Mux/yt-dlp video branch) — the one special file is
|
||||
the ugoira frame zip, downloaded as-is; FC's archive-containment import
|
||||
extracts the frames, and the frame DELAYS ride the post record (the zip
|
||||
carries none — a future ugoira→video conversion needs them).
|
||||
|
||||
PURE: no DB; the seen-skip is an injected predicate. FC runs on a plain-HTTP
|
||||
homelab; nothing here uses a secure-context Web API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
from .native_ingest_common import (
|
||||
BaseNativeDownloader,
|
||||
MediaOutcome,
|
||||
PostRecordOutcome,
|
||||
)
|
||||
from .pixiv_client import PIXIV_APP_HEADERS, rating_label
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Control chars (0x00–0x1f + 0x7f DEL) — gallery-dl's default `path-remove`.
|
||||
_GDL_PATH_REMOVE_RE = re.compile(r"[\x00-\x1f\x7f]")
|
||||
|
||||
|
||||
def gdl_clean_filename(name: str) -> str:
|
||||
"""Reproduce gallery-dl's on-disk filename EXACTLY as it wrote it on this
|
||||
Linux host, so the tier-2 disk-skip recognizes pre-cutover files instead of
|
||||
re-downloading them.
|
||||
|
||||
gallery-dl's PathFormat.build_filename is `clean_path(clean_segment(name))`.
|
||||
On Linux (verified against gallery-dl 1.32.5 path.py) the defaults resolve to:
|
||||
- path-restrict "auto" → "/" → clean_segment replaces ONLY "/" → "_"
|
||||
- path-remove "\\x00-\\x1f\\x7f" → clean_path DELETES control chars
|
||||
- path-strip "auto" → "" → NO trailing dot/space stripping
|
||||
Crucially it does NOT touch the Windows-forbidden set (<>:"|?*) — those stay
|
||||
raw in titles on disk. A stricter sanitizer here would rename any such title,
|
||||
miss the on-disk match, and re-pull the whole work. Order mirrors gallery-dl
|
||||
(segment inner, path outer); for these disjoint char sets it's commutative.
|
||||
"""
|
||||
return _GDL_PATH_REMOVE_RE.sub("", name.replace("/", "_"))
|
||||
|
||||
# Enrichment keys copied verbatim from the app-API work dict into the post
|
||||
# record (they're already JSON scalars/objects). Everything lands in
|
||||
# Post.raw_metadata via the importer, so the archive keeps pixiv's stats and
|
||||
# structure without a schema change.
|
||||
_WORK_PASSTHROUGH_KEYS = (
|
||||
"type",
|
||||
"page_count",
|
||||
"width",
|
||||
"height",
|
||||
"total_view",
|
||||
"total_bookmarks",
|
||||
"total_comments",
|
||||
"is_bookmarked",
|
||||
"illust_ai_type",
|
||||
"series",
|
||||
)
|
||||
|
||||
|
||||
class PixivDownloader(BaseNativeDownloader):
|
||||
"""Download resolved Pixiv media to gallery-dl's on-disk layout.
|
||||
Subclasses BaseNativeDownloader for the shared streaming GET
|
||||
(transient-retry + Range-resume) and validation/quarantine. PURE: no DB."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
images_root: Path,
|
||||
cookies_path: str | None = None,
|
||||
*,
|
||||
validate: bool = True,
|
||||
rate_limit: float = 0.0,
|
||||
session: requests.Session | None = None,
|
||||
ugoira_frames_fetcher: Callable[[dict], None] | None = None,
|
||||
):
|
||||
super().__init__(
|
||||
images_root, cookies_path, platform="pixiv",
|
||||
validate=validate, rate_limit=rate_limit, session=session,
|
||||
)
|
||||
# Injected by the ingester (client.fetch_ugoira_frames) so write_post_record
|
||||
# can populate frame timings — which extract_media memoizes, but the core
|
||||
# writes the post record FIRST. Mirrors Patreon's content_fetcher.
|
||||
self._ugoira_frames_fetcher = ugoira_frames_fetcher
|
||||
if session is None:
|
||||
# i.pximg.net 403s any GET without the app Referer; mirror the
|
||||
# client's full app-header profile (gallery-dl serves media off
|
||||
# the same session it drives the API with). An injected session
|
||||
# (tests) owns its own headers.
|
||||
self.session.headers.update(PIXIV_APP_HEADERS)
|
||||
|
||||
# -- public ------------------------------------------------------------
|
||||
|
||||
def download_post(
|
||||
self,
|
||||
post: dict,
|
||||
media_items: list,
|
||||
artist_slug: str,
|
||||
*,
|
||||
is_seen: Callable[[object], bool] = lambda m: False,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
recapture: bool = False,
|
||||
) -> list[MediaOutcome]:
|
||||
"""Download every media item of one work; return per-item outcomes.
|
||||
Mirrors SubscribeStarDownloader.download_post (two-tier skip, mid-post
|
||||
time-box, recapture surfacing)."""
|
||||
flat_dir = self._flat_dir(artist_slug)
|
||||
outcomes: list[MediaOutcome] = []
|
||||
for media in media_items:
|
||||
if should_stop():
|
||||
break
|
||||
try:
|
||||
outcomes.append(
|
||||
self._download_one(
|
||||
post, media, flat_dir, artist_slug, is_seen,
|
||||
recapture=recapture,
|
||||
)
|
||||
)
|
||||
except Exception as exc: # resilient: isolate one item's failure
|
||||
log.warning(
|
||||
"Pixiv media failed (work %s, %s): %s",
|
||||
post.get("id"), getattr(media, "media_id", "?"), exc,
|
||||
)
|
||||
outcomes.append(
|
||||
MediaOutcome(media=media, status="error", path=None, error=str(exc))
|
||||
)
|
||||
return outcomes
|
||||
|
||||
def _flat_dir(self, artist_slug: str) -> Path:
|
||||
# Double platform segment — gallery-dl layout parity (module docstring).
|
||||
return self.images_root / artist_slug / "pixiv" / "pixiv"
|
||||
|
||||
# -- per-item ----------------------------------------------------------
|
||||
|
||||
def _download_one(
|
||||
self,
|
||||
post: dict,
|
||||
media,
|
||||
flat_dir: Path,
|
||||
artist_slug: str,
|
||||
is_seen: Callable[[object], bool],
|
||||
*,
|
||||
recapture: bool = False,
|
||||
) -> MediaOutcome:
|
||||
seen = is_seen(media)
|
||||
if seen and not recapture:
|
||||
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
|
||||
|
||||
# The client's filename already carries the {id}_{title50}_{NN} shape
|
||||
# (raw title, gallery-dl-template order); clean it to the byte-exact
|
||||
# name gallery-dl wrote on disk so tier-2 disk-skip matches (else a
|
||||
# re-download of the whole work). See gdl_clean_filename.
|
||||
media_path = flat_dir / gdl_clean_filename(media.filename)
|
||||
|
||||
if media_path.exists(): # tier-2: already on disk
|
||||
return MediaOutcome(
|
||||
media=media, status="skipped_disk", path=media_path, error=None
|
||||
)
|
||||
# recapture: a seen item not on disk is NOT re-downloaded (recovery's job).
|
||||
if seen:
|
||||
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
|
||||
|
||||
flat_dir.mkdir(parents=True, exist_ok=True)
|
||||
if self._rate_limit > 0:
|
||||
time.sleep(self._rate_limit)
|
||||
|
||||
out_path = self._fetch_get(media.url, media_path)
|
||||
reason, quarantine_dest = self._validate_path(out_path, artist_slug, media.url)
|
||||
if reason is not None:
|
||||
return MediaOutcome(
|
||||
media=media, status="quarantined", path=quarantine_dest, error=reason,
|
||||
)
|
||||
self._write_minimal_sidecar(post, out_path, source_url=media.url)
|
||||
return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
|
||||
|
||||
# -- post record ---------------------------------------------------------
|
||||
|
||||
def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
|
||||
"""Write the post-first `_post_<id>.json` — the sole writer of the post
|
||||
body/metadata on the native path. Beyond the standard body fields, the
|
||||
record carries pixiv's own structure (tags + EN translations, rating,
|
||||
series, view/bookmark counts, AI flag, dimensions, author, ugoira frame
|
||||
delays) so the archive keeps what the platform knows about the work."""
|
||||
attrs = post.get("attributes") or {}
|
||||
work = post.get("_work") or {}
|
||||
title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
|
||||
post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
|
||||
pid = str(post.get("id") or "")
|
||||
if not pid:
|
||||
return PostRecordOutcome(
|
||||
path=None, post_type=post_type, title=title, body_chars=0,
|
||||
)
|
||||
|
||||
content = attrs.get("content")
|
||||
content = content if isinstance(content, str) else ""
|
||||
data: dict = {
|
||||
"category": "pixiv",
|
||||
"id": pid,
|
||||
"title": title or "",
|
||||
"content": content,
|
||||
"published_at": attrs.get("published_at"),
|
||||
# The post permalink is synthesized by platforms/pixiv.py
|
||||
# derive_post_url from `id` at parse time — no url key here.
|
||||
"rating": rating_label(work.get("x_restrict")),
|
||||
}
|
||||
for key in _WORK_PASSTHROUGH_KEYS:
|
||||
if key in work:
|
||||
data[key] = work[key]
|
||||
tags = work.get("tags")
|
||||
if isinstance(tags, list):
|
||||
data["tags"] = [
|
||||
{
|
||||
"name": t.get("name"),
|
||||
"translated_name": t.get("translated_name"),
|
||||
}
|
||||
for t in tags
|
||||
if isinstance(t, dict)
|
||||
]
|
||||
user = work.get("user")
|
||||
if isinstance(user, dict):
|
||||
data["user"] = {
|
||||
"id": user.get("id"),
|
||||
"account": user.get("account"),
|
||||
"name": user.get("name"),
|
||||
}
|
||||
# Ugoira frame timings. extract_media memoizes these, but the core writes
|
||||
# the post record BEFORE extracting media, so fetch them here (shared +
|
||||
# idempotent via the client's memoization) so the record actually keeps
|
||||
# them — the zip carries no timings.
|
||||
if (
|
||||
work.get("type") == "ugoira"
|
||||
and not work.get("_ugoira_frames")
|
||||
and self._ugoira_frames_fetcher is not None
|
||||
):
|
||||
self._ugoira_frames_fetcher(post)
|
||||
frames = work.get("_ugoira_frames")
|
||||
if frames:
|
||||
data["ugoira_frames"] = frames
|
||||
|
||||
flat_dir = self._flat_dir(artist_slug)
|
||||
flat_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = flat_dir / f"_post_{pid}.json"
|
||||
path.write_text(json.dumps(data, indent=2, ensure_ascii=False))
|
||||
return PostRecordOutcome(
|
||||
path=path, post_type=post_type, title=title, body_chars=len(content),
|
||||
)
|
||||
@@ -1,121 +0,0 @@
|
||||
"""Native Pixiv ingester — the Pixiv ADAPTER over the platform-agnostic core
|
||||
(`ingest_core.Ingester`).
|
||||
|
||||
Thin counterpart to patreon_ingester / subscribestar_ingester: wires the Pixiv
|
||||
client/downloader/ledger models/constraints/key into the core and supplies the
|
||||
Pixiv failure mapping. The modes (tick / backfill / recovery / recapture), the
|
||||
seen + dead-letter ledgers, cursor checkpointing, and the post-first capture
|
||||
all live in the core. `download_service.download_source` drives
|
||||
`PixivIngester.run` exactly as it drives the other two.
|
||||
|
||||
`campaign_id` is the numeric pixiv user id (download_backends extracts it from
|
||||
the source URL — no network resolver). Auth is the operator's OAuth refresh
|
||||
token (the token-type Credential), passed as `auth_token` — pixiv is the first
|
||||
native platform authenticating by token rather than cookies, so the uniform
|
||||
constructor accepts both and ignores what it doesn't need.
|
||||
|
||||
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from ..models import PixivFailedMedia, PixivSeenMedia
|
||||
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
|
||||
from .pixiv_client import MediaItem, PixivAPIError, PixivClient
|
||||
from .pixiv_downloader import PixivDownloader
|
||||
|
||||
__all__ = [
|
||||
"DEAD_LETTER_THRESHOLD",
|
||||
"PixivIngester",
|
||||
"_ledger_key",
|
||||
"verify_pixiv_credential",
|
||||
]
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_LEDGER_KEY_MAX = 128
|
||||
|
||||
|
||||
def _ledger_key(media: MediaItem) -> str:
|
||||
"""Stable per-media identity for the cross-run seen-ledger. Pixiv original
|
||||
URLs carry no content hash, so the key is the page/zip identity scoped to
|
||||
its work: `<illust_id>:p<num>` / `<illust_id>:ugoira`. Bounded to the
|
||||
column width."""
|
||||
if media.filehash:
|
||||
return media.filehash
|
||||
return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
|
||||
|
||||
|
||||
class PixivIngester(Ingester):
|
||||
"""Walk a pixiv user's works, download unseen originals, return a
|
||||
`DownloadResult`. A thin adapter over `ingest_core.Ingester`; `client` /
|
||||
`downloader` are injectable seams so unit tests run without network."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
images_root: Path,
|
||||
cookies_path: str | None,
|
||||
session_factory: Callable[[], object],
|
||||
*,
|
||||
validate: bool = True,
|
||||
rate_limit: float = 0.0,
|
||||
request_sleep: float = 0.0,
|
||||
auth_token: str | None = None,
|
||||
client: PixivClient | None = None,
|
||||
downloader: PixivDownloader | None = None,
|
||||
):
|
||||
self.images_root = Path(images_root)
|
||||
self.cookies_path = str(cookies_path) if cookies_path else None
|
||||
resolved_client = (
|
||||
client
|
||||
if client is not None
|
||||
else PixivClient(auth_token, request_sleep=request_sleep)
|
||||
)
|
||||
resolved_downloader = (
|
||||
downloader
|
||||
if downloader is not None
|
||||
else PixivDownloader(
|
||||
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit,
|
||||
# write_post_record runs before extract_media in the core, so it
|
||||
# fetches ugoira frame timings via the SAME client (shared,
|
||||
# memoized) — else the record's ugoira_frames stays empty.
|
||||
ugoira_frames_fetcher=resolved_client.fetch_ugoira_frames,
|
||||
)
|
||||
)
|
||||
super().__init__(
|
||||
client=resolved_client,
|
||||
downloader=resolved_downloader,
|
||||
session_factory=session_factory,
|
||||
seen_model=PixivSeenMedia,
|
||||
failed_model=PixivFailedMedia,
|
||||
seen_constraint="uq_pixiv_seen_media_source_id",
|
||||
failed_constraint="uq_pixiv_failed_media_source_id",
|
||||
ledger_key=_ledger_key,
|
||||
platform="pixiv",
|
||||
error_base=PixivAPIError,
|
||||
# API_DRIFT message phrasing; the base Ingester._failure_result owns
|
||||
# the auth/drift/HTTP→error_type mapping (shared across platforms).
|
||||
drift_label="Pixiv app API",
|
||||
# Captions are legitimately empty for many pixiv artists, so the
|
||||
# zero-bodies #862 canary would false-positive here; the client's
|
||||
# response-shape checks (missing `illusts` → drift) cover the same
|
||||
# failure class structurally.
|
||||
body_canary=False,
|
||||
)
|
||||
|
||||
|
||||
async def verify_pixiv_credential(
|
||||
auth_token: str | None,
|
||||
) -> tuple[bool | None, str]:
|
||||
"""Native Pixiv credential probe — one OAuth refresh via
|
||||
PixivClient.verify_auth (the exact call that fails when the token is
|
||||
bad; no feed walk). Returns the uniform `(ok, message)` contract so
|
||||
download_backends.verify_source_credential treats it like the others."""
|
||||
client = PixivClient(auth_token)
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(None, client.verify_auth)
|
||||
@@ -13,9 +13,8 @@ URL patterns match GS exactly so the existing browser extension
|
||||
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
|
||||
FC downloaders are art-dedicated services only. pixiv was retired at
|
||||
milestone #406 (2026-09-13, rule #171): unregistered here first, which
|
||||
switches it off everywhere this registry is consulted; `pixiv.py` and the
|
||||
pixiv client/downloader/ingester stay in the tree, uncalled, until the
|
||||
milestone's phase 2 deletes them.
|
||||
switched it off everywhere this registry is consulted, then removed from
|
||||
the tree entirely in the milestone's phase 2 (2026-09-21).
|
||||
"""
|
||||
|
||||
from .base import (
|
||||
|
||||
@@ -24,8 +24,8 @@ from typing import Literal
|
||||
# external_post_id chain: `post_id` MUST come before `id` because
|
||||
# SubscribeStar gallery-dl puts the per-attachment id in `id` and the
|
||||
# actual post id in `post_id`; picking `id` first fragments
|
||||
# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
|
||||
# no `post_id` so `id` still wins for them; HF uses `index`, Discord
|
||||
# multi-image SubscribeStar posts into N Post rows. Patreon has
|
||||
# no `post_id` so `id` still wins for it; HF uses `index`, Discord
|
||||
# uses `message_id` — all reached via the remaining chain entries.
|
||||
# (Banked 2026-05-27 during the sidecar audit.)
|
||||
DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
|
||||
@@ -62,7 +62,7 @@ class PlatformInfo:
|
||||
# --- Behavioral hooks ---
|
||||
# Synthesize a post permalink from sidecar data. Required when
|
||||
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
||||
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
||||
# permalink (subscribestar/hf/discord). None = trust the bare
|
||||
# `url` field (patreon).
|
||||
derive_post_url: Callable[[dict], str | None] | None = None
|
||||
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
"""Pixiv — one quirk.
|
||||
|
||||
post_url: the sidecar's `url` (legacy gallery-dl era) is the image URL
|
||||
on `i.pximg.net`, and the native post record (#129) writes no url key
|
||||
at all — the permalink is synthesized from `id` here either way:
|
||||
/artworks/<id>. external_post_id (= `id`) was already correct, so no
|
||||
override there.
|
||||
|
||||
Downloads run through the native ingester (pixiv_ingester.py), not
|
||||
gallery-dl; this registry entry still owns URL validation, sidecar
|
||||
parsing, and the credential surface (the OAuth refresh token).
|
||||
"""
|
||||
|
||||
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
|
||||
|
||||
|
||||
def derive_post_url(data: dict) -> str | None:
|
||||
pid = str_id_value(data.get("id"))
|
||||
if pid:
|
||||
return f"https://www.pixiv.net/artworks/{pid}"
|
||||
return None
|
||||
|
||||
|
||||
INFO = PlatformInfo(
|
||||
key="pixiv",
|
||||
name="Pixiv",
|
||||
description="Download artwork from Pixiv artists",
|
||||
auth_type="token",
|
||||
requires_auth=True,
|
||||
url_pattern=r"^https?://(www\.)?pixiv\.net/",
|
||||
url_examples=[
|
||||
"https://www.pixiv.net/users/12345678",
|
||||
"https://www.pixiv.net/en/users/12345678",
|
||||
],
|
||||
default_config={**GD_DEFAULTS, "content_types": ["all"]},
|
||||
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
|
||||
derive_post_url=derive_post_url,
|
||||
)
|
||||
Reference in New Issue
Block a user