feat: retire pixiv entirely — delete its code, its ledgers, its credential (3977, 3978, 3979)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 34s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m54s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m19s

Milestone #406 phase 2, with issue #3980 folded in. Phase 1 (2026-09-13)
unregistered pixiv so nothing could reach it; the code has sat in the tree
uncalled since. DeviantArt is why the second half is not left for later —
#3069 retired it in code on 2026-08-27 and its stored session was still in
the database seven weeks on.

Step 5 — the code. Deletes pixiv_client, pixiv_downloader, pixiv_ingester,
platforms/pixiv and their three test modules and fixture, then edits out
every remaining reference: the dispatch entry, the campaign-id and verify
branches in download_backends, the display-name branch in extension_service,
and the comments that still described pixiv as live.

The consolidation check the step asked for comes back negative:
native_ingest_common has seven non-pixiv callers (patreon, subscribestar,
membership_reconcile, membership_roster, ingest_core), so nothing there
drops to a single user.

Step 6 — the data, alembic 0102. Drops pixiv_seen_media and
pixiv_failed_media, and deletes credential rows whose platform is not
registered. Written as "not registered" rather than "pixiv" at the step's
explicit ask, which is what makes one migration cover two retirements: the
pixiv OAuth refresh token and DeviantArt's leftover session (#3980). It is
also the only way either row can go — the credentials UI renders one card
per platform from /api/platforms and looks the credential up by key, so an
unregistered platform's row has no card and no Remove button.

Pixiv's Source rows are KEPT, changing the milestone's original data table
on the operator's call. `platform` is stored only on Source; neither Post
nor ImageRecord carries it. Both FKs are ON DELETE SET NULL, so a delete
would not lose the art — but it would drop every pixiv image into the
gallery's __unsourced__ bucket and strip the platform chip off every pixiv
post. The rows stay disabled (0097) and unregistered, so nothing schedules
or downloads through them. Keeping them costs nothing and keeps the
attribution that "the art already downloaded from pixiv stays" is about.

Step 7 — the guard. test_pixiv_code_and_tables_are_gone asserts absence from
the module table and from Base.metadata, not from prose (snippet #3352's
trap). The extension and registry negative assertions were already in place
from phase 1. The final sweep found one real residue step 4 missed:
extension/README.md still advertised pixiv support and carried a "Pixiv
OAuth" manual-test item.

Also replaces the two deleted dispatch tests with one over the whole
NATIVE_INGESTER_PLATFORMS set, so adding a platform and forgetting its
ingester class now fails at unit level rather than as a mid-download
KeyError.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-21 20:46:15 -04:00
co-authored by Claude Opus 5
parent 5aa8e3d81b
commit 24b10d0ffa
23 changed files with 213 additions and 2340 deletions
+6 -22
View File
@@ -26,14 +26,12 @@ from pathlib import Path
from .gallery_dl import DownloadResult, ErrorType
from .patreon_ingester import PatreonIngester
from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
from .pixiv_client import user_id_from_url
from .pixiv_ingester import PixivIngester
from .platforms import known_platform_keys
from .subscribestar_ingester import SubscribeStarIngester
# Platforms whose download + verify go through the native ingester rather than
# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
# they migrate too. pixiv left this set when it was retired (milestone #406).
# they migrate too.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
@@ -67,7 +65,6 @@ def _native_ingester_cls(platform: str):
dispatch pick up the replacement."""
return {
"patreon": PatreonIngester,
"pixiv": PixivIngester,
"subscribestar": SubscribeStarIngester,
}[platform]
@@ -127,25 +124,17 @@ async def _resolve_native_campaign_id(
platform: str, url: str, cookies_path: str | None, overrides: dict,
) -> tuple[str | None, str | None]:
"""`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
feed id IS the creator URL; Pixiv's is the numeric user id parsed straight
from it (no lookup → resolved None either way). Patreon resolves the
feed id IS the creator URL (no lookup → resolved None). Patreon resolves the
campaign id from the vanity URL (resolved non-None when a lookup actually ran,
so phase 3 caches it)."""
if platform == "subscribestar":
return url, None
if platform == "pixiv":
return user_id_from_url(url), None
return await resolve_campaign_id_for_source(url, cookies_path, overrides)
def _campaign_resolution_error(platform: str, url: str) -> str:
"""Operator-facing message for a native source whose campaign id could not
be resolved — names the platform's own lookup mechanism."""
if platform == "pixiv":
return (
f"Could not extract a pixiv user id. source_url={url!r} — expected "
"a URL like https://www.pixiv.net/users/<id>."
)
vanity = extract_vanity(url)
return (
f"Could not resolve Patreon campaign id. source_url={url!r}; "
@@ -172,8 +161,8 @@ async def _run_native_ingester(
platform, ctx["url"], ctx["cookies_path"], overrides
)
if not campaign_id:
# Patreon: vanity lookup failed. Pixiv: no numeric user id in the URL.
# (SubscribeStar's campaign id is the URL itself — never lands here.)
# Patreon: vanity lookup failed. (SubscribeStar's campaign id is the
# URL itself — never lands here.)
url = ctx["url"]
return (
DownloadResult(
@@ -205,7 +194,7 @@ async def _run_native_ingester(
validate=gdl._validate_files,
rate_limit=rate_limit,
request_sleep=request_sleep,
# Uniform across adapters: token platforms (pixiv) authenticate with
# Uniform across adapters: a token platform would authenticate with
# it, cookie platforms accept-and-ignore — so this construction stays
# platform-agnostic.
auth_token=ctx["auth_token"],
@@ -252,16 +241,11 @@ async def verify_source_credential(
if uses_native_ingester(platform):
# Native ingester platforms verify via their own lightweight auth probe.
# SubscribeStar's probe takes the creator URL directly; Patreon's
# resolves the campaign id first; Pixiv's is one OAuth refresh (the
# exact call that fails when the token is bad — no feed walk).
# resolves the campaign id first.
if platform == "subscribestar":
from .subscribestar_ingester import verify_subscribestar_credential
return await verify_subscribestar_credential(url, cookies_path, config_overrides)
if platform == "pixiv":
from .pixiv_ingester import verify_pixiv_credential
return await verify_pixiv_credential(auth_token)
from .patreon_ingester import verify_patreon_credential
return await verify_patreon_credential(url, cookies_path, config_overrides)
+8 -16
View File
@@ -61,8 +61,8 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
class ExtensionService:
def __init__(self, session: AsyncSession, crypto=None) -> None:
self.session = session
# Optional decryptor for resolving a token-auth platform's display name
# (pixiv) at add-time. None → skip resolution, fall back to the handle.
# Optional decryptor for resolving a platform's display name at
# add-time. None → skip resolution, fall back to the handle.
self._crypto = crypto
async def quick_add_source(self, url: str) -> dict:
@@ -112,12 +112,12 @@ class ExtensionService:
self, platform: str, raw_slug: str, url: str
) -> str:
"""The real display name for a new artist, resolved from the platform at
add-time (#130). Our native platforms each have a name source — pixiv the
app API (token), patreon the campaigns API, subscribestar the profile
page (both cookies). Other platforms (and any failure — no credential,
network error) fall back to the URL handle, which is already readable.
add-time (#130). Our native platforms each have a name source — patreon
the campaigns API, subscribestar the profile page (both cookies). Other
platforms (and any failure — no credential, network error) fall back to
the URL handle, which is already readable.
The resolvers are sync, so they run in an executor."""
if self._crypto is None or platform not in ("pixiv", "patreon", "subscribestar"):
if self._crypto is None or platform not in ("patreon", "subscribestar"):
return raw_slug
import asyncio
@@ -125,15 +125,7 @@ class ExtensionService:
cred = CredentialService(self.session, self._crypto)
loop = asyncio.get_running_loop()
try:
if platform == "pixiv":
token = await cred.get_token("pixiv")
if not token:
return raw_slug
from .pixiv_client import PixivClient
name = await loop.run_in_executor(
None, PixivClient(token).resolve_display_name, raw_slug
)
elif platform == "patreon":
if platform == "patreon":
cookies = await cred.get_cookies_path("patreon")
from .patreon_resolver import resolve_display_name
name = await loop.run_in_executor(
+1 -1
View File
@@ -45,7 +45,7 @@ from .tag_query import (
# provenance (filesystem imports). Returned by facets() as a null-valued
# bucket; the frontend maps that null back to this sentinel in the URL so the
# bucket is selectable. Underscore-wrapped so it can't collide with a real
# gallery-dl platform name (patreon/pixiv/...).
# gallery-dl platform name (patreon/hentaifoundry/...).
UNSOURCED_PLATFORM = "__unsourced__"
+3 -3
View File
@@ -113,9 +113,9 @@ class Ingester:
# (e.g. "Patreon API", "SubscribeStar markup").
self._drift_label = drift_label or platform
# #862 canary opt-out: platforms whose posts legitimately have empty
# bodies across large samples (pixiv — caption-less artists are common)
# would false-positive the zero-bodies-means-drift alarm; their clients
# catch drift structurally (response-shape checks) instead. The
# bodies across large samples would false-positive the
# zero-bodies-means-drift alarm; their clients catch drift structurally
# (response-shape checks) instead. The
# "bodies X/N" summary line still surfaces the ratio either way.
self._body_canary = body_canary
-579
View File
@@ -1,579 +0,0 @@
"""Native Pixiv client — the Pixiv adapter's read path.
Pixiv has a real (if unofficial) API: the mobile app API gallery-dl drives
(`PixivAppAPI`). Per the downloader ground rule — gallery-dl is the
known-working base — this client mirrors gallery-dl 1.32.5's request profile
EXACTLY: the same iOS app headers on every request, the same OAuth
refresh-token dance against oauth.secure.pixiv.net (X-Client-Time +
X-Client-Hash), and the same `/v1/user/illusts` walk paginated by `next_url`.
Deviating from that profile is how the SubscribeStar/Patreon spikes broke, so
any change here should be diffed against gallery-dl's extractor first.
Feed shape (characterized from gallery-dl 1.32.5, extractor/pixiv.py):
- `GET /v1/user/illusts?user_id=<id>` returns `{"illusts": [work...],
"next_url": "https://app-api...?user_id=..&offset=30" | null}`.
- Pagination: re-issue the SAME endpoint with `next_url`'s query params. The
query string doubles as our resumable page cursor (re-fetching it re-serves
the same page — the ingest-core resume contract).
- A work carries id/title/type(illust|manga|ugoira)/caption(HTML)/
create_date(ISO+09:00)/tags[{name,translated_name}]/user/page_count/
x_restrict/series/total_view/total_bookmarks/meta_single_page/meta_pages.
- Files: multi-page → meta_pages[].image_urls.original; single page →
meta_single_page.original_image_url; ugoira → `/v1/ugoira/metadata` zip
(600x600 → 1920x1080 URL swap, gallery-dl's default non-original mode).
`campaign_id` for Pixiv is the numeric user id (extracted from the source URL
by `user_id_from_url` — no network resolver needed).
Gated works: pixiv serves a `https://s.pximg.net/common/images/limit_*.png`
placeholder as the "original" when a work is blocked for this account
(sanity-level filter, my-pixiv lock, deleted). gallery-dl's fallback for those
is a web-AJAX scrape that needs PHPSESSID browser cookies — FC stores only the
OAuth refresh token, so (exactly like our previous gallery-dl configuration,
which warned "No PHPSESSID cookie set") those works are skipped, via the
post_is_gated seam. Auth failures are loud (rotate the refresh token); a
response missing the fields we depend on is DRIFT (update this client).
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import hashlib
import logging
import time
from collections.abc import Iterator
from dataclasses import dataclass
from datetime import UTC, datetime
from urllib.parse import parse_qsl, urlsplit
import requests
from ..utils.paths import safe_ext
from .native_ingest_common import (
_MAX_429_RETRIES,
NativeAuthError,
NativeDriftError,
NativeIngestError,
make_session,
retry_after_seconds,
)
log = logging.getLogger(__name__)
_TIMEOUT_SECONDS = 30.0
_API_ROOT = "https://app-api.pixiv.net"
_OAUTH_URL = "https://oauth.secure.pixiv.net/auth/token"
# gallery-dl's public Pixiv-app credentials (PixivAppAPI, also pixivpy's) —
# these identify the official iOS app to the API, NOT the operator; the
# operator's identity is the OAuth refresh token.
_CLIENT_ID = "MOBrBDS8blbauoSck0ZfDbtuzpyT"
_CLIENT_SECRET = "lsACyCD94FhDUtGTXi3QzcFE2uU1hqtDaKeqrdwj"
_HASH_SECRET = (
"28c1fdd170a5204386cb1313c7077b34"
"f83e4aaf4aa829ce78c231e05b0bae2c"
)
# The exact header set gallery-dl 1.32.5 installs on its session — the proven
# app-API request profile. The Referer also unlocks i.pximg.net media GETs
# (403 without it), so the downloader reuses this constant.
PIXIV_APP_HEADERS = {
"App-OS": "ios",
"App-OS-Version": "16.7.2",
"App-Version": "7.19.1",
"User-Agent": "PixivIOSApp/7.19.1 (iOS 16.7.2; iPhone12,8)",
"Referer": "https://app-api.pixiv.net/",
}
# Placeholder image prefix pixiv serves instead of a blocked work's original
# (limit_sanity_level / limit_mypixiv / limit_unknown variants).
_LIMIT_URL = "https://s.pximg.net/common/images/limit_"
# The app API reports rate-limiting as an error MESSAGE (often on HTTP 403),
# not only as HTTP 429. gallery-dl sleeps 300s in-walk; sleeping that long
# inside our time-boxed chunk would eat the whole budget, so we surface it as
# a typed 429 and let download_service's cooldown machinery honor the wait.
_RATE_LIMIT_RETRY_AFTER = 300.0
_TITLE_MAX = 50 # gallery-dl pixiv filename template: {title[:50]}
_RATINGS = {0: "General", 1: "R-18", 2: "R-18G"}
class PixivAPIError(NativeIngestError):
"""Base for native Pixiv client failures. status_code / retry_after are
inherited from NativeIngestError."""
class PixivAuthError(PixivAPIError, NativeAuthError):
"""Auth failure — missing/expired/revoked OAuth refresh token. Fix =
rotate the credential (Settings → Credentials → Pixiv), not update the
client. Maps to error_type 'auth_error'."""
class PixivDriftError(PixivAPIError, NativeDriftError):
"""A response did not match the shape this client depends on (missing
`illusts`, un-parseable JSON where JSON was promised). Fail loud so the
run flags 'the Pixiv app API changed' instead of silently importing
nothing. Maps to API_DRIFT."""
@dataclass
class MediaItem:
"""One resolved downloadable file belonging to a Pixiv work.
Fields mirror the other native clients' MediaItem so the downloader and
ledger are structurally the same. Pixiv original URLs carry no content
hash, so `filehash` is always None and the ledger keys on
`<post_id>:<media_id>` where media_id is `p<num>` (page) or `ugoira`
(the frame zip) — stable across URL-shape drift.
"""
url: str
filename: str
kind: str
filehash: str | None
post_id: str
media_id: str
def user_id_from_url(url: str) -> str | None:
"""The numeric pixiv user id from a source URL, or None.
Handles the modern forms FC accepts as sources
(https://www.pixiv.net/users/<id>, /en/users/<id>) plus the legacy
member.php?id=<id>. This IS the campaign id — no network resolver.
"""
parts = urlsplit(url or "")
if "pixiv.net" not in parts.netloc:
return None
segs = [s for s in parts.path.split("/") if s]
if segs and segs[0] == "en":
segs = segs[1:]
if len(segs) >= 2 and segs[0] == "users" and segs[1].isdigit():
return segs[1]
if segs and segs[0] == "member.php":
qid = dict(parse_qsl(parts.query)).get("id", "")
if qid.isdigit():
return qid
return None
def _work_filename(work: dict, num: int, url: str) -> str:
"""gallery-dl layout parity: `{id}_{title[:50]}_{num:>02}.{extension}`
(the downloader sanitizes the final segment)."""
title = work.get("title")
title50 = (title if isinstance(title, str) else "")[:_TITLE_MAX]
ext = safe_ext(urlsplit(url).path.rsplit("/", 1)[-1])
return f"{work.get('id')}_{title50}_{num:02d}{ext}"
class PixivClient:
"""Synchronous Pixiv app-API read client. Construct with the operator's
OAuth refresh token (the same token-type Credential the gallery-dl path
consumed as `extractor.pixiv.refresh-token`)."""
def __init__(
self,
refresh_token: str | None,
*,
request_sleep: float = 0.0,
max_retries: int = _MAX_429_RETRIES,
session: requests.Session | None = None,
):
self.refresh_token = refresh_token
self._request_sleep = request_sleep or 0.0
self._max_retries = max_retries
# No cookies — the app API authenticates via the Bearer token _login
# installs. make_session still supplies the retry/UA plumbing; the
# extra_headers overwrite its browser UA with the app profile.
self._session = (
session if session is not None
else make_session(None, extra_headers=PIXIV_APP_HEADERS)
)
self._authed_user: dict = {}
# Monotonic deadline after which the access token must be refreshed;
# 0 forces a refresh on first use.
self._token_deadline = 0.0
# -- auth ----------------------------------------------------------------
def _login(self) -> None:
"""Exchange the refresh token for a Bearer access token (gallery-dl's
`_login_impl`, including the X-Client-Time/X-Client-Hash pair the
endpoint validates). No-op while the current token is still fresh."""
if time.monotonic() < self._token_deadline:
return
if not self.refresh_token:
raise PixivAuthError(
"No Pixiv refresh token configured — add the OAuth refresh "
"token as the Pixiv credential (token type)."
)
# gallery-dl stamps naive-UTC with a literal +00:00 suffix.
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S+00:00")
headers = {
"X-Client-Time": now,
"X-Client-Hash": hashlib.md5(
(now + _HASH_SECRET).encode()
).hexdigest(),
}
data = {
"client_id": _CLIENT_ID,
"client_secret": _CLIENT_SECRET,
"grant_type": "refresh_token",
"refresh_token": self.refresh_token,
"get_secure_url": "1",
}
try:
resp = self._session.post(
_OAUTH_URL, data=data, headers=headers,
timeout=_TIMEOUT_SECONDS,
)
except requests.RequestException as exc:
raise PixivAPIError(f"Pixiv OAuth request failed: {exc}") from exc
if resp.status_code >= 400:
raise PixivAuthError(
"Pixiv rejected the refresh token (HTTP "
f"{resp.status_code}) — rotate the Pixiv credential.",
status_code=resp.status_code,
)
try:
payload = resp.json()["response"]
access = payload["access_token"]
except (ValueError, KeyError, TypeError) as exc:
raise PixivDriftError(
f"Pixiv OAuth response shape changed: {exc}"
) from exc
self._authed_user = payload.get("user") or {}
self._session.headers["Authorization"] = f"Bearer {access}"
# expires_in is 3600 today; refresh 60s early so a long walk never
# rides an expiring token into a spurious 400.
expires_in = payload.get("expires_in")
lifetime = float(expires_in) if isinstance(expires_in, (int, float)) else 3600.0
self._token_deadline = time.monotonic() + max(60.0, lifetime - 60.0)
# -- request -------------------------------------------------------------
def _call(self, endpoint: str, params: dict) -> dict:
"""Authenticated app-API GET → parsed JSON body, with the shared 429
backoff and the loud auth/drift/rate-limit mapping."""
self._login()
if self._request_sleep > 0:
time.sleep(self._request_sleep)
url = _API_ROOT + endpoint
attempt = 0
while True:
try:
resp = self._session.get(
url, params=params, timeout=_TIMEOUT_SECONDS
)
except requests.RequestException as exc:
raise PixivAPIError(
f"Pixiv request failed ({endpoint}): {exc}"
) from exc
if resp.status_code == 429 and attempt < self._max_retries:
attempt += 1
delay = retry_after_seconds(resp, attempt)
log.warning(
"Pixiv 429 (%s) — backing off %.1fs (retry %d/%d)",
endpoint, delay, attempt, self._max_retries,
)
time.sleep(delay)
continue
break
try:
body = resp.json()
except ValueError as exc:
raise PixivDriftError(
f"Pixiv returned non-JSON for {endpoint} "
f"(HTTP {resp.status_code})"
) from exc
error = body.get("error") if isinstance(body, dict) else None
message = ""
if isinstance(error, dict):
message = str(
error.get("user_message") or error.get("message") or ""
)
# Rate limiting first: the app API reports it as an error MESSAGE
# (often on HTTP 403), which must not be mistaken for an auth failure.
if resp.status_code == 429 or "rate limit" in message.lower():
raise PixivAPIError(
f"Pixiv rate limit hit ({endpoint}): {message or 'HTTP 429'}",
status_code=429,
retry_after=_RATE_LIMIT_RETRY_AFTER,
)
if resp.status_code in (400, 401, 403):
# Invalid/expired access token surfaces as 400 invalid_grant-style
# errors on the app API; 401/403 are straight auth rejections.
raise PixivAuthError(
f"Pixiv rejected the request ({endpoint}, HTTP "
f"{resp.status_code}): {message or 'auth rejected'}"
"rotate the Pixiv refresh token.",
status_code=resp.status_code,
)
if resp.status_code >= 400:
raise PixivAPIError(
f"Pixiv API error ({endpoint}, HTTP {resp.status_code}): "
f"{message or 'unknown error'}",
status_code=resp.status_code,
)
if error:
# HTTP 200 carrying an error object — unexpected, but never
# silently treat it as data.
raise PixivAPIError(
f"Pixiv API error ({endpoint}): {message or error}",
status_code=resp.status_code,
)
return body
# -- normalization -------------------------------------------------------
@staticmethod
def _normalize(work: dict) -> dict:
"""Wrap an app-API work in the `{"id", "attributes", ...}` post shape
the platform-agnostic core and shared helpers read. The raw work rides
along under `_work` for extract_media / the post record."""
title = work.get("title")
caption = work.get("caption")
wtype = work.get("type")
return {
"id": work.get("id"),
"attributes": {
"title": title if isinstance(title, str) else "",
"content": caption if isinstance(caption, str) else "",
"published_at": work.get("create_date"),
"post_type": wtype if isinstance(wtype, str) else "illust",
},
"_work": work,
}
# -- post-first seams ----------------------------------------------------
@staticmethod
def post_record_key(post: dict) -> tuple[str, str] | None:
"""`(ledger_key, post_id)` gating post-record capture through the seen
ledger (`post:<id>` synthetic key), or None when the work has no id."""
pid = post.get("id")
pid = str(pid) if pid is not None else ""
if not pid:
return None
return (f"post:{pid}", pid)
@staticmethod
def post_meta(post: dict) -> dict:
attrs = post.get("attributes") or {}
return {"title": attrs.get("title") or None, "date": attrs.get("published_at")}
@staticmethod
def post_is_gated(post: dict) -> bool:
"""True when this account cannot fetch the work's real files: pixiv
substitutes a `limit_*` placeholder for the original (sanity-level
filter / my-pixiv lock / deleted), or zeroes the author (deleted
account). Mirrors #874 semantics: gated content leaves NO trace — a
placeholder thumbnail and an empty stub would only pollute the
archive. (gallery-dl's PHPSESSID web-scrape fallback for these is out
of scope: FC holds no pixiv browser cookies — module docstring.)"""
work = post.get("_work") or {}
user = work.get("user") or {}
if not user.get("id"):
return True
if work.get("meta_pages"):
return False
single = work.get("meta_single_page") or {}
original = single.get("original_image_url")
return isinstance(original, str) and original.startswith(_LIMIT_URL)
# -- media ---------------------------------------------------------------
def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
"""Resolve a work's downloadable files (gallery-dl's `_extract_files`):
multi-page originals, the single-page original, or the ugoira frame
zip. `included_index` is unused (pixiv works are self-contained)."""
work = post.get("_work") or {}
pid = str(post.get("id") or "")
if not pid or self.post_is_gated(post):
return []
if work.get("type") == "ugoira":
return self._ugoira_media(work, pid)
meta_pages = work.get("meta_pages") or []
if meta_pages:
items = []
for num, page in enumerate(meta_pages):
urls = page.get("image_urls") or {}
url = urls.get("original")
if not isinstance(url, str) or not url:
continue
items.append(
MediaItem(
url=url,
filename=_work_filename(work, num, url),
kind="image",
filehash=None,
post_id=pid,
media_id=f"p{num}",
)
)
return items
single = work.get("meta_single_page") or {}
url = single.get("original_image_url")
if not isinstance(url, str) or not url or url.startswith(_LIMIT_URL):
return []
return [
MediaItem(
url=url,
filename=_work_filename(work, 0, url),
kind="image",
filehash=None,
post_id=pid,
media_id="p0",
)
]
def _ugoira_meta(self, work: dict, pid: str) -> dict | None:
"""Fetch + memoize the ugoira metadata (frames + zip urls) for a work.
Idempotent and cached on the work dict, so the post record and the
media extraction share ONE `/v1/ugoira/metadata` call regardless of
which runs first (the core writes the post record BEFORE it extracts
media). Returns None — and caches the miss — on a non-auth failure
(matching gallery-dl's downgrade); auth failures stay loud."""
if "_ugoira_meta" in work:
return work["_ugoira_meta"]
try:
body = self._call("/v1/ugoira/metadata", {"illust_id": pid})
meta = body["ugoira_metadata"]
except PixivAuthError:
raise
except (PixivAPIError, KeyError, TypeError) as exc:
log.warning("Pixiv ugoira metadata failed for %s: %s", pid, exc)
work["_ugoira_meta"] = None
return None
work["_ugoira_meta"] = meta
# Frame delays: a future ugoira→video conversion needs the timings (the
# zip alone has none), so the post record captures them.
work["_ugoira_frames"] = meta.get("frames") or []
return meta
def fetch_ugoira_frames(self, post: dict) -> None:
"""Populate `post['_work']['_ugoira_frames']` for an ugoira post (no-op
otherwise). The core writes the post record BEFORE extract_media, so
without this the frame timings would never reach the record; this
fetches (and memoizes, so extract_media reuses it) the metadata. Injected
into the downloader by the ingester, mirroring Patreon's content_fetcher.
Auth errors propagate; other failures leave frames unset."""
work = post.get("_work") or {}
if work.get("type") != "ugoira":
return
pid = str(post.get("id") or "")
if pid:
self._ugoira_meta(work, pid)
def _ugoira_media(self, work: dict, pid: str) -> list[MediaItem]:
"""The ugoira frame zip (gallery-dl's default non-original mode):
`/v1/ugoira/metadata` → zip_urls.medium with the 600x600→1920x1080
swap. A metadata failure downgrades to 'no media' with a warning
(matching gallery-dl) instead of failing the walk — except auth
failures, which stay loud."""
meta = self._ugoira_meta(work, pid)
if meta is None:
return []
try:
zip_url = meta["zip_urls"]["medium"]
except (KeyError, TypeError) as exc:
log.warning("Pixiv ugoira zip url missing for %s: %s", pid, exc)
return []
url = zip_url.replace("_ugoira600x600", "_ugoira1920x1080", 1)
return [
MediaItem(
url=url,
filename=_work_filename(work, 0, url),
kind="ugoira",
filehash=None,
post_id=pid,
media_id="ugoira",
)
]
# -- iteration -----------------------------------------------------------
def iter_posts(
self, campaign_id: str, cursor: str | None = None
) -> Iterator[tuple[dict, dict, str | None]]:
"""Yield (post, {}, page_cursor) for every work in the user's feed.
`campaign_id` is the numeric pixiv user id. `cursor` is the query
string of the app API's `next_url` (offset pagination); None fetches
page 1. The yielded `page_cursor` is the cursor that FETCHED this
work's page, so the core checkpoints a value that re-serves the same
page on resume (the shared cursor contract)."""
if not str(campaign_id or "").isdigit():
raise PixivDriftError(
f"Pixiv campaign id must be a numeric user id, got "
f"{campaign_id!r}"
)
current = cursor
while True:
page_cursor = current
if current is None:
params: dict = {"user_id": campaign_id}
else:
params = dict(parse_qsl(current))
data = self._call("/v1/user/illusts", params)
works = data.get("illusts")
if not isinstance(works, list):
raise PixivDriftError(
"Pixiv user-illusts response had no 'illusts' list "
f"(keys: {sorted(data)[:8]})"
)
for work in works:
if not isinstance(work, dict):
continue
yield self._normalize(work), {}, page_cursor
next_url = data.get("next_url")
if not next_url:
return
current = str(next_url).rpartition("?")[2]
# -- user detail ---------------------------------------------------------
def resolve_display_name(self, user_id: str) -> str | None:
"""The pixiv user's display name via `/v1/user/detail` (gallery-dl's
user_detail) — used to name the Artist when a source is added by numeric
id. None on any failure (the caller falls back to the id)."""
try:
body = self._call("/v1/user/detail", {"user_id": str(user_id)})
except PixivAPIError:
return None
name = (body.get("user") or {}).get("name") if isinstance(body, dict) else None
return name if isinstance(name, str) and name.strip() else None
# -- verify --------------------------------------------------------------
def verify_auth(self) -> tuple[bool | None, str]:
"""Cheap credential probe: run the OAuth refresh (the thing that fails
when the token is bad) without walking any feed."""
try:
self._token_deadline = 0.0 # force a real refresh
self._login()
except PixivAuthError as exc:
return False, f"Pixiv rejected the credential — {exc}"
except PixivAPIError as exc:
return None, f"Couldn't verify (network/HTTP issue): {exc}"
account = self._authed_user.get("account") or self._authed_user.get("name")
suffix = f" as {account}" if account else ""
return True, f"Credentials valid — Pixiv OAuth refresh succeeded{suffix}."
def rating_label(x_restrict) -> str | None:
"""Human rating from pixiv's x_restrict (0/1/2) — written into the post
record so the archive keeps the R-18 flag without the reader needing to
know pixiv's numeric scheme."""
if isinstance(x_restrict, bool) or not isinstance(x_restrict, int):
return None
return _RATINGS.get(x_restrict)
-276
View File
@@ -1,276 +0,0 @@
"""Native Pixiv media downloader — the Pixiv counterpart to
patreon_downloader / subscribestar_downloader.
Given a normalized Pixiv work and its resolved `MediaItem`s
(pixiv_client.extract_media), download the originals to gallery-dl's on-disk
layout (so pre-cutover gallery-dl downloads are recognized on disk and not
re-fetched), write the post-first sidecars the importer consumes, and report
per-media outcomes.
On-disk layout (matches FC's gallery-dl pixiv config, PLATFORM_DEFAULTS:
base-directory `<images_root>/<artist_slug>/pixiv` + `directory:
["{category}"]` + filename `{id}_{title[:50]}_{num:>02}.{extension}`):
<images_root>/<artist_slug>/pixiv/pixiv/<id>_<title50>_<NN>.<ext>
— note the intentional DOUBLE `pixiv` segment: gallery-dl appended
`{category}` under a base-directory that already ended in the platform name,
and tier-2 disk-skip parity requires reproducing that exactly. The layout is
FLAT (no per-post directory), so the post-first record is `_post_<id>.json`
in the same directory (the id suffix prevents the collisions a bare
`_post.json` would have here; phase 3 receives explicit post_record_paths, so
the name is a convention, not a discovery key).
Simpler than Patreon (no Mux/yt-dlp video branch) — the one special file is
the ugoira frame zip, downloaded as-is; FC's archive-containment import
extracts the frames, and the frame DELAYS ride the post record (the zip
carries none — a future ugoira→video conversion needs them).
PURE: no DB; the seen-skip is an injected predicate. FC runs on a plain-HTTP
homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import json
import logging
import re
import time
from collections.abc import Callable
from pathlib import Path
import requests
from .native_ingest_common import (
BaseNativeDownloader,
MediaOutcome,
PostRecordOutcome,
)
from .pixiv_client import PIXIV_APP_HEADERS, rating_label
log = logging.getLogger(__name__)
# Control chars (0x000x1f + 0x7f DEL) — gallery-dl's default `path-remove`.
_GDL_PATH_REMOVE_RE = re.compile(r"[\x00-\x1f\x7f]")
def gdl_clean_filename(name: str) -> str:
"""Reproduce gallery-dl's on-disk filename EXACTLY as it wrote it on this
Linux host, so the tier-2 disk-skip recognizes pre-cutover files instead of
re-downloading them.
gallery-dl's PathFormat.build_filename is `clean_path(clean_segment(name))`.
On Linux (verified against gallery-dl 1.32.5 path.py) the defaults resolve to:
- path-restrict "auto""/" → clean_segment replaces ONLY "/""_"
- path-remove "\\x00-\\x1f\\x7f" → clean_path DELETES control chars
- path-strip "auto""" → NO trailing dot/space stripping
Crucially it does NOT touch the Windows-forbidden set (<>:"|?*) — those stay
raw in titles on disk. A stricter sanitizer here would rename any such title,
miss the on-disk match, and re-pull the whole work. Order mirrors gallery-dl
(segment inner, path outer); for these disjoint char sets it's commutative.
"""
return _GDL_PATH_REMOVE_RE.sub("", name.replace("/", "_"))
# Enrichment keys copied verbatim from the app-API work dict into the post
# record (they're already JSON scalars/objects). Everything lands in
# Post.raw_metadata via the importer, so the archive keeps pixiv's stats and
# structure without a schema change.
_WORK_PASSTHROUGH_KEYS = (
"type",
"page_count",
"width",
"height",
"total_view",
"total_bookmarks",
"total_comments",
"is_bookmarked",
"illust_ai_type",
"series",
)
class PixivDownloader(BaseNativeDownloader):
"""Download resolved Pixiv media to gallery-dl's on-disk layout.
Subclasses BaseNativeDownloader for the shared streaming GET
(transient-retry + Range-resume) and validation/quarantine. PURE: no DB."""
def __init__(
self,
images_root: Path,
cookies_path: str | None = None,
*,
validate: bool = True,
rate_limit: float = 0.0,
session: requests.Session | None = None,
ugoira_frames_fetcher: Callable[[dict], None] | None = None,
):
super().__init__(
images_root, cookies_path, platform="pixiv",
validate=validate, rate_limit=rate_limit, session=session,
)
# Injected by the ingester (client.fetch_ugoira_frames) so write_post_record
# can populate frame timings — which extract_media memoizes, but the core
# writes the post record FIRST. Mirrors Patreon's content_fetcher.
self._ugoira_frames_fetcher = ugoira_frames_fetcher
if session is None:
# i.pximg.net 403s any GET without the app Referer; mirror the
# client's full app-header profile (gallery-dl serves media off
# the same session it drives the API with). An injected session
# (tests) owns its own headers.
self.session.headers.update(PIXIV_APP_HEADERS)
# -- public ------------------------------------------------------------
def download_post(
self,
post: dict,
media_items: list,
artist_slug: str,
*,
is_seen: Callable[[object], bool] = lambda m: False,
should_stop: Callable[[], bool] = lambda: False,
recapture: bool = False,
) -> list[MediaOutcome]:
"""Download every media item of one work; return per-item outcomes.
Mirrors SubscribeStarDownloader.download_post (two-tier skip, mid-post
time-box, recapture surfacing)."""
flat_dir = self._flat_dir(artist_slug)
outcomes: list[MediaOutcome] = []
for media in media_items:
if should_stop():
break
try:
outcomes.append(
self._download_one(
post, media, flat_dir, artist_slug, is_seen,
recapture=recapture,
)
)
except Exception as exc: # resilient: isolate one item's failure
log.warning(
"Pixiv media failed (work %s, %s): %s",
post.get("id"), getattr(media, "media_id", "?"), exc,
)
outcomes.append(
MediaOutcome(media=media, status="error", path=None, error=str(exc))
)
return outcomes
def _flat_dir(self, artist_slug: str) -> Path:
# Double platform segment — gallery-dl layout parity (module docstring).
return self.images_root / artist_slug / "pixiv" / "pixiv"
# -- per-item ----------------------------------------------------------
def _download_one(
self,
post: dict,
media,
flat_dir: Path,
artist_slug: str,
is_seen: Callable[[object], bool],
*,
recapture: bool = False,
) -> MediaOutcome:
seen = is_seen(media)
if seen and not recapture:
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
# The client's filename already carries the {id}_{title50}_{NN} shape
# (raw title, gallery-dl-template order); clean it to the byte-exact
# name gallery-dl wrote on disk so tier-2 disk-skip matches (else a
# re-download of the whole work). See gdl_clean_filename.
media_path = flat_dir / gdl_clean_filename(media.filename)
if media_path.exists(): # tier-2: already on disk
return MediaOutcome(
media=media, status="skipped_disk", path=media_path, error=None
)
# recapture: a seen item not on disk is NOT re-downloaded (recovery's job).
if seen:
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
flat_dir.mkdir(parents=True, exist_ok=True)
if self._rate_limit > 0:
time.sleep(self._rate_limit)
out_path = self._fetch_get(media.url, media_path)
reason, quarantine_dest = self._validate_path(out_path, artist_slug, media.url)
if reason is not None:
return MediaOutcome(
media=media, status="quarantined", path=quarantine_dest, error=reason,
)
self._write_minimal_sidecar(post, out_path, source_url=media.url)
return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
# -- post record ---------------------------------------------------------
def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
"""Write the post-first `_post_<id>.json` — the sole writer of the post
body/metadata on the native path. Beyond the standard body fields, the
record carries pixiv's own structure (tags + EN translations, rating,
series, view/bookmark counts, AI flag, dimensions, author, ugoira frame
delays) so the archive keeps what the platform knows about the work."""
attrs = post.get("attributes") or {}
work = post.get("_work") or {}
title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
pid = str(post.get("id") or "")
if not pid:
return PostRecordOutcome(
path=None, post_type=post_type, title=title, body_chars=0,
)
content = attrs.get("content")
content = content if isinstance(content, str) else ""
data: dict = {
"category": "pixiv",
"id": pid,
"title": title or "",
"content": content,
"published_at": attrs.get("published_at"),
# The post permalink is synthesized by platforms/pixiv.py
# derive_post_url from `id` at parse time — no url key here.
"rating": rating_label(work.get("x_restrict")),
}
for key in _WORK_PASSTHROUGH_KEYS:
if key in work:
data[key] = work[key]
tags = work.get("tags")
if isinstance(tags, list):
data["tags"] = [
{
"name": t.get("name"),
"translated_name": t.get("translated_name"),
}
for t in tags
if isinstance(t, dict)
]
user = work.get("user")
if isinstance(user, dict):
data["user"] = {
"id": user.get("id"),
"account": user.get("account"),
"name": user.get("name"),
}
# Ugoira frame timings. extract_media memoizes these, but the core writes
# the post record BEFORE extracting media, so fetch them here (shared +
# idempotent via the client's memoization) so the record actually keeps
# them — the zip carries no timings.
if (
work.get("type") == "ugoira"
and not work.get("_ugoira_frames")
and self._ugoira_frames_fetcher is not None
):
self._ugoira_frames_fetcher(post)
frames = work.get("_ugoira_frames")
if frames:
data["ugoira_frames"] = frames
flat_dir = self._flat_dir(artist_slug)
flat_dir.mkdir(parents=True, exist_ok=True)
path = flat_dir / f"_post_{pid}.json"
path.write_text(json.dumps(data, indent=2, ensure_ascii=False))
return PostRecordOutcome(
path=path, post_type=post_type, title=title, body_chars=len(content),
)
-121
View File
@@ -1,121 +0,0 @@
"""Native Pixiv ingester — the Pixiv ADAPTER over the platform-agnostic core
(`ingest_core.Ingester`).
Thin counterpart to patreon_ingester / subscribestar_ingester: wires the Pixiv
client/downloader/ledger models/constraints/key into the core and supplies the
Pixiv failure mapping. The modes (tick / backfill / recovery / recapture), the
seen + dead-letter ledgers, cursor checkpointing, and the post-first capture
all live in the core. `download_service.download_source` drives
`PixivIngester.run` exactly as it drives the other two.
`campaign_id` is the numeric pixiv user id (download_backends extracts it from
the source URL — no network resolver). Auth is the operator's OAuth refresh
token (the token-type Credential), passed as `auth_token` — pixiv is the first
native platform authenticating by token rather than cookies, so the uniform
constructor accepts both and ignores what it doesn't need.
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Callable
from pathlib import Path
from ..models import PixivFailedMedia, PixivSeenMedia
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
from .pixiv_client import MediaItem, PixivAPIError, PixivClient
from .pixiv_downloader import PixivDownloader
__all__ = [
"DEAD_LETTER_THRESHOLD",
"PixivIngester",
"_ledger_key",
"verify_pixiv_credential",
]
log = logging.getLogger(__name__)
_LEDGER_KEY_MAX = 128
def _ledger_key(media: MediaItem) -> str:
"""Stable per-media identity for the cross-run seen-ledger. Pixiv original
URLs carry no content hash, so the key is the page/zip identity scoped to
its work: `<illust_id>:p<num>` / `<illust_id>:ugoira`. Bounded to the
column width."""
if media.filehash:
return media.filehash
return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
class PixivIngester(Ingester):
"""Walk a pixiv user's works, download unseen originals, return a
`DownloadResult`. A thin adapter over `ingest_core.Ingester`; `client` /
`downloader` are injectable seams so unit tests run without network."""
def __init__(
self,
images_root: Path,
cookies_path: str | None,
session_factory: Callable[[], object],
*,
validate: bool = True,
rate_limit: float = 0.0,
request_sleep: float = 0.0,
auth_token: str | None = None,
client: PixivClient | None = None,
downloader: PixivDownloader | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
resolved_client = (
client
if client is not None
else PixivClient(auth_token, request_sleep=request_sleep)
)
resolved_downloader = (
downloader
if downloader is not None
else PixivDownloader(
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit,
# write_post_record runs before extract_media in the core, so it
# fetches ugoira frame timings via the SAME client (shared,
# memoized) — else the record's ugoira_frames stays empty.
ugoira_frames_fetcher=resolved_client.fetch_ugoira_frames,
)
)
super().__init__(
client=resolved_client,
downloader=resolved_downloader,
session_factory=session_factory,
seen_model=PixivSeenMedia,
failed_model=PixivFailedMedia,
seen_constraint="uq_pixiv_seen_media_source_id",
failed_constraint="uq_pixiv_failed_media_source_id",
ledger_key=_ledger_key,
platform="pixiv",
error_base=PixivAPIError,
# API_DRIFT message phrasing; the base Ingester._failure_result owns
# the auth/drift/HTTP→error_type mapping (shared across platforms).
drift_label="Pixiv app API",
# Captions are legitimately empty for many pixiv artists, so the
# zero-bodies #862 canary would false-positive here; the client's
# response-shape checks (missing `illusts` → drift) cover the same
# failure class structurally.
body_canary=False,
)
async def verify_pixiv_credential(
auth_token: str | None,
) -> tuple[bool | None, str]:
"""Native Pixiv credential probe — one OAuth refresh via
PixivClient.verify_auth (the exact call that fails when the token is
bad; no feed walk). Returns the uniform `(ok, message)` contract so
download_backends.verify_source_credential treats it like the others."""
client = PixivClient(auth_token)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, client.verify_auth)
+2 -3
View File
@@ -13,9 +13,8 @@ URL patterns match GS exactly so the existing browser extension
hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
FC downloaders are art-dedicated services only. pixiv was retired at
milestone #406 (2026-09-13, rule #171): unregistered here first, which
switches it off everywhere this registry is consulted; `pixiv.py` and the
pixiv client/downloader/ingester stay in the tree, uncalled, until the
milestone's phase 2 deletes them.
switched it off everywhere this registry is consulted, then removed from
the tree entirely in the milestone's phase 2 (2026-09-21).
"""
from .base import (
+3 -3
View File
@@ -24,8 +24,8 @@ from typing import Literal
# external_post_id chain: `post_id` MUST come before `id` because
# SubscribeStar gallery-dl puts the per-attachment id in `id` and the
# actual post id in `post_id`; picking `id` first fragments
# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
# no `post_id` so `id` still wins for them; HF uses `index`, Discord
# multi-image SubscribeStar posts into N Post rows. Patreon has
# no `post_id` so `id` still wins for it; HF uses `index`, Discord
# uses `message_id` — all reached via the remaining chain entries.
# (Banked 2026-05-27 during the sidecar audit.)
DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
@@ -62,7 +62,7 @@ class PlatformInfo:
# --- Behavioral hooks ---
# Synthesize a post permalink from sidecar data. Required when
# gallery-dl's `url` field is the file/CDN URL rather than the post
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
# permalink (subscribestar/hf/discord). None = trust the bare
# `url` field (patreon).
derive_post_url: Callable[[dict], str | None] | None = None
-38
View File
@@ -1,38 +0,0 @@
"""Pixiv — one quirk.
post_url: the sidecar's `url` (legacy gallery-dl era) is the image URL
on `i.pximg.net`, and the native post record (#129) writes no url key
at all — the permalink is synthesized from `id` here either way:
/artworks/<id>. external_post_id (= `id`) was already correct, so no
override there.
Downloads run through the native ingester (pixiv_ingester.py), not
gallery-dl; this registry entry still owns URL validation, sidecar
parsing, and the credential surface (the OAuth refresh token).
"""
from .base import GD_DEFAULTS, PlatformInfo, str_id_value
def derive_post_url(data: dict) -> str | None:
pid = str_id_value(data.get("id"))
if pid:
return f"https://www.pixiv.net/artworks/{pid}"
return None
INFO = PlatformInfo(
key="pixiv",
name="Pixiv",
description="Download artwork from Pixiv artists",
auth_type="token",
requires_auth=True,
url_pattern=r"^https?://(www\.)?pixiv\.net/",
url_examples=[
"https://www.pixiv.net/users/12345678",
"https://www.pixiv.net/en/users/12345678",
],
default_config={**GD_DEFAULTS, "content_types": ["all"]},
notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
derive_post_url=derive_post_url,
)