feat: switch pixiv off — unregistered, unreachable, and refused at dispatch (406 phase 1)
Milestone 406 retires pixiv (rule 171) in two phases at the operator's explicit ask: switch it off, then later delete its code. This is the switch-off. Steps 2 and 3 ship together because each is a half-state of the other: unregistered but still in the extension, pixiv creator pages would offer a button the backend then refuses. Reachability removed, never gated (rule 22 - no flag, no `if platform == "pixiv"`): - platforms registry: pixiv unregistered, so /api/platforms, the source validator and quick-add all refuse it through their existing unknown-platform paths. - NATIVE_INGESTER_PLATFORMS: pixiv removed. - extension_service: pixiv's quick-add URL pattern removed (the Python half of the JS mirror). - extension: pixiv's host permissions, content-script match, platform entry and artist pattern removed; popup's pixiv branches removed; and the whole pixiv PKCE OAuth flow cut out of background.js. That last one could not wait for phase 2 - a webRequest listener on a host the manifest no longer grants is at best dead and at worst a startup failure for the entire background script. On startup the extension now also removes any pixiv refresh token a browser still holds in storage, for the same reason as the server-side credential cleanup (3980). - frontend: the extension card stops listing pixiv; SourceActions' copy of the native list drops it. platformColor keeps rendering a pixiv key so existing pixiv posts do not look broken. The guard, and why a registry change alone was not enough. A source outlives its platform: the live instance still had one ENABLED pixiv source (step 1). Tracing it: the scheduler only selects enabled rows and every platform lookup uses .get(), so a disabled row is inert - but re-enabling it and pressing Check would have routed pixiv, no longer native, straight into the gallery-dl branch, which still has a pixiv extractor. And a worker can pick up a still-enabled row before a deploy's migration runs. So run_download and verify_source_credential - the two functions every download and credential probe pass through - now refuse any platform not in the registry: an unsupported_url failure for downloads, and an inconclusive (None, not False) verify, since nothing was probed so nothing was rejected. Generic by registration, so it covers deviantart's leftovers too. Positive-controlled: a supported gallery-dl platform must still reach gallery-dl, or a guard that refused everything would pass (rule 167). Migration 0097 disables sources on retired platforms (pixiv, deviantart) and clears their failure state exactly as disabling through the app does (1285), so the stale row stops being scheduled and stops showing as failing. Nothing is deleted: removing a source can collide with uq_post_artist_external_id_null_source on real data, which is phase 2's step 6 to check. No post or image is touched. Tests: the known-platform lists drop pixiv and gain retirement assertions beside deviantart's; pixiv's positive extension cases become negative guards; the pixiv sidecar post-URL test is deleted with the behaviour it tested; quick-add rejects a pixiv URL. The pixiv client/downloader/ingester suites stay - that code stays until phase 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
This commit is contained in:
@@ -129,8 +129,6 @@ async def test_resolve_artist_name_dispatches_per_platform(db, monkeypatch):
|
||||
("https://www.subscribestar.com/foobar", "subscribestar", "foobar"),
|
||||
("https://subscribestar.adult/foobar", "subscribestar", "foobar"),
|
||||
("https://www.hentai-foundry.com/user/Foo/profile", "hentaifoundry", "Foo"),
|
||||
("https://www.pixiv.net/users/12345", "pixiv", "12345"),
|
||||
("https://www.pixiv.net/en/users/12345", "pixiv", "12345"),
|
||||
])
|
||||
@pytest.mark.asyncio
|
||||
async def test_quick_add_source_url_patterns(client, ext_key, url, platform, slug):
|
||||
@@ -176,6 +174,22 @@ async def test_quick_add_source_rejects_retired_deviantart(client, ext_key):
|
||||
assert "deviantart" not in body["known"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quick_add_source_rejects_retired_pixiv(client, ext_key):
|
||||
"""Milestone #406: the same shape as deviantart's retirement above. An
|
||||
un-updated extension can still offer the button on a pixiv creator page, so
|
||||
the backend refuses rather than creating a source nothing can download."""
|
||||
resp = await client.post(
|
||||
"/api/extension/quick-add-source",
|
||||
json={"url": "https://www.pixiv.net/users/12345"},
|
||||
headers={"X-Extension-Key": ext_key},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
body = await resp.get_json()
|
||||
assert body["error"] == "unknown_platform"
|
||||
assert "pixiv" not in body["known"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_quick_add_source_invalid_url_400(client, ext_key):
|
||||
resp = await client.post(
|
||||
|
||||
@@ -6,17 +6,17 @@ pytestmark = pytest.mark.integration
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_platforms_returns_gs_five(client):
|
||||
async def test_platforms_returns_the_supported_four(client):
|
||||
resp = await client.get("/api/platforms")
|
||||
assert resp.status_code == 200
|
||||
body = await resp.get_json()
|
||||
platforms = body["platforms"]
|
||||
assert set(platforms.keys()) == {
|
||||
"patreon", "subscribestar", "hentaifoundry",
|
||||
"discord", "pixiv",
|
||||
"patreon", "subscribestar", "hentaifoundry", "discord",
|
||||
}
|
||||
assert "fanbox" not in platforms
|
||||
assert "deviantart" not in platforms # retired at #3069
|
||||
assert "pixiv" not in platforms # retired at milestone #406
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -37,5 +37,4 @@ async def test_platforms_record_shape(client):
|
||||
async def test_platform_auth_types_match_gs(client):
|
||||
body = await (await client.get("/api/platforms")).get_json()
|
||||
assert body["platforms"]["discord"]["auth_type"] == "token"
|
||||
assert body["platforms"]["pixiv"]["auth_type"] == "token"
|
||||
assert body["platforms"]["patreon"]["auth_type"] == "cookies"
|
||||
|
||||
@@ -1,21 +1,107 @@
|
||||
"""download_backends — the single predicate that routes a platform to the
|
||||
native ingester vs. gallery-dl. Pure, no DB."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.app.services.download_backends import (
|
||||
NATIVE_INGESTER_PLATFORMS,
|
||||
_campaign_resolution_error,
|
||||
_native_ingester_cls,
|
||||
_unsupported_platform_message,
|
||||
run_download,
|
||||
uses_native_ingester,
|
||||
verify_source_credential,
|
||||
)
|
||||
from backend.app.services.gallery_dl import ErrorType
|
||||
from backend.app.services.pixiv_ingester import PixivIngester
|
||||
|
||||
|
||||
def test_native_platforms():
|
||||
for platform in ("patreon", "subscribestar", "pixiv"):
|
||||
for platform in ("patreon", "subscribestar"):
|
||||
assert uses_native_ingester(platform) is True
|
||||
assert platform in NATIVE_INGESTER_PLATFORMS
|
||||
|
||||
|
||||
def test_pixiv_is_no_longer_native():
|
||||
"""Retired at milestone #406. The refusal below is what stops it falling
|
||||
through to gallery-dl now that it is not native."""
|
||||
assert uses_native_ingester("pixiv") is False
|
||||
assert "pixiv" not in NATIVE_INGESTER_PLATFORMS
|
||||
|
||||
|
||||
# --- the retired-platform guard (#406 phase 1) -----------------------------
|
||||
|
||||
|
||||
class _RecordingGalleryDL:
|
||||
"""Stands in for GalleryDLService: records whether a download was attempted."""
|
||||
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
async def download(self, **kwargs):
|
||||
self.calls.append(kwargs["platform"])
|
||||
return "reached gallery-dl"
|
||||
|
||||
|
||||
def _ctx(platform):
|
||||
return {
|
||||
"platform": platform, "url": f"https://example.invalid/{platform}",
|
||||
"artist_slug": "someone", "cookies_path": None, "auth_token": None,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_retired_platform_never_reaches_a_downloader():
|
||||
"""An enabled source on a retired platform is data that survives a deploy.
|
||||
Unguarded, pixiv — no longer native — would fall straight through to the
|
||||
gallery-dl branch, which still has a pixiv extractor."""
|
||||
gdl = _RecordingGalleryDL()
|
||||
result, campaign_id = await run_download(
|
||||
ctx=_ctx("pixiv"), source_config=None, skip_value=False, mode=None,
|
||||
gdl=gdl, sync_session_factory=None,
|
||||
)
|
||||
assert gdl.calls == []
|
||||
assert result.success is False
|
||||
assert result.error_type == ErrorType.UNSUPPORTED_URL
|
||||
assert "pixiv" in result.error_message
|
||||
assert campaign_id is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_supported_gallery_dl_platform_still_reaches_gallery_dl():
|
||||
"""The positive control. Without it, a guard that refused EVERY platform
|
||||
would pass the test above just as well (rule #167)."""
|
||||
gdl = _RecordingGalleryDL()
|
||||
result, _ = await run_download(
|
||||
ctx=_ctx("hentaifoundry"), source_config=None, skip_value=False, mode=None,
|
||||
gdl=gdl, sync_session_factory=None,
|
||||
)
|
||||
assert gdl.calls == ["hentaifoundry"]
|
||||
assert result == "reached gallery-dl"
|
||||
|
||||
|
||||
def test_the_guard_discriminates_by_registration():
|
||||
assert _unsupported_platform_message("hentaifoundry") is None
|
||||
assert _unsupported_platform_message("patreon") is None
|
||||
assert _unsupported_platform_message("pixiv") is not None
|
||||
assert _unsupported_platform_message("deviantart") is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_verifying_a_retired_platform_is_inconclusive_not_rejected():
|
||||
"""Nothing is probed, so nothing is rejected — returning False would tell the
|
||||
operator their credential is bad when the platform is simply gone."""
|
||||
ok, message = await verify_source_credential(
|
||||
platform="pixiv", url="https://www.pixiv.net/users/1", artist_slug="someone",
|
||||
config_overrides=None, cookies_path=None, auth_token=None,
|
||||
images_root=Path("/nonexistent"),
|
||||
)
|
||||
assert ok is None
|
||||
assert "pixiv" in message
|
||||
|
||||
|
||||
def test_gallery_dl_platforms_are_not_native():
|
||||
# The platforms still served by gallery-dl must NOT route to the native
|
||||
# ingester — guards an accidental over-broad migration.
|
||||
|
||||
@@ -11,13 +11,21 @@ from backend.app.services.platforms import (
|
||||
)
|
||||
|
||||
|
||||
def test_known_platform_keys_is_gs_five():
|
||||
def test_known_platform_keys_are_the_supported_four():
|
||||
# GS's original five, less pixiv (retired at milestone #406, rule #171).
|
||||
assert known_platform_keys() == frozenset({
|
||||
"patreon", "subscribestar", "hentaifoundry",
|
||||
"discord", "pixiv",
|
||||
"patreon", "subscribestar", "hentaifoundry", "discord",
|
||||
})
|
||||
|
||||
|
||||
def test_pixiv_is_retired():
|
||||
# Milestone #406 phase 1. Unregistering is what switches pixiv off: the
|
||||
# registry feeds /api/platforms, the source validator and the download
|
||||
# guard, so this one absence is load-bearing everywhere else.
|
||||
assert "pixiv" not in PLATFORMS
|
||||
assert "pixiv" not in known_platform_keys()
|
||||
|
||||
|
||||
def test_fanbox_not_in_registry():
|
||||
# Sanity check — FC-3a added 'fanbox' by mistake; it's not a GS platform.
|
||||
assert "fanbox" not in PLATFORMS
|
||||
|
||||
@@ -183,19 +183,6 @@ def test_parse_subscribestar_post_url_derived_and_post_id_wins():
|
||||
assert sd.post_url == "https://www.subscribestar.com/posts/360360"
|
||||
|
||||
|
||||
def test_parse_pixiv_post_url_derived():
|
||||
"""Pixiv's `url` is the image URL (i.pximg.net); must be replaced
|
||||
with the post permalink under /artworks/<id>."""
|
||||
sd = parse_sidecar({
|
||||
"category": "pixiv",
|
||||
"id": 140466853,
|
||||
"url": "https://i.pximg.net/img-original/img/2026/01/28/10/28/24/140466853_p0.jpg",
|
||||
"title": "Nerissa x Jailbird",
|
||||
})
|
||||
assert sd.external_post_id == "140466853"
|
||||
assert sd.post_url == "https://www.pixiv.net/artworks/140466853"
|
||||
|
||||
|
||||
def test_parse_hentaifoundry_post_url_derived():
|
||||
"""HF sidecars omit `url` entirely and use `index`+`user` for the
|
||||
post's natural key. Synthesize the canonical /pictures/user/<u>/<i>
|
||||
|
||||
@@ -23,14 +23,15 @@ async def _artist(db, name="Alice"):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_known_platforms_is_gs_five(db):
|
||||
async def test_known_platforms_are_the_supported_four(db):
|
||||
assert KNOWN_PLATFORMS == frozenset({
|
||||
"patreon", "subscribestar", "hentaifoundry",
|
||||
"discord", "pixiv",
|
||||
"patreon", "subscribestar", "hentaifoundry", "discord",
|
||||
})
|
||||
assert "fanbox" not in KNOWN_PLATFORMS
|
||||
# Retired at #3069 — a source can no longer be created on it.
|
||||
assert "deviantart" not in KNOWN_PLATFORMS
|
||||
# Retired at milestone #406 — likewise.
|
||||
assert "pixiv" not in KNOWN_PLATFORMS
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user