bvandeusen
abafc3265e
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."
**Layout**
services/platforms/
base.py PlatformInfo dataclass + module-default key
chains + shared helpers (str_id_value, str_field)
__init__.py PLATFORMS dict + public API (auth_type_for,
known_platform_keys, to_dict,
external_post_id_keys_for, description_keys_for)
patreon.py metadata only — the reference platform, no quirks
subscribestar.py metadata + augment_cookies (18+ agreement) +
derive_post_url (synthetic /posts/<post_id>)
hentaifoundry.py metadata + augment_cookies (host-only PHPSESSID
duplicate) + derive_post_url (/pictures/user/...)
pixiv.py metadata + derive_post_url (/artworks/<id>)
discord.py metadata + derive_post_url
(channels/<server>/<channel>/<message>)
deviantart.py metadata only — un-audited; quirks to be added
when an operator first exercises DA
**PlatformInfo extensions**
Existing fields preserved. Four new optional fields:
external_post_id_keys: tuple[str, ...] | None
Override the sidecar external_post_id lookup chain. None falls
back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
("post_id", "id", "index", "message_id") — covers every current
platform.
description_keys: tuple[str, ...] | None
Override the description body lookup chain. None falls back to
DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
"message") — Discord's "message" body field is covered by the
default's trailing entry.
derive_post_url: Callable[[dict], str | None] | None
Synthesize the post permalink from sidecar metadata. None = trust
the bare `url` / `post_url` field (patreon, deviantart).
subscribestar/pixiv/hf/discord override this because their `url`
is the file CDN URL.
augment_cookies: Callable[[str], str] | None
Post-process the materialized cookies.txt before gallery-dl
consumes it. None = no-op. Used by subscribestar (age cookie) and
hentaifoundry (host-only PHPSESSID duplicate).
**Consumer changes**
- credential_service._augment_cookies(platform, netscape) shrunk from a
per-platform-conditional dispatcher (~80 lines of inlined helpers) to
a 5-line lookup: `info.augment_cookies(netscape) if info and
info.augment_cookies else netscape`. The platform-specific helper
bodies moved verbatim into the per-platform modules.
- sidecar.parse_sidecar similarly delegates: external_post_id chain via
external_post_id_keys_for(category), description chain via
description_keys_for(category), post_url via
PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
and inline _derive_post_url body both gone. Added a shared `_first_id`
helper for bool-safe id coercion.
**Public API preserved**
PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.
**Adding a new platform**
1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
pick it up automatically.
109 lines
3.8 KiB
Python
109 lines
3.8 KiB
Python
"""PlatformInfo dataclass + shared defaults + small helpers.
|
|
|
|
Per-platform modules import from here, register their PlatformInfo via
|
|
INFO, optionally attaching `derive_post_url` and/or `augment_cookies`
|
|
callables for behavior that diverges from gallery-dl's mainline shape
|
|
(Patreon).
|
|
|
|
Adding a new platform: drop a new module under `services/platforms/`,
|
|
declare an INFO, and add it to the import list in
|
|
`services/platforms/__init__.py`. Sidecar parsing, cookie
|
|
materialization, and the /api/platforms response pick it up
|
|
automatically.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Callable
|
|
from dataclasses import dataclass
|
|
from typing import Literal
|
|
|
|
|
|
# Sidecar parsing defaults. Per-platform PlatformInfo entries can
|
|
# override these by setting `external_post_id_keys=` /
|
|
# `description_keys=`. Most don't need to — the defaults already cover
|
|
# every platform FC supports.
|
|
#
|
|
# external_post_id chain: `post_id` MUST come before `id` because
|
|
# SubscribeStar gallery-dl puts the per-attachment id in `id` and the
|
|
# actual post id in `post_id`; picking `id` first fragments
|
|
# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
|
|
# no `post_id` so `id` still wins for them; HF uses `index`, Discord
|
|
# uses `message_id` — all reached via the remaining chain entries.
|
|
# (Banked 2026-05-27 during the sidecar audit.)
|
|
DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
|
|
"post_id", "id", "index", "message_id",
|
|
)
|
|
|
|
# Description body chain: Discord's gallery-dl extractor uses `message`
|
|
# (no `content`); appended to the chain so Discord posts surface body
|
|
# text.
|
|
DEFAULT_DESCRIPTION_KEYS: tuple[str, ...] = (
|
|
"content", "description", "caption", "message",
|
|
)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class PlatformInfo:
|
|
# --- Identity / metadata ---
|
|
key: str
|
|
name: str
|
|
description: str
|
|
auth_type: Literal["cookies", "token"]
|
|
requires_auth: bool
|
|
url_pattern: str
|
|
url_examples: list[str]
|
|
default_config: dict
|
|
notes: str | None = None
|
|
|
|
# --- Sidecar parsing overrides ---
|
|
# Each is None to mean "use the module default above"; a platform
|
|
# only sets one of these when its sidecar shape genuinely differs.
|
|
external_post_id_keys: tuple[str, ...] | None = None
|
|
description_keys: tuple[str, ...] | None = None
|
|
|
|
# --- Behavioral hooks ---
|
|
# Synthesize a post permalink from sidecar data. Required when
|
|
# gallery-dl's `url` field is the file/CDN URL rather than the post
|
|
# permalink (subscribestar/pixiv/hf/discord). None = trust the bare
|
|
# `url` field (patreon, deviantart).
|
|
derive_post_url: Callable[[dict], str | None] | None = None
|
|
|
|
# Post-process the materialized cookies.txt for gallery-dl. Used by
|
|
# platforms whose server gates or extractor quirks need synthetic
|
|
# cookies the extension can't capture (subscribestar age cookie, HF
|
|
# host-only PHPSESSID duplicate). None = no-op.
|
|
augment_cookies: Callable[[str], str] | None = None
|
|
|
|
|
|
def str_id_value(v) -> str | None:
|
|
"""Coerce a JSON scalar id into a non-empty string, rejecting bool
|
|
(Python's bool is an int subclass so `isinstance(True, int)` is
|
|
True; without this guard a sidecar with `"id": true` would produce
|
|
external_post_id="True")."""
|
|
if isinstance(v, bool):
|
|
return None
|
|
if isinstance(v, (str, int)) and str(v).strip():
|
|
return str(v).strip()
|
|
return None
|
|
|
|
|
|
def str_field(v) -> str | None:
|
|
"""Same idea as str_id_value but for plain string fields (no int
|
|
coercion)."""
|
|
if isinstance(v, str) and v.strip():
|
|
return v.strip()
|
|
return None
|
|
|
|
|
|
# Shared gallery-dl invocation defaults. Embedded in each platform's
|
|
# default_config (with platform-specific overrides) so per-platform
|
|
# choices stay explicit.
|
|
GD_DEFAULTS = {
|
|
"sleep": 3.0,
|
|
"sleep_request": 1.5,
|
|
"skip_existing": True,
|
|
"save_metadata": True,
|
|
"timeout": 3600,
|
|
}
|