bvandeusen/FabledCurator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 33 Wiki Activity

refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation

Browse Source 
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."

**Layout**

  services/platforms/
    base.py            PlatformInfo dataclass + module-default key
                       chains + shared helpers (str_id_value, str_field)
    __init__.py        PLATFORMS dict + public API (auth_type_for,
                       known_platform_keys, to_dict,
                       external_post_id_keys_for, description_keys_for)
    patreon.py         metadata only — the reference platform, no quirks
    subscribestar.py   metadata + augment_cookies (18+ agreement) +
                       derive_post_url (synthetic /posts/<post_id>)
    hentaifoundry.py   metadata + augment_cookies (host-only PHPSESSID
                       duplicate) + derive_post_url (/pictures/user/...)
    pixiv.py           metadata + derive_post_url (/artworks/<id>)
    discord.py         metadata + derive_post_url
                       (channels/<server>/<channel>/<message>)
    deviantart.py      metadata only — un-audited; quirks to be added
                       when an operator first exercises DA

**PlatformInfo extensions**

Existing fields preserved. Four new optional fields:

  external_post_id_keys: tuple[str, ...] | None
      Override the sidecar external_post_id lookup chain. None falls
      back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
      ("post_id", "id", "index", "message_id") — covers every current
      platform.

  description_keys: tuple[str, ...] | None
      Override the description body lookup chain. None falls back to
      DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
      "message") — Discord's "message" body field is covered by the
      default's trailing entry.

  derive_post_url: Callable[[dict], str | None] | None
      Synthesize the post permalink from sidecar metadata. None = trust
      the bare `url` / `post_url` field (patreon, deviantart).
      subscribestar/pixiv/hf/discord override this because their `url`
      is the file CDN URL.

  augment_cookies: Callable[[str], str] | None
      Post-process the materialized cookies.txt before gallery-dl
      consumes it. None = no-op. Used by subscribestar (age cookie) and
      hentaifoundry (host-only PHPSESSID duplicate).

**Consumer changes**

- credential_service._augment_cookies(platform, netscape) shrunk from a
  per-platform-conditional dispatcher (~80 lines of inlined helpers) to
  a 5-line lookup: `info.augment_cookies(netscape) if info and
  info.augment_cookies else netscape`. The platform-specific helper
  bodies moved verbatim into the per-platform modules.

- sidecar.parse_sidecar similarly delegates: external_post_id chain via
  external_post_id_keys_for(category), description chain via
  description_keys_for(category), post_url via
  PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
  and inline _derive_post_url body both gone. Added a shared `_first_id`
  helper for bool-safe id coercion.

**Public API preserved**

PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.

**Adding a new platform**

1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
   and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
   tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
   pick it up automatically.
This commit is contained in:
Bryan Van Deusen
2026-05-27 19:46:05 -04:00
parent 2394e47370
commit abafc3265e
11 changed files with 512 additions and 305 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/utils/sidecar.py
+36 -73
View File
@@ -1,7 +1,9 @@
"""Minimal gallery-dl sidecar parsing (one-time filesystem-import aid).
No per-platform branching: a small common key set with fallbacks; the
full JSON is kept in raw so anything unmapped is recoverable later.
Per-platform quirks (post_url synthesis, key-chain overrides) live in
the platforms registry — `backend/app/services/platforms/`. This module
is platform-agnostic: it looks up `category` in the sidecar and asks
the registry for the right behavior.
"""
import re
@@ -9,6 +11,12 @@ from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from ..services.platforms import (
PLATFORMS,
description_keys_for,
external_post_id_keys_for,
)
@dataclass(frozen=True)
class SidecarData:
@@ -55,6 +63,19 @@ def _first_str(data: dict, keys: tuple[str, ...]) -> str | None:
return None
def _first_id(data: dict, keys: tuple[str, ...]) -> str | None:
"""Like `_first_str` but accepts ints and rejects bool (Python's
bool subclasses int, so a literal `"id": true` would otherwise
yield external_post_id="True")."""
for k in keys:
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
# Strip HTML tags + collapse whitespace + take the first non-empty line.
# Used to derive a display title from a body when the platform doesn't
# expose a separate title field (subscribestar posts always write
@@ -111,22 +132,7 @@ def parse_sidecar(data: dict) -> SidecarData:
cat = data.get("category")
platform = cat if isinstance(cat, str) and cat.strip() else None
# external_post_id lookup order: post_id MUST come before id.
# SubscribeStar gallery-dl writes the per-attachment id in `id`
# (e.g. 711509) and the actual post id in `post_id` (e.g. 360360);
# picking `id` first fragments every multi-image subscribestar post
# into N distinct Post rows in FC. Patreon/Pixiv have no `post_id`
# so `id` still wins for them; HF uses `index`, Discord uses
# `message_id` — all reached via the remaining chain entries.
# Operator-flagged 2026-05-27 during the sidecar audit.
external_post_id = None
for k in ("post_id", "id", "index", "message_id"):
v = data.get(k)
if isinstance(v, bool):
continue
if isinstance(v, (str, int)) and str(v).strip():
external_post_id = str(v)
break
external_post_id = _first_id(data, external_post_id_keys_for(platform))
pc = data.get("page_count")
if isinstance(pc, bool):
@@ -146,30 +152,23 @@ def parse_sidecar(data: dict) -> SidecarData:
if post_date is not None:
break
# `message` is Discord gallery-dl's body field (no `content`); added
# 2026-05-27 to the description fallback chain.
description = _first_str(
data, ("content", "description", "caption", "message"),
)
description = _first_str(data, description_keys_for(platform))
# SubscribeStar posts always write `title: ""` and put the leading
# sentence inside `content` (confirmed against the operator's
# /mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27). When
# no explicit title is present, synthesize one from the description
# body's first non-empty line. Patreon retains its explicit titles
# because they're non-empty and short-circuit the fallback.
# When `title` is empty (subscribestar always; sometimes elsewhere),
# synthesize from the description body's first non-empty text line.
# Patreon's explicit titles short-circuit the fallback.
post_title = _first_str(data, ("title",))
if post_title is None and description:
post_title = _first_line_text(description)
# post_url derivation: SubscribeStar/Pixiv/HF/Discord put the FILE
# download URL in `url`, not a post permalink. Synthesize the
# permalink from per-platform fields when possible. Patreon's `url`
# IS a permalink and is used as-is. For the four file-URL platforms,
# the bare `url` is NEVER trusted — derive or return None rather
# than persist a CDN URL in post.post_url.
if platform in _DERIVED_URL_PLATFORMS:
post_url = _derive_post_url(platform, data)
# post_url: ask the platform module to synthesize a permalink.
# When the platform registers a `derive_post_url`, it owns the
# field (the bare `url`/`post_url` value is a file CDN URL and
# must NEVER be persisted). When it doesn't register one, trust
# the sidecar's `url` (Patreon's case — real permalink).
info = PLATFORMS.get(platform) if platform else None
if info is not None and info.derive_post_url is not None:
post_url = info.derive_post_url(data)
else:
post_url = _first_str(data, ("url", "post_url"))
@@ -183,39 +182,3 @@ def parse_sidecar(data: dict) -> SidecarData:
post_date=post_date,
raw=data,
)
_DERIVED_URL_PLATFORMS = frozenset({
"subscribestar", "pixiv", "hentaifoundry", "discord",
})
def _derive_post_url(platform: str, data: dict) -> str | None:
"""Synthesize the post-permalink URL from per-platform metadata.
gallery-dl writes the file-download URL in `url` for these four
platforms; we need a real permalink for the PostCard "open original"
button. Returns None if the platform-specific fields are missing
(rare in well-formed sidecars but defensive).
"""
if platform == "subscribestar":
pid = data.get("post_id")
if isinstance(pid, (str, int)) and str(pid).strip():
return f"https://www.subscribestar.com/posts/{pid}"
elif platform == "pixiv":
pid = data.get("id")
if isinstance(pid, (str, int)) and not isinstance(pid, bool) and str(pid).strip():
return f"https://www.pixiv.net/artworks/{pid}"
elif platform == "hentaifoundry":
user = _first_str(data, ("user", "artist"))
idx = data.get("index")
if user and isinstance(idx, (str, int)) and not isinstance(idx, bool) and str(idx).strip():
return f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
elif platform == "discord":
sid = data.get("server_id")
cid = data.get("channel_id")
mid = data.get("message_id")
if all(isinstance(v, (str, int)) and not isinstance(v, bool) and str(v).strip()
for v in (sid, cid, mid)):
return f"https://discord.com/channels/{sid}/{cid}/{mid}"
return None