Files
FabledCurator/backend/app/services/subscribestar_ingester.py
T
bvandeusenandClaude Opus 5.5 da98195ada
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
refactor: the native downloaders share one download loop, file record and seen-ledger key; a clean walk marks its credential verified
Patreon, SubscribeStar and Discord each carried their own copy of the same
per-post download loop, file record and seen-ledger key, and every copy was
another place a fix could be missed.

- BaseNativeDownloader runs one post's files: the seen and on-disk skips,
  the mid-post time-box, pacing, validation and the per-file sidecar. Each
  platform overrides only where its layout differs: Discord's channel folder,
  file names and message-id sidecar, and Patreon's Mux video through yt-dlp.
- One MediaItem and one ledger_key in native_ingest_common replace the three
  per-client copies. Keys are unchanged, except that a SubscribeStar file with
  no gallery id is now keyed by its filename instead of an empty id.
- DownloadResult, ErrorType and the run-stats and tier-gated helpers move
  from gallery_dl.py, which now serves only HentaiFoundry, to
  download_result.py. The unused delegator methods on GalleryDLService go.
- The download lock's platform list is the native-ingester list, not a copy.
- A walk that used a credential and finished with no error type stamps that
  credential's last-verified time, so the credentials page shows when the
  account last worked. A tier-limited walk does not count: an expired session
  hides paid posts the same way.
- The extension no longer clears a retired pixiv token on startup.
- Two comments that still described every download as gallery-dl now name
  both backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:03:32 -05:00

103 lines
4.0 KiB
Python

"""Native SubscribeStar ingester — the SubscribeStar ADAPTER over the
platform-agnostic core (`ingest_core.Ingester`).
Thin counterpart to patreon_ingester: wires the SubscribeStar client/downloader/
ledger models/constraints/key into the core and supplies the SubscribeStar
failure mapping. The three modes (tick / backfill / recovery / recapture), the
seen + dead-letter ledgers, cursor checkpointing, and the post-first capture all
live in the core — identical to Patreon. `download_service.download_source`
drives `SubscribeStarIngester.run` exactly as it drives the Patreon one.
`campaign_id` is the creator URL (the client derives host + slug from it), so no
campaign-id resolver is needed. FC runs on a plain-HTTP homelab; nothing here
uses a secure-context Web API.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Callable
from pathlib import Path
from ..models import SubscribeStarFailedMedia, SubscribeStarSeenMedia
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
from .native_ingest_common import ledger_key
from .subscribestar_client import SubscribeStarAPIError, SubscribeStarClient
from .subscribestar_downloader import SubscribeStarDownloader
__all__ = [
"DEAD_LETTER_THRESHOLD",
"SubscribeStarIngester",
"verify_subscribestar_credential",
]
log = logging.getLogger(__name__)
class SubscribeStarIngester(Ingester):
"""Walk a SubscribeStar creator's posts, download unseen media, return a
`DownloadResult`. A thin adapter over `ingest_core.Ingester`; `client` /
`downloader` are injectable seams so unit tests run without network."""
def __init__(
self,
images_root: Path,
cookies_path: str | None,
session_factory: Callable[[], object],
*,
validate: bool = True,
rate_limit: float = 0.0,
request_sleep: float = 0.0,
auth_token: str | None = None,
client: SubscribeStarClient | None = None,
downloader: SubscribeStarDownloader | None = None,
):
# auth_token: accepted for the uniform native-ingester construction
# (download_backends passes it to every adapter); SubscribeStar
# authenticates by cookies, so it's unused here.
del auth_token
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
resolved_client = (
client
if client is not None
else SubscribeStarClient(cookies_path, request_sleep=request_sleep)
)
resolved_downloader = (
downloader
if downloader is not None
else SubscribeStarDownloader(
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit,
)
)
super().__init__(
client=resolved_client,
downloader=resolved_downloader,
session_factory=session_factory,
seen_model=SubscribeStarSeenMedia,
failed_model=SubscribeStarFailedMedia,
seen_constraint="uq_subscribestar_seen_media_source_id",
failed_constraint="uq_subscribestar_failed_media_source_id",
ledger_key=ledger_key,
platform="subscribestar",
error_base=SubscribeStarAPIError,
# API_DRIFT message phrasing; the base Ingester._failure_result owns
# the auth/drift/HTTP→error_type mapping (shared across platforms).
drift_label="SubscribeStar markup",
)
async def verify_subscribestar_credential(
url: str,
cookies_path: str | None,
overrides: dict | None,
) -> tuple[bool | None, str]:
"""Native SubscribeStar credential probe — fetches ONE feed page via
SubscribeStarClient.verify_auth. `campaign_id` is just the creator URL (no
resolver). Returns the uniform `(ok, message)` contract so
download_backends.verify_credential treats it like the gallery-dl probe."""
client = SubscribeStarClient(cookies_path)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, client.verify_auth, url)