refactor: the native downloaders share one download loop, file record and seen-ledger key; a clean walk marks its credential verified
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Failing after 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 23s
CI and images / frontend-build (push) Successful in 26s
CI and images / backend-lint-and-test (push) Successful in 53s
CI and images / integration (push) Successful in 2m50s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
Patreon, SubscribeStar and Discord each carried their own copy of the same per-post download loop, file record and seen-ledger key, and every copy was another place a fix could be missed. - BaseNativeDownloader runs one post's files: the seen and on-disk skips, the mid-post time-box, pacing, validation and the per-file sidecar. Each platform overrides only where its layout differs: Discord's channel folder, file names and message-id sidecar, and Patreon's Mux video through yt-dlp. - One MediaItem and one ledger_key in native_ingest_common replace the three per-client copies. Keys are unchanged, except that a SubscribeStar file with no gallery id is now keyed by its filename instead of an empty id. - DownloadResult, ErrorType and the run-stats and tier-gated helpers move from gallery_dl.py, which now serves only HentaiFoundry, to download_result.py. The unused delegator methods on GalleryDLService go. - The download lock's platform list is the native-ingester list, not a copy. - A walk that used a credential and finished with no error type stamps that credential's last-verified time, so the credentials page shows when the account last worked. A tier-limited walk does not count: an expired session hides paid posts the same way. - The extension no longer clears a retired pixiv token on startup. - Two comments that still described every download as gallery-dl now name both backends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,30 +22,18 @@ from pathlib import Path
|
||||
|
||||
from ..models import SubscribeStarFailedMedia, SubscribeStarSeenMedia
|
||||
from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
|
||||
from .subscribestar_client import MediaItem, SubscribeStarAPIError, SubscribeStarClient
|
||||
from .native_ingest_common import ledger_key
|
||||
from .subscribestar_client import SubscribeStarAPIError, SubscribeStarClient
|
||||
from .subscribestar_downloader import SubscribeStarDownloader
|
||||
|
||||
__all__ = [
|
||||
"DEAD_LETTER_THRESHOLD",
|
||||
"SubscribeStarIngester",
|
||||
"_ledger_key",
|
||||
"verify_subscribestar_credential",
|
||||
]
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_LEDGER_KEY_MAX = 128
|
||||
|
||||
|
||||
def _ledger_key(media: MediaItem) -> str:
|
||||
"""Stable per-media identity for the cross-run seen-ledger. SubscribeStar's
|
||||
full-res URL is an opaque `/post_uploads?payload=...` (no content hash), so
|
||||
`media.filehash` is normally None and the stable proxy is the gallery item id
|
||||
scoped to its post: `<post_id>:<media_id>`. Bounded to the column width."""
|
||||
if media.filehash:
|
||||
return media.filehash
|
||||
return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
|
||||
|
||||
|
||||
class SubscribeStarIngester(Ingester):
|
||||
"""Walk a SubscribeStar creator's posts, download unseen media, return a
|
||||
@@ -91,7 +79,7 @@ class SubscribeStarIngester(Ingester):
|
||||
failed_model=SubscribeStarFailedMedia,
|
||||
seen_constraint="uq_subscribestar_seen_media_source_id",
|
||||
failed_constraint="uq_subscribestar_failed_media_source_id",
|
||||
ledger_key=_ledger_key,
|
||||
ledger_key=ledger_key,
|
||||
platform="subscribestar",
|
||||
error_base=SubscribeStarAPIError,
|
||||
# API_DRIFT message phrasing; the base Ingester._failure_result owns
|
||||
|
||||
Reference in New Issue
Block a user