feat(subscribestar): seen/failed ledger models + migration 0054 (#889)
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
"""SubscribeStarSeenMedia — per-source ledger of SubscribeStar media already
|
||||
downloaded+processed.
|
||||
|
||||
Mirror of PatreonSeenMedia for the SubscribeStar native ingester (replacing
|
||||
gallery-dl). One queryable row per (source, media) so routine walks skip media
|
||||
we've already ingested; recovery mode bypasses the ledger to re-walk.
|
||||
|
||||
`filehash` is a CDN content hash when the media URL carries one, else a
|
||||
synthesized ``<post_id>:<filename>`` key (SubscribeStar URLs aren't always
|
||||
content-addressed) — hence String(128) rather than 32.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
from sqlalchemy.types import DateTime
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class SubscribeStarSeenMedia(Base):
|
||||
__tablename__ = "subscribestar_seen_media"
|
||||
__table_args__ = (
|
||||
# Dedup key the downloader upserts against: one ledger row per
|
||||
# (source, media). A second sighting of the same media is a no-op.
|
||||
UniqueConstraint(
|
||||
"source_id", "filehash", name="uq_subscribestar_seen_media_source_id"
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
source_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
seen_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
Reference in New Issue
Block a user