bvandeusen
f678819093
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
41 lines
1.6 KiB
Python
41 lines
1.6 KiB
Python
"""SubscribeStarSeenMedia — per-source ledger of SubscribeStar media already
|
|
downloaded+processed.
|
|
|
|
Mirror of PatreonSeenMedia for the SubscribeStar native ingester (replacing
|
|
gallery-dl). One queryable row per (source, media) so routine walks skip media
|
|
we've already ingested; recovery mode bypasses the ledger to re-walk.
|
|
|
|
`filehash` is a CDN content hash when the media URL carries one, else a
|
|
synthesized ``<post_id>:<filename>`` key (SubscribeStar URLs aren't always
|
|
content-addressed) — hence String(128) rather than 32.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
from sqlalchemy.types import DateTime
|
|
|
|
from .base import Base
|
|
|
|
|
|
class SubscribeStarSeenMedia(Base):
|
|
__tablename__ = "subscribestar_seen_media"
|
|
__table_args__ = (
|
|
# Dedup key the downloader upserts against: one ledger row per
|
|
# (source, media). A second sighting of the same media is a no-op.
|
|
UniqueConstraint(
|
|
"source_id", "filehash", name="uq_subscribestar_seen_media_source_id"
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
source_id: Mapped[int] = mapped_column(
|
|
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
)
|
|
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
|
seen_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|