bvandeusen
f678819093
Phase 1, step 1 of moving SubscribeStar off gallery-dl onto the native core ingester (milestone: SubscribeStar native). Mirror of the Patreon ledger: SubscribeStarSeenMedia (skip already-ingested media on routine walks; recovery bypasses) and SubscribeStarFailedMedia (dead-letter so persistently-failing media stops re-burning backfill chunks). Per operator decision, dedicated per-platform tables (not a generalized shared ledger). filehash is String(128): a CDN content hash when the URL carries one, else a synthesized <post_id>:<filename> key. UNIQUE (source_id, filehash) upsert key. Registered in models/__init__; migration 0054 creates both tables (down 0053). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
45 lines
1.8 KiB
Python
45 lines
1.8 KiB
Python
"""SubscribeStarFailedMedia — per-source dead-letter ledger of SubscribeStar
|
|
media that keeps failing to download/validate.
|
|
|
|
Mirror of PatreonFailedMedia. Media that fails every walk (404'd CDN URL,
|
|
deleted post, persistently-corrupt bytes) would otherwise re-error forever and
|
|
re-burn backfill chunks. After ``attempts`` reaches the dead-letter threshold
|
|
the ingester skips it on routine tick/backfill walks (recovery still
|
|
re-attempts). A later clean download clears the row.
|
|
|
|
`filehash` is the same per-media key the seen-ledger uses (CDN content hash or a
|
|
synthesized ``<post_id>:<filename>`` key) — hence String(128). UNIQUE
|
|
(source_id, filehash) is the upsert key.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
from sqlalchemy.types import DateTime
|
|
|
|
from .base import Base
|
|
|
|
|
|
class SubscribeStarFailedMedia(Base):
|
|
__tablename__ = "subscribestar_failed_media"
|
|
__table_args__ = (
|
|
UniqueConstraint(
|
|
"source_id", "filehash", name="uq_subscribestar_failed_media_source_id"
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
source_id: Mapped[int] = mapped_column(
|
|
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
)
|
|
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
|
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
first_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
last_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|