bvandeusen
7a872a3619
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux, persistently-corrupt bytes) used to re-error forever and re-burn chunks. New `patreon_failed_media` table (alembic 0038, chains 0037) records per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the ingester skips it on routine tick/backfill walks (tier-1.5, folded into the seen/skip predicate). Recovery BYPASSES it (the operator's "try everything again" re-attempts dead media). A clean download clears the row (recovered); errors/quarantines upsert-increment it. Surfaced as run_stats.dead_lettered_count. - New PatreonFailedMedia model + migration; ingester _dead_keys / _record_failures (on_conflict increment) / _clear_failures. - skip = seen | dead (empty in recovery); failures recorded post-fetch on short sessions (same pattern as the seen-ledger). Tests: a media erroring 3× is dead-lettered + skipped (no download attempt); recovery re-attempts a dead media and clears it on success; a clean download clears a sub-threshold failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
46 lines
1.8 KiB
Python
46 lines
1.8 KiB
Python
"""PatreonFailedMedia — per-source dead-letter ledger of Patreon media that
|
|
keeps failing to download/validate.
|
|
|
|
Plan #705 (#7). A media that fails every walk (404'd CDN URL, deleted post,
|
|
geo-blocked Mux stream, persistently-corrupt bytes) would otherwise re-error
|
|
forever and re-burn backfill chunks. After ``attempts`` reaches the dead-letter
|
|
threshold the ingester skips it on routine tick/backfill walks (recovery still
|
|
re-attempts it — the operator's "try everything again"). A later clean download
|
|
clears the row (the media recovered).
|
|
|
|
`filehash` is the same per-media key the seen-ledger uses (32-hex CDN MD5, or a
|
|
``video:`` / ``post:filename`` synthesized key) — hence String(128). UNIQUE
|
|
(source_id, filehash) is the upsert key.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
from sqlalchemy.types import DateTime
|
|
|
|
from .base import Base
|
|
|
|
|
|
class PatreonFailedMedia(Base):
|
|
__tablename__ = "patreon_failed_media"
|
|
__table_args__ = (
|
|
UniqueConstraint(
|
|
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
source_id: Mapped[int] = mapped_column(
|
|
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
)
|
|
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
|
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
first_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
last_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|