bvandeusen
7a872a3619
A media that fails every walk (404'd CDN, deleted post, geo-blocked Mux, persistently-corrupt bytes) used to re-error forever and re-burn chunks. New `patreon_failed_media` table (alembic 0038, chains 0037) records per-media attempts; once attempts reach DEAD_LETTER_THRESHOLD (3) the ingester skips it on routine tick/backfill walks (tier-1.5, folded into the seen/skip predicate). Recovery BYPASSES it (the operator's "try everything again" re-attempts dead media). A clean download clears the row (recovered); errors/quarantines upsert-increment it. Surfaced as run_stats.dead_lettered_count. - New PatreonFailedMedia model + migration; ingester _dead_keys / _record_failures (on_conflict increment) / _clear_failures. - skip = seen | dead (empty in recovery); failures recorded post-fetch on short sessions (same pattern as the seen-ledger). Tests: a media erroring 3× is dead-lettered + skipped (no download attempt); recovery re-attempts a dead media and clears it on success; a clean download clears a sub-threshold failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
59 lines
1.8 KiB
Python
59 lines
1.8 KiB
Python
"""patreon_failed_media: per-source dead-letter ledger for failing Patreon media
|
|
|
|
Revision ID: 0038
|
|
Revises: 0037
|
|
Create Date: 2026-06-06
|
|
|
|
Plan #705 (#7). Media that keeps failing to download/validate (404'd CDN,
|
|
deleted post, geo-blocked Mux, persistently-corrupt bytes) gets recorded here
|
|
with an attempt counter; once it crosses the dead-letter threshold the ingester
|
|
skips it on routine walks (recovery still re-attempts). A clean download clears
|
|
the row. UNIQUE (source_id, filehash) is the upsert key (same media key the
|
|
seen-ledger uses).
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0038"
|
|
down_revision: Union[str, None] = "0037"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.create_table(
|
|
"patreon_failed_media",
|
|
sa.Column("id", sa.Integer, primary_key=True),
|
|
sa.Column(
|
|
"source_id",
|
|
sa.Integer,
|
|
sa.ForeignKey("source.id", ondelete="CASCADE"),
|
|
nullable=False,
|
|
index=True,
|
|
),
|
|
sa.Column("filehash", sa.String(128), nullable=False),
|
|
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
|
|
sa.Column("last_error", sa.Text, nullable=True),
|
|
sa.Column(
|
|
"first_failed_at",
|
|
sa.DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=sa.text("NOW()"),
|
|
),
|
|
sa.Column(
|
|
"last_failed_at",
|
|
sa.DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=sa.text("NOW()"),
|
|
),
|
|
sa.UniqueConstraint(
|
|
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
|
|
),
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_table("patreon_failed_media")
|