bvandeusen
0563b2d750
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira — pximg URLs carry no content hash). PixivIngester wires client/downloader/ ledgers into ingest_core with drift label 'Pixiv app API' and the new body_canary=False opt-out: caption-less pixiv artists are common, so the zero-bodies #862 alarm would false-positive here — the client's response-shape drift checks cover that failure class instead. auth_token joins the uniform adapter constructor (pixiv is the first token-auth native platform). verify_pixiv_credential = one OAuth refresh, no feed walk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
46 lines
1.7 KiB
Python
46 lines
1.7 KiB
Python
"""PixivFailedMedia — per-source dead-letter ledger of Pixiv media that keeps
|
|
failing to download/validate.
|
|
|
|
Mirror of PatreonFailedMedia/SubscribeStarFailedMedia. Media that fails every
|
|
walk (404'd pximg URL, deleted work, persistently-corrupt bytes) would
|
|
otherwise re-error forever and re-burn backfill chunks. After ``attempts``
|
|
reaches the dead-letter threshold the ingester skips it on routine
|
|
tick/backfill walks (recovery still re-attempts). A later clean download
|
|
clears the row.
|
|
|
|
`filehash` is the same synthesized ``<illust_id>:p<num>`` /
|
|
``<illust_id>:ugoira`` key the seen-ledger uses. UNIQUE (source_id, filehash)
|
|
is the upsert key.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
from sqlalchemy.types import DateTime
|
|
|
|
from .base import Base
|
|
|
|
|
|
class PixivFailedMedia(Base):
|
|
__tablename__ = "pixiv_failed_media"
|
|
__table_args__ = (
|
|
UniqueConstraint(
|
|
"source_id", "filehash", name="uq_pixiv_failed_media_source_id"
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
source_id: Mapped[int] = mapped_column(
|
|
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
|
)
|
|
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
|
|
last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
first_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
last_failed_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|