feat(pixiv): ledger models + migration 0076 + PixivIngester adapter (#129 step 3)
pixiv_seen_media / pixiv_failed_media mirror the Patreon/SubscribeStar ledgers (keys are always synthesized <illust_id>:p<num> / <illust_id>:ugoira — pximg URLs carry no content hash). PixivIngester wires client/downloader/ ledgers into ingest_core with drift label 'Pixiv app API' and the new body_canary=False opt-out: caption-less pixiv artists are common, so the zero-bodies #862 alarm would false-positive here — the client's response-shape drift checks cover that failure class instead. auth_token joins the uniform adapter constructor (pixiv is the first token-auth native platform). verify_pixiv_credential = one OAuth refresh, no feed walk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
@@ -90,6 +90,7 @@ class Ingester:
|
||||
platform: str,
|
||||
error_base: type[Exception],
|
||||
drift_label: str | None = None,
|
||||
body_canary: bool = True,
|
||||
):
|
||||
self.client = client
|
||||
self.downloader = downloader
|
||||
@@ -105,6 +106,12 @@ class Ingester:
|
||||
# update"). Defaults to the platform name; adapters pass a richer phrase
|
||||
# (e.g. "Patreon API", "SubscribeStar markup").
|
||||
self._drift_label = drift_label or platform
|
||||
# #862 canary opt-out: platforms whose posts legitimately have empty
|
||||
# bodies across large samples (pixiv — caption-less artists are common)
|
||||
# would false-positive the zero-bodies-means-drift alarm; their clients
|
||||
# catch drift structurally (response-shape checks) instead. The
|
||||
# "bodies X/N" summary line still surfaces the ratio either way.
|
||||
self._body_canary = body_canary
|
||||
|
||||
# -- public ------------------------------------------------------------
|
||||
|
||||
@@ -536,7 +543,11 @@ class Ingester:
|
||||
# creds") so the breakage screams instead of silently archiving empties.
|
||||
# Only reached on an otherwise-clean walk (timeout/stop/error returned
|
||||
# above), so it never masks a more specific failure.
|
||||
if posts_recorded >= _CANARY_MIN_SAMPLE and posts_with_body == 0:
|
||||
if (
|
||||
self._body_canary
|
||||
and posts_recorded >= _CANARY_MIN_SAMPLE
|
||||
and posts_with_body == 0
|
||||
):
|
||||
msg = (
|
||||
f"Post-body canary: extracted a body from 0 of {posts_recorded} "
|
||||
"posts — Patreon's body field shape likely changed; the ingester "
|
||||
|
||||
Reference in New Issue
Block a user