feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)
If Patreon renames/restructures the post body field again (as content → content_json_string already did), every body silently comes back empty and we'd archive empty posts without noticing. Surface that as a loud failure. Research-grounded design (Patreon `content` is officially null|string, body has no post_type gate, gallery-dl independently added the same content_json_string fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction threshold would false-positive constantly. The robust, creator-independent break signature is "a meaningful sample of posts, a body extracted from NONE of them." - ingest_core counts posts_recorded / posts_with_body on the native post-record path (gallery-dl never enters it, so the canary is native-only by construction). - When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are literally "fix the field-set/parser, not creds"). Placed after the timeout/stop returns so it never masks a more specific failure. - Run summary always appends "bodies X/Y" for sub-threshold observability (a partial regression that still extracts some bodies shows in the Raw stdout). Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success; below the sample floor -> success (tick safety). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -57,6 +57,19 @@ _ERROR_MAX = 1000
|
||||
# page-tied update would lurch). Trivial churn (~one single-row UPDATE / 5s).
|
||||
_LIVE_PROGRESS_INTERVAL = 5.0
|
||||
|
||||
# Post-body schema-drift canary (#862). Patreon's body lives in
|
||||
# content/content_json_string with NO post_type gate, so a field rename (as
|
||||
# content→content_json_string already was) zeroes EVERY body at once — across
|
||||
# every artist, every walk. If a native walk records at least this many posts
|
||||
# and extracts a body from NONE of them, treat it as that break (fail the run
|
||||
# API_DRIFT) rather than silently archiving empties. A *fraction* threshold would
|
||||
# false-positive on gallery/art creators who legitimately post images with no
|
||||
# caption, so the gate is "zero across a minimum sample": a real creator nearly
|
||||
# always has SOME text across this many posts, a broken parser has none. Set high
|
||||
# enough that a small tick (a few new posts) can't trip it — only a backfill /
|
||||
# recapture (the operator's schema-test flow) reaches the sample.
|
||||
_CANARY_MIN_SAMPLE = 30
|
||||
|
||||
|
||||
class Ingester:
|
||||
"""Generic native-ingest orchestration. Subclass with a platform adapter
|
||||
@@ -149,6 +162,12 @@ class Ingester:
|
||||
dead_lettered = 0
|
||||
skipped_count = 0
|
||||
posts_processed = 0
|
||||
# Post-body schema-drift canary counters (#862, native ingester only —
|
||||
# gallery-dl walks never enter the post-record block below so these stay 0
|
||||
# and the canary can't fire there). posts_recorded = post-records attempted
|
||||
# this walk; posts_with_body = how many yielded a non-empty body.
|
||||
posts_recorded = 0
|
||||
posts_with_body = 0
|
||||
# Net-new posts THIS chunk for the live progress badge (plan #704 #5);
|
||||
# excludes the re-walked resume page so _backfill_posts stays a monotonic
|
||||
# absolute across chunks instead of an inflating sum. posts_processed
|
||||
@@ -261,6 +280,9 @@ class Ingester:
|
||||
)
|
||||
if pkey not in already:
|
||||
rec = write_post_record(post, artist_slug)
|
||||
posts_recorded += 1
|
||||
if rec.body_chars:
|
||||
posts_with_body += 1
|
||||
if rec.path is not None:
|
||||
post_records.append(str(rec.path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
@@ -414,6 +436,10 @@ class Ingester:
|
||||
f"{dead_lettered} dead-lettered, {errors} error(s), "
|
||||
f"{posts_processed} post(s), {len(post_records)} post-record(s), "
|
||||
f"{len(relink)} relinked"
|
||||
# Body-capture health (#862): even below the canary's red-alarm
|
||||
# threshold, surfacing the ratio makes a partial extraction regression
|
||||
# visible in the Raw stdout (e.g. "bodies 3/180" reads as off).
|
||||
+ (f", bodies {posts_with_body}/{posts_recorded}" if posts_recorded else "")
|
||||
+ (", reached end" if reached_bottom else "")
|
||||
+ (", time-boxed" if budget_hit else "")
|
||||
)
|
||||
@@ -439,6 +465,27 @@ class Ingester:
|
||||
error_message="Chunk timed out with no progress",
|
||||
)
|
||||
|
||||
# Post-body schema-drift canary (#862): a native walk recorded a
|
||||
# meaningful sample of posts but extracted a body from NONE of them. Since
|
||||
# the body field has no post_type gate, that's the signature of Patreon
|
||||
# renaming/restructuring the body field (as content→content_json_string
|
||||
# already was) — fail RED (API_DRIFT: "fix is the field-set/parser, not
|
||||
# creds") so the breakage screams instead of silently archiving empties.
|
||||
# Only reached on an otherwise-clean walk (timeout/stop/error returned
|
||||
# above), so it never masks a more specific failure.
|
||||
if posts_recorded >= _CANARY_MIN_SAMPLE and posts_with_body == 0:
|
||||
msg = (
|
||||
f"Post-body canary: extracted a body from 0 of {posts_recorded} "
|
||||
"posts — Patreon's body field shape likely changed; the ingester "
|
||||
"needs a field-set/parser update."
|
||||
)
|
||||
log_lines.append(msg)
|
||||
log.error("%s (artist=%s)", msg, artist_slug)
|
||||
return _result(
|
||||
success=False, return_code=-1,
|
||||
error_type=ErrorType.API_DRIFT, error_message=msg,
|
||||
)
|
||||
|
||||
# Normal success: reached the bottom, or a tick that early-outed. rc 0 +
|
||||
# error_type None is REQUIRED for a backfill/recovery walk that reached
|
||||
# the bottom to be marked COMPLETE by
|
||||
|
||||
Reference in New Issue
Block a user