feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)
CI / integration (push) Successful in 3m17s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 28s

If Patreon renames/restructures the post body field again (as content →
content_json_string already did), every body silently comes back empty and we'd
archive empty posts without noticing. Surface that as a loud failure.

Research-grounded design (Patreon `content` is officially null|string, body has
no post_type gate, gallery-dl independently added the same content_json_string
fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction
threshold would false-positive constantly. The robust, creator-independent break
signature is "a meaningful sample of posts, a body extracted from NONE of them."

- ingest_core counts posts_recorded / posts_with_body on the native post-record
  path (gallery-dl never enters it, so the canary is native-only by construction).
- When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return
  ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are
  literally "fix the field-set/parser, not creds"). Placed after the timeout/stop
  returns so it never masks a more specific failure.
- Run summary always appends "bodies X/Y" for sub-threshold observability (a
  partial regression that still extracts some bodies shows in the Raw stdout).

Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success;
below the sample floor -> success (tick safety).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 08:56:11 -04:00
parent 949c9abcc6
commit 00607a309b
2 changed files with 111 additions and 2 deletions
+47
View File
@@ -57,6 +57,19 @@ _ERROR_MAX = 1000
# page-tied update would lurch). Trivial churn (~one single-row UPDATE / 5s).
_LIVE_PROGRESS_INTERVAL = 5.0
# Post-body schema-drift canary (#862). Patreon's body lives in
# content/content_json_string with NO post_type gate, so a field rename (as
# content→content_json_string already was) zeroes EVERY body at once — across
# every artist, every walk. If a native walk records at least this many posts
# and extracts a body from NONE of them, treat it as that break (fail the run
# API_DRIFT) rather than silently archiving empties. A *fraction* threshold would
# false-positive on gallery/art creators who legitimately post images with no
# caption, so the gate is "zero across a minimum sample": a real creator nearly
# always has SOME text across this many posts, a broken parser has none. Set high
# enough that a small tick (a few new posts) can't trip it — only a backfill /
# recapture (the operator's schema-test flow) reaches the sample.
_CANARY_MIN_SAMPLE = 30
class Ingester:
"""Generic native-ingest orchestration. Subclass with a platform adapter
@@ -149,6 +162,12 @@ class Ingester:
dead_lettered = 0
skipped_count = 0
posts_processed = 0
# Post-body schema-drift canary counters (#862, native ingester only —
# gallery-dl walks never enter the post-record block below so these stay 0
# and the canary can't fire there). posts_recorded = post-records attempted
# this walk; posts_with_body = how many yielded a non-empty body.
posts_recorded = 0
posts_with_body = 0
# Net-new posts THIS chunk for the live progress badge (plan #704 #5);
# excludes the re-walked resume page so _backfill_posts stays a monotonic
# absolute across chunks instead of an inflating sum. posts_processed
@@ -261,6 +280,9 @@ class Ingester:
)
if pkey not in already:
rec = write_post_record(post, artist_slug)
posts_recorded += 1
if rec.body_chars:
posts_with_body += 1
if rec.path is not None:
post_records.append(str(rec.path))
self._mark_seen(source_id, [(pkey, ppid)])
@@ -414,6 +436,10 @@ class Ingester:
f"{dead_lettered} dead-lettered, {errors} error(s), "
f"{posts_processed} post(s), {len(post_records)} post-record(s), "
f"{len(relink)} relinked"
# Body-capture health (#862): even below the canary's red-alarm
# threshold, surfacing the ratio makes a partial extraction regression
# visible in the Raw stdout (e.g. "bodies 3/180" reads as off).
+ (f", bodies {posts_with_body}/{posts_recorded}" if posts_recorded else "")
+ (", reached end" if reached_bottom else "")
+ (", time-boxed" if budget_hit else "")
)
@@ -439,6 +465,27 @@ class Ingester:
error_message="Chunk timed out with no progress",
)
# Post-body schema-drift canary (#862): a native walk recorded a
# meaningful sample of posts but extracted a body from NONE of them. Since
# the body field has no post_type gate, that's the signature of Patreon
# renaming/restructuring the body field (as content→content_json_string
# already was) — fail RED (API_DRIFT: "fix is the field-set/parser, not
# creds") so the breakage screams instead of silently archiving empties.
# Only reached on an otherwise-clean walk (timeout/stop/error returned
# above), so it never masks a more specific failure.
if posts_recorded >= _CANARY_MIN_SAMPLE and posts_with_body == 0:
msg = (
f"Post-body canary: extracted a body from 0 of {posts_recorded} "
"posts — Patreon's body field shape likely changed; the ingester "
"needs a field-set/parser update."
)
log_lines.append(msg)
log.error("%s (artist=%s)", msg, artist_slug)
return _result(
success=False, return_code=-1,
error_type=ErrorType.API_DRIFT, error_message=msg,
)
# Normal success: reached the bottom, or a tick that early-outed. rc 0 +
# error_type None is REQUIRED for a backfill/recovery walk that reached
# the bottom to be marked COMPLETE by