Nested-archive extraction (#718) + post-first ingest (#67) + post-body canary (#862) #109

Merged

bvandeusen merged 6 commits from dev into main 2026-06-15 21:37:39 -04:00

Conversation 0 Commits 6 Files Changed 8

+457 -92

bvandeusen commented 2026-06-15 21:37:34 -04:00

Owner

Copy Link

Copy Source

Merges the dev stack (all CI-green, tip b48ba60) to main / :latest.

• #718 — archives showing "no images" fixed: _import_archive now recurses into nested archives (depth-capped, bomb-probed per level) so a per-chapter .rar/.zip wrapped in a "High Resolution files" pack is extracted; and _import_media enrich-on-duplicate resolves the artist from the path so deduped/cross-posted archive members link to their post (the post-first minimal-sidecar gap). Precise per-outcome reason string; benign all-deduped no longer false-flags.
• #67 — post-first ingest: on the native path the post-record is the sole body/links writer; per-media sidecar is minimal {category,id,source_url}. Migration contract documented (rule #120), self-enforcing via uses_native_ingester.
• #862 — post-body schema-drift canary: a native walk that extracts a body from zero of ≥30 posts fails red (API_DRIFT); run summary shows bodies X/Y.

No new migrations (head stays 0051). No new runtime deps. After merge: force a real re-pull of :latest, then run Settings → Maintenance → re-extract archives to retroactively fix the ~48 existing nested-archive posts.

🤖 Generated with Claude Code

Merges the dev stack (all CI-green, tip `b48ba60`) to main / `:latest`. - **#718** — archives showing "no images" fixed: `_import_archive` now **recurses into nested archives** (depth-capped, bomb-probed per level) so a per-chapter `.rar`/`.zip` wrapped in a "High Resolution files" pack is extracted; and `_import_media` enrich-on-duplicate resolves the artist **from the path** so deduped/cross-posted archive members link to their post (the post-first minimal-sidecar gap). Precise per-outcome reason string; benign all-deduped no longer false-flags. - **#67** — post-first ingest: on the native path the post-record is the sole body/links writer; per-media sidecar is minimal `{category,id,source_url}`. Migration contract documented (rule #120), self-enforcing via `uses_native_ingester`. - **#862** — post-body schema-drift canary: a native walk that extracts a body from zero of ≥30 posts fails red (`API_DRIFT`); run summary shows `bodies X/Y`. No new migrations (head stays 0051). No new runtime deps. **After merge:** force a real re-pull of `:latest`, then run **Settings → Maintenance → re-extract archives** to retroactively fix the ~48 existing nested-archive posts. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 6 commits 2026-06-15 21:37:34 -04:00

feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862) ... 00607a309b

If Patreon renames/restructures the post body field again (as content →
content_json_string already did), every body silently comes back empty and we'd
archive empty posts without noticing. Surface that as a loud failure.

Research-grounded design (Patreon `content` is officially null|string, body has
no post_type gate, gallery-dl independently added the same content_json_string
fallback): empty bodies are LEGITIMATE for gallery/art posts, so a fraction
threshold would false-positive constantly. The robust, creator-independent break
signature is "a meaningful sample of posts, a body extracted from NONE of them."

- ingest_core counts posts_recorded / posts_with_body on the native post-record
  path (gallery-dl never enters it, so the canary is native-only by construction).
- When posts_recorded >= _CANARY_MIN_SAMPLE (30) and posts_with_body == 0, return
  ErrorType.API_DRIFT (maps to task_run status "error" — red; its semantics are
  literally "fix the field-set/parser, not creds"). Placed after the timeout/stop
  returns so it never masks a more specific failure.
- Run summary always appends "bodies X/Y" for sub-threshold observability (a
  partial regression that still extracts some bodies shows in the Raw stdout).

Tests: zero bodies over the sample -> API_DRIFT; bodies present -> success;
below the sample floor -> success (tick safety).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(ingest): post-first — post-record is the sole body writer on the native path (#856) ... dcbc3ae335

Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.

Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).

- patreon_downloader: the per-media sidecar is now minimal — {category, id,
  source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
  resolution + detail-fetch (the post-record, written first in the walk, already
  did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
  `_apply_sidecar` still writes source_filehash + provenance + primary_post_id
  but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
  so applying a body-less sidecar would clobber raw_metadata + re-sync links off
  empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
  uses_native_ingester(platform) — the future-proof seam, so a platform migrating
  onto the native core flips to post-first automatically (step 3). Media imports
  before post-records but both unify on external_post_id, so the post ends with
  its body either way.

Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test(downloader): fix test_sidecar_written_and_findable for post-first minimal sidecar (#856) ... 8b99dc9b81

The per-media sidecar no longer carries title/url/content (post-first, #856) —
update the assertion to expect image identity only (category/id/source_url).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs(ingest): document the post-first migration contract at the native seam (#857) ... 41aa8fe39e

Milestone #67 step 3. Spell out, at the IngestCore.run seam resolution, that
post_record_key + write_post_record are the post-first contract a platform
implements when migrating onto the native core ingester — the post-record owns
the body/links, the per-media sidecar carries image identity only. The import
side is already self-enforcing via uses_native_ingester → importer.post_first.
Durable directive recorded as FC project rule #120.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(import): recurse nested archives + precise "no images" reason (#718) ... 8dee2f9628

Root cause (operator-confirmed via event metadata + lsar): a "High Resolution
files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase).
_import_archive only extracted one level — a nested-archive member failed
is_supported and was skipped, so the real pages were silently dropped and the post
showed "archive but no images". The disk scan found this pattern recurring across
the attachment store.

- Recurse into nested archives via _collect_archive_members: a member that is
  itself an archive is bomb-probed and extracted too, depth-capped at
  _ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so
  they link to the right Post. Each level is wrapped so one bad nested archive
  can't abort the import. The shared path means external (mega/gdrive) archives
  recurse too.
- Replace the catch-all "held no supported members" string with a per-outcome
  tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped
  case is now recognised as BENIGN — images already in the library, re-linked to
  this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no
  longer false-flags in event metadata.unextracted_archives. Genuine failures
  carry the precise breakdown.

Tests: nested zip-in-cbz imports both inner images + links them to the outer post;
all-deduped archive returns attached with error=None and links images to both posts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(import): resolve artist from path for enrich-on-duplicate (#718) ... b48ba60830

The dedup branches of _import_media linked the existing image to the new post via
_apply_sidecar(artist=None), relying on the SIDECAR to carry the artist. But an
archive member's artist comes from its path, and under post-first the per-media
sidecar is minimal (no artist) — so a re-packed / cross-posted archive image
deduped and was left UNLINKED from the new post, i.e. the post showed "no images".

Resolve the path-anchored artist (derive_top_level_artist) up-front in
_import_media and pass it to both enrich-on-duplicate branches (sha256 + phash
larger_exists) and the new-record path. Drop the now-dead _attach_artist helper
(its logic is inlined at the single new-record call site).

Surfaced by the new test_archive_all_deduped_is_benign_not_flagged (was asserting
2==4: the second post got no provenance links).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit cffdd93418 into main 2026-06-15 21:37:39 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#109