feat(ingest): recapture body + links for every walked post (Phase 5)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m14s

Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.

Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.

Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.

Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 16:14:40 -04:00
parent 8dbf29f803
commit 5e1655384f
2 changed files with 55 additions and 22 deletions
+23 -17
View File
@@ -227,25 +227,31 @@ class Ingester:
# resume_cursor None, so everything counts.
if not (resume_cursor and page_cursor == resume_cursor):
chunk_new_posts += 1
# Capture the post body + external links ONCE per post (gated by
# the synthetic post key in the seen-ledger), for EVERY post —
# whether or not it has downloadable media. This is what makes a
# backfill/recovery re-walk RECAPTURE bodies + links for posts
# whose media is already on disk: re-downloading existing media
# never fills links the system never had, so the body recapture
# has to ride the walk itself. Detail-fetch (for an empty feed
# body) happens at most once per post — the gate then spares it on
# later walks. bypass_seen (recovery) re-captures unconditionally.
if post_record_key and write_post_record:
rk = post_record_key(post)
if rk is not None:
pkey, ppid = rk
already = (
set() if bypass_seen
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
rec_path = write_post_record(post, artist_slug)
if rec_path is not None:
post_records.append(str(rec_path))
self._mark_seen(source_id, [(pkey, ppid)])
media = self.client.extract_media(post, included)
if not media:
# No downloadable media — still capture the post itself
# (title/body/external links) so text-only posts aren't lost.
# Gate via the seen-ledger (synthetic post key) so a text post
# is detail-fetched + recorded ONCE, not re-walked every tick.
if post_record_key and write_post_record:
rk = post_record_key(post)
if rk is not None:
pkey, ppid = rk
already = (
set() if bypass_seen
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
rec_path = write_post_record(post, artist_slug)
if rec_path is not None:
post_records.append(str(rec_path))
self._mark_seen(source_id, [(pkey, ppid)])
continue
keys = [ledger_key(m) for m in media]