feat(ingest): recapture body + links for every walked post (Phase 5)
Operator reframed backfill as inherent to the existing walk: you can't fill links the system never had by re-downloading media that's already on disk, so the body/link recapture has to ride the walk itself. Hoist the post-record capture out of the media-less branch so it runs for EVERY post — gated once per post by the synthetic post key in the seen-ledger (detail-fetch for an empty feed body happens at most once; recovery re-captures unconditionally). A normal BACKFILL now walks history and recaptures each post's body + external links (which phase 3 imports via upsert_post_record → _sync_external_links → the download sweep, all already wired). A tick captures new posts going forward. No separate button — the backfill is the backfill. Tests: media posts now also carry a synthetic post-key ledger row (count assertions +1); new test proves an already-on-disk media post still recaptures its body/links on a re-walk. Completes the core of #830 (Phase 5). Phase 2 (inline-image localization) remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -227,25 +227,31 @@ class Ingester:
|
||||
# resume_cursor None, so everything counts.
|
||||
if not (resume_cursor and page_cursor == resume_cursor):
|
||||
chunk_new_posts += 1
|
||||
# Capture the post body + external links ONCE per post (gated by
|
||||
# the synthetic post key in the seen-ledger), for EVERY post —
|
||||
# whether or not it has downloadable media. This is what makes a
|
||||
# backfill/recovery re-walk RECAPTURE bodies + links for posts
|
||||
# whose media is already on disk: re-downloading existing media
|
||||
# never fills links the system never had, so the body recapture
|
||||
# has to ride the walk itself. Detail-fetch (for an empty feed
|
||||
# body) happens at most once per post — the gate then spares it on
|
||||
# later walks. bypass_seen (recovery) re-captures unconditionally.
|
||||
if post_record_key and write_post_record:
|
||||
rk = post_record_key(post)
|
||||
if rk is not None:
|
||||
pkey, ppid = rk
|
||||
already = (
|
||||
set() if bypass_seen
|
||||
else self._seen_keys(source_id, [pkey])
|
||||
)
|
||||
if pkey not in already:
|
||||
rec_path = write_post_record(post, artist_slug)
|
||||
if rec_path is not None:
|
||||
post_records.append(str(rec_path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
|
||||
media = self.client.extract_media(post, included)
|
||||
if not media:
|
||||
# No downloadable media — still capture the post itself
|
||||
# (title/body/external links) so text-only posts aren't lost.
|
||||
# Gate via the seen-ledger (synthetic post key) so a text post
|
||||
# is detail-fetched + recorded ONCE, not re-walked every tick.
|
||||
if post_record_key and write_post_record:
|
||||
rk = post_record_key(post)
|
||||
if rk is not None:
|
||||
pkey, ppid = rk
|
||||
already = (
|
||||
set() if bypass_seen
|
||||
else self._seen_keys(source_id, [pkey])
|
||||
)
|
||||
if pkey not in already:
|
||||
rec_path = write_post_record(post, artist_slug)
|
||||
if rec_path is not None:
|
||||
post_records.append(str(rec_path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
continue
|
||||
|
||||
keys = [ledger_key(m) for m in media]
|
||||
|
||||
Reference in New Issue
Block a user