feat(ingest): recapture body + links for every walked post (Phase 5)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m14s

Operator reframed backfill as inherent to the existing walk: you can't fill
links the system never had by re-downloading media that's already on disk, so
the body/link recapture has to ride the walk itself.

Hoist the post-record capture out of the media-less branch so it runs for EVERY
post — gated once per post by the synthetic post key in the seen-ledger
(detail-fetch for an empty feed body happens at most once; recovery re-captures
unconditionally). A normal BACKFILL now walks history and recaptures each post's
body + external links (which phase 3 imports via upsert_post_record →
_sync_external_links → the download sweep, all already wired). A tick captures
new posts going forward. No separate button — the backfill is the backfill.

Tests: media posts now also carry a synthetic post-key ledger row (count
assertions +1); new test proves an already-on-disk media post still recaptures
its body/links on a re-walk.

Completes the core of #830 (Phase 5). Phase 2 (inline-image localization)
remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 16:14:40 -04:00
parent 8dbf29f803
commit 5e1655384f
2 changed files with 55 additions and 22 deletions
+32 -5
View File
@@ -173,7 +173,10 @@ async def test_tick_downloads_unseen_and_marks_seen(source_id, sync_engine, tmp_
# plan #704: structured run_stats carry the real counts.
assert result.run_stats["downloaded_count"] == 2
assert result.posts_processed == 1
assert _count_ledger(sync_engine, source_id) == 2
# 2 media keys + 1 synthetic post key (body/links recaptured per post).
assert _count_ledger(sync_engine, source_id) == 3
# The post body + links are captured for media posts too (rides the walk).
assert len(result.post_record_paths) == 1
@pytest.mark.asyncio
@@ -196,8 +199,9 @@ async def test_quarantined_media_surfaced_in_result(source_id, sync_engine, tmp_
assert result.run_stats["quarantined_count"] == 1
assert result.run_stats["downloaded_count"] == 1
assert len(result.written_paths) == 1 # quarantined NOT written
# Quarantined media is NOT marked seen (a fixed file may be re-fetched).
assert _count_ledger(sync_engine, source_id) == 1
# Quarantined media is NOT marked seen (a fixed file may be re-fetched);
# m1 + the synthetic post key (body/links captured per post) = 2.
assert _count_ledger(sync_engine, source_id) == 2
@pytest.mark.asyncio
@@ -550,8 +554,31 @@ async def test_recovery_tier2_disk_still_skips(source_id, sync_engine, tmp_path)
)
assert result.files_downloaded == 0
assert downloader.download_calls == 0
# Disk-skip still reconciles the ledger so a later tick skips at tier-1.
assert _count_ledger(sync_engine, source_id) == 1
# Disk-skip reconciles the media key + the synthetic post key (recovery
# recaptures the body/links per post) = 2.
assert _count_ledger(sync_engine, source_id) == 2
@pytest.mark.asyncio
async def test_backfill_recaptures_body_for_already_downloaded_post(
source_id, sync_engine, tmp_path,
):
"""Phase 5 guarantee: a post whose media is ALREADY on disk still gets its
body + external links recaptured on a re-walk. Re-downloading existing media
can never fill links the system never had — so the recapture rides the walk
itself, and a normal backfill is the backfill."""
m1 = _media("p1", 1)
client = _FakeClient([(None, [("p1", [m1])])])
downloader = _FakeDownloader(tmp_path, on_disk={_ledger_key(m1)})
ing = _ingester(sync_engine, tmp_path, client, downloader)
result = ing.run(
source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="backfill",
)
assert result.files_downloaded == 0 # media already on disk
assert len(result.post_record_paths) == 1 # body/links recaptured anyway
assert downloader.post_records == 1
# --- dead-letter ledger (plan #705 #7) ------------------------------------