feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s
CI / lint (push) Successful in 4s

A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 20:58:40 -04:00
parent 96c29c370b
commit 65ec29ba9b
18 changed files with 467 additions and 29 deletions
+41
View File
@@ -267,6 +267,47 @@ async def test_start_recovery_arms_bypass_flag(db):
assert "_backfill_bypass_seen" not in (co2 or {})
@pytest.mark.asyncio
async def test_start_recapture_arms_recapture_flag(db):
"""#830: start_recapture arms the backfill state machine PLUS the
_backfill_recapture flag, surfaced on the record so the badge can label it
'Recapturing'. Mutually exclusive with recovery (clears bypass_seen); stop
clears it."""
from backend.app.services.source_service import BACKFILL_MAX_CHUNKS
artist = await _artist(db, "Alice")
svc = SourceService(db)
rec = await svc.create(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice",
)
# Pre-arm recovery, then recapture must clear bypass_seen (mutual exclusion).
await svc.start_recovery(rec.id)
updated = await svc.start_recapture(rec.id)
assert updated.backfill_state == "running"
assert updated.backfill_recapture is True
assert updated.backfill_bypass_seen is False
assert updated.backfill_runs_remaining == BACKFILL_MAX_CHUNKS
co = (await db.execute(
select(Source.config_overrides).where(Source.id == rec.id)
)).scalar_one()
assert co.get("_backfill_recapture") is True
assert "_backfill_bypass_seen" not in co
# to_dict carries the new field for the API/UI.
assert updated.to_dict()["backfill_recapture"] is True
# Stop clears the recapture flag too.
stopped = await svc.stop_backfill(rec.id)
assert stopped.backfill_recapture is False
co2 = (await db.execute(
select(Source.config_overrides).where(Source.id == rec.id)
)).scalar_one()
assert "_backfill_recapture" not in (co2 or {})
@pytest.mark.asyncio
async def test_start_backfill_raises_when_source_missing(db):
svc = SourceService(db)