feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s
CI / lint (push) Successful in 4s

A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 20:58:40 -04:00
parent 96c29c370b
commit 65ec29ba9b
18 changed files with 467 additions and 29 deletions
+36 -1
View File
@@ -70,6 +70,10 @@ class SourceRecord:
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
backfill_bypass_seen: bool
# #830: a running deep-walk in RECAPTURE mode (re-grab post bodies/links +
# localize on-disk inline images, no media re-download). Lets the badge label
# it "Recapturing".
backfill_recapture: bool
# plan #704: cumulative posts processed across the walk's chunks — live
# progress for the badge.
backfill_posts: int
@@ -94,6 +98,7 @@ class SourceRecord:
"backfill_state": self.backfill_state,
"backfill_chunks": self.backfill_chunks,
"backfill_bypass_seen": self.backfill_bypass_seen,
"backfill_recapture": self.backfill_recapture,
"backfill_posts": self.backfill_posts,
}
@@ -171,6 +176,7 @@ class SourceService:
backfill_state=co.get("_backfill_state"),
backfill_chunks=int(co.get("_backfill_chunks", 0)),
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
backfill_recapture=bool(co.get("_backfill_recapture")),
backfill_posts=int(co.get("_backfill_posts", 0)),
)
@@ -359,6 +365,34 @@ class SourceService:
await self.session.commit()
return await self._row_to_record(source)
async def start_recapture(self, source_id: int) -> SourceRecord:
"""#830: arm a RECAPTURE walk — a backfill that re-grabs EVERY post's body
+ external links (detail-fetching empty bodies) and localizes already-on-
disk inline images, WITHOUT re-downloading media. Reuses the entire #693
backfill state machine plus a `_backfill_recapture` flag that flips
download mode to recapture. Distinct from recovery (which re-downloads the
whole source); the two flags are mutually exclusive, so arming recapture
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
fresh from the top. The flag is cleared on completion (download_service)
and on stop. Recapture is Patreon-only (the native ingester's post-record
capture); inert elsewhere. The UI gates the action to Patreon sources."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
co["_backfill_state"] = "running"
co["_backfill_recapture"] = True
co.pop("_backfill_bypass_seen", None) # mutually exclusive with recovery
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
"_backfill_posts"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def stop_backfill(self, source_id: int) -> SourceRecord:
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
Clears the running state + cursor/chunk/stall bookkeeping."""
@@ -369,7 +403,8 @@ class SourceService:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_posts"):
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_recapture",
"_backfill_posts"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = 0