feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -70,6 +70,10 @@ class SourceRecord:
|
||||
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
|
||||
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
|
||||
backfill_bypass_seen: bool
|
||||
# #830: a running deep-walk in RECAPTURE mode (re-grab post bodies/links +
|
||||
# localize on-disk inline images, no media re-download). Lets the badge label
|
||||
# it "Recapturing".
|
||||
backfill_recapture: bool
|
||||
# plan #704: cumulative posts processed across the walk's chunks — live
|
||||
# progress for the badge.
|
||||
backfill_posts: int
|
||||
@@ -94,6 +98,7 @@ class SourceRecord:
|
||||
"backfill_state": self.backfill_state,
|
||||
"backfill_chunks": self.backfill_chunks,
|
||||
"backfill_bypass_seen": self.backfill_bypass_seen,
|
||||
"backfill_recapture": self.backfill_recapture,
|
||||
"backfill_posts": self.backfill_posts,
|
||||
}
|
||||
|
||||
@@ -171,6 +176,7 @@ class SourceService:
|
||||
backfill_state=co.get("_backfill_state"),
|
||||
backfill_chunks=int(co.get("_backfill_chunks", 0)),
|
||||
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
|
||||
backfill_recapture=bool(co.get("_backfill_recapture")),
|
||||
backfill_posts=int(co.get("_backfill_posts", 0)),
|
||||
)
|
||||
|
||||
@@ -359,6 +365,34 @@ class SourceService:
|
||||
await self.session.commit()
|
||||
return await self._row_to_record(source)
|
||||
|
||||
async def start_recapture(self, source_id: int) -> SourceRecord:
|
||||
"""#830: arm a RECAPTURE walk — a backfill that re-grabs EVERY post's body
|
||||
+ external links (detail-fetching empty bodies) and localizes already-on-
|
||||
disk inline images, WITHOUT re-downloading media. Reuses the entire #693
|
||||
backfill state machine plus a `_backfill_recapture` flag that flips
|
||||
download mode to recapture. Distinct from recovery (which re-downloads the
|
||||
whole source); the two flags are mutually exclusive, so arming recapture
|
||||
clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
|
||||
fresh from the top. The flag is cleared on completion (download_service)
|
||||
and on stop. Recapture is Patreon-only (the native ingester's post-record
|
||||
capture); inert elsewhere. The UI gates the action to Patreon sources."""
|
||||
source = (await self.session.execute(
|
||||
select(Source).where(Source.id == source_id)
|
||||
)).scalar_one_or_none()
|
||||
if source is None:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
co = dict(source.config_overrides or {})
|
||||
co["_backfill_state"] = "running"
|
||||
co["_backfill_recapture"] = True
|
||||
co.pop("_backfill_bypass_seen", None) # mutually exclusive with recovery
|
||||
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
|
||||
"_backfill_posts"):
|
||||
co.pop(k, None)
|
||||
source.config_overrides = co
|
||||
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
|
||||
await self.session.commit()
|
||||
return await self._row_to_record(source)
|
||||
|
||||
async def stop_backfill(self, source_id: int) -> SourceRecord:
|
||||
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
|
||||
Clears the running state + cursor/chunk/stall bookkeeping."""
|
||||
@@ -369,7 +403,8 @@ class SourceService:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
co = dict(source.config_overrides or {})
|
||||
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
|
||||
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_posts"):
|
||||
"_backfill_chunks", "_backfill_bypass_seen", "_backfill_recapture",
|
||||
"_backfill_posts"):
|
||||
co.pop(k, None)
|
||||
source.config_overrides = co
|
||||
source.backfill_runs_remaining = 0
|
||||
|
||||
Reference in New Issue
Block a user