feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -111,6 +111,10 @@ class DownloadService:
|
||||
# `_backfill_bypass_seen` flag rides alongside the running backfill state;
|
||||
# download mode is "recovery" when both are set.
|
||||
bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
|
||||
# #830 recapture: re-grab post bodies/links + localize on-disk inline
|
||||
# images, WITHOUT re-downloading media. Rides alongside the backfill
|
||||
# state like _backfill_bypass_seen; mutually exclusive with it.
|
||||
recapture = bool(overrides.get("_backfill_recapture"))
|
||||
if in_backfill:
|
||||
skip_value: bool | str = BACKFILL_SKIP_VALUE
|
||||
source_config.timeout = BACKFILL_CHUNK_SECONDS
|
||||
@@ -130,6 +134,8 @@ class DownloadService:
|
||||
if uses_native_ingester(ctx["platform"]):
|
||||
if in_backfill and bypass_seen:
|
||||
mode = "recovery"
|
||||
elif in_backfill and recapture:
|
||||
mode = "recapture"
|
||||
elif in_backfill:
|
||||
mode = "backfill"
|
||||
else:
|
||||
@@ -396,6 +402,20 @@ class DownloadService:
|
||||
|
||||
await loop.run_in_executor(None, _upsert)
|
||||
|
||||
# #830 recapture: backfill source_filehash on EXISTING on-disk images so
|
||||
# their post-body inline <img src=CDN> remaps to the local copy. A
|
||||
# SEPARATE non-deleting channel (NOT the import list — that would unlink
|
||||
# the duplicate file). Empty outside recapture mode.
|
||||
for rel_str, rel_url in getattr(dl_result, "relink_source_paths", None) or []:
|
||||
rel_path = Path(rel_str)
|
||||
if not rel_path.exists(): # noqa: ASYNC240
|
||||
continue
|
||||
|
||||
def _relink(p=rel_path, u=rel_url):
|
||||
return self.importer.relink_source_filehash(p, u, artist=artist)
|
||||
|
||||
await loop.run_in_executor(None, _relink)
|
||||
|
||||
# Kick the off-platform file-host downloader for any links this run
|
||||
# recorded (mega/gdrive/…). Global + idempotent (only claims pending/
|
||||
# retryable rows); the beat sweep is the backstop. Lazy import dodges a
|
||||
@@ -508,6 +528,8 @@ class DownloadService:
|
||||
# plan #697: a recovery walk shares this lifecycle; clear its bypass
|
||||
# flag on completion so the next routine tick honors the seen-ledger.
|
||||
new_overrides.pop("_backfill_bypass_seen", None)
|
||||
# #830: same for the recapture flag.
|
||||
new_overrides.pop("_backfill_recapture", None)
|
||||
src.config_overrides = new_overrides
|
||||
src.backfill_runs_remaining = 0
|
||||
return
|
||||
|
||||
Reference in New Issue
Block a user