feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s

A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 20:58:40 -04:00
parent 96c29c370b
commit 65ec29ba9b
18 changed files with 467 additions and 29 deletions
+22
View File
@@ -111,6 +111,10 @@ class DownloadService:
# `_backfill_bypass_seen` flag rides alongside the running backfill state;
# download mode is "recovery" when both are set.
bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
# #830 recapture: re-grab post bodies/links + localize on-disk inline
# images, WITHOUT re-downloading media. Rides alongside the backfill
# state like _backfill_bypass_seen; mutually exclusive with it.
recapture = bool(overrides.get("_backfill_recapture"))
if in_backfill:
skip_value: bool | str = BACKFILL_SKIP_VALUE
source_config.timeout = BACKFILL_CHUNK_SECONDS
@@ -130,6 +134,8 @@ class DownloadService:
if uses_native_ingester(ctx["platform"]):
if in_backfill and bypass_seen:
mode = "recovery"
elif in_backfill and recapture:
mode = "recapture"
elif in_backfill:
mode = "backfill"
else:
@@ -396,6 +402,20 @@ class DownloadService:
await loop.run_in_executor(None, _upsert)
# #830 recapture: backfill source_filehash on EXISTING on-disk images so
# their post-body inline <img src=CDN> remaps to the local copy. A
# SEPARATE non-deleting channel (NOT the import list — that would unlink
# the duplicate file). Empty outside recapture mode.
for rel_str, rel_url in getattr(dl_result, "relink_source_paths", None) or []:
rel_path = Path(rel_str)
if not rel_path.exists(): # noqa: ASYNC240
continue
def _relink(p=rel_path, u=rel_url):
return self.importer.relink_source_filehash(p, u, artist=artist)
await loop.run_in_executor(None, _relink)
# Kick the off-platform file-host downloader for any links this run
# recorded (mega/gdrive/…). Global + idempotent (only claims pending/
# retryable rows); the beat sweep is the backstop. Lazy import dodges a
@@ -508,6 +528,8 @@ class DownloadService:
# plan #697: a recovery walk shares this lifecycle; clear its bypass
# flag on completion so the next routine tick honors the seen-ledger.
new_overrides.pop("_backfill_bypass_seen", None)
# #830: same for the recapture flag.
new_overrides.pop("_backfill_recapture", None)
src.config_overrides = new_overrides
src.backfill_runs_remaining = 0
return