feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s

A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 20:58:40 -04:00
parent 96c29c370b
commit 65ec29ba9b
18 changed files with 467 additions and 29 deletions
+24 -3
View File
@@ -198,6 +198,7 @@ class PatreonDownloader:
*,
is_seen: Callable[[object], bool] = lambda m: False,
should_stop: Callable[[], bool] = lambda: False,
recapture: bool = False,
) -> list[MediaOutcome]:
"""Download every media item of one post; return per-item outcomes.
@@ -212,6 +213,11 @@ class PatreonDownloader:
re-checks the budget between posts), so we honour the deadline mid-post
and return the items done so far — the rest re-fetch next chunk (they
were never marked seen). Bounds chunk overrun to one media download.
`recapture` (#830): don't re-download already-present media, but DO surface
on-disk media as `skipped_disk` (with its path) even when the seen-ledger
would tier-1 skip it — so the engine can backfill source_filehash for
inline-image localization. Genuinely-missing seen media is NOT refetched.
"""
post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
outcomes: list[MediaOutcome] = []
@@ -221,7 +227,10 @@ class PatreonDownloader:
break
try:
outcomes.append(
self._download_one(post, media, post_dir, artist_slug, i, is_seen)
self._download_one(
post, media, post_dir, artist_slug, i, is_seen,
recapture=recapture,
)
)
except Exception as exc: # resilient: isolate one item's failure
log.warning(
@@ -243,9 +252,15 @@ class PatreonDownloader:
artist_slug: str,
index: int,
is_seen: Callable[[object], bool],
*,
recapture: bool = False,
) -> MediaOutcome:
# tier-1: seen ledger (injected; no DB here).
if is_seen(media):
# tier-1: seen ledger (injected; no DB here). In recapture mode we DON'T
# short-circuit here — we fall through to the disk check so an on-disk
# seen file is surfaced as skipped_disk (with its path) for source_filehash
# backfill; a seen file that's NOT on disk is left alone (not refetched).
seen = is_seen(media)
if seen and not recapture:
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
nn = f"{index:02d}"
@@ -265,6 +280,12 @@ class PatreonDownloader:
media=media, status="skipped_disk", path=existing, error=None
)
# recapture: a seen item that isn't on disk is NOT re-downloaded (that's
# recovery's job) — recapture only re-grabs post text + localizes existing
# files. Returns skipped_seen so the run-of-seen / counts stay consistent.
if seen:
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
post_dir.mkdir(parents=True, exist_ok=True)
# Pace real downloads only (the skips above already returned). plan #703.