feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -198,6 +198,7 @@ class PatreonDownloader:
|
||||
*,
|
||||
is_seen: Callable[[object], bool] = lambda m: False,
|
||||
should_stop: Callable[[], bool] = lambda: False,
|
||||
recapture: bool = False,
|
||||
) -> list[MediaOutcome]:
|
||||
"""Download every media item of one post; return per-item outcomes.
|
||||
|
||||
@@ -212,6 +213,11 @@ class PatreonDownloader:
|
||||
re-checks the budget between posts), so we honour the deadline mid-post
|
||||
and return the items done so far — the rest re-fetch next chunk (they
|
||||
were never marked seen). Bounds chunk overrun to one media download.
|
||||
|
||||
`recapture` (#830): don't re-download already-present media, but DO surface
|
||||
on-disk media as `skipped_disk` (with its path) even when the seen-ledger
|
||||
would tier-1 skip it — so the engine can backfill source_filehash for
|
||||
inline-image localization. Genuinely-missing seen media is NOT refetched.
|
||||
"""
|
||||
post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
|
||||
outcomes: list[MediaOutcome] = []
|
||||
@@ -221,7 +227,10 @@ class PatreonDownloader:
|
||||
break
|
||||
try:
|
||||
outcomes.append(
|
||||
self._download_one(post, media, post_dir, artist_slug, i, is_seen)
|
||||
self._download_one(
|
||||
post, media, post_dir, artist_slug, i, is_seen,
|
||||
recapture=recapture,
|
||||
)
|
||||
)
|
||||
except Exception as exc: # resilient: isolate one item's failure
|
||||
log.warning(
|
||||
@@ -243,9 +252,15 @@ class PatreonDownloader:
|
||||
artist_slug: str,
|
||||
index: int,
|
||||
is_seen: Callable[[object], bool],
|
||||
*,
|
||||
recapture: bool = False,
|
||||
) -> MediaOutcome:
|
||||
# tier-1: seen ledger (injected; no DB here).
|
||||
if is_seen(media):
|
||||
# tier-1: seen ledger (injected; no DB here). In recapture mode we DON'T
|
||||
# short-circuit here — we fall through to the disk check so an on-disk
|
||||
# seen file is surfaced as skipped_disk (with its path) for source_filehash
|
||||
# backfill; a seen file that's NOT on disk is left alone (not refetched).
|
||||
seen = is_seen(media)
|
||||
if seen and not recapture:
|
||||
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
|
||||
|
||||
nn = f"{index:02d}"
|
||||
@@ -265,6 +280,12 @@ class PatreonDownloader:
|
||||
media=media, status="skipped_disk", path=existing, error=None
|
||||
)
|
||||
|
||||
# recapture: a seen item that isn't on disk is NOT re-downloaded (that's
|
||||
# recovery's job) — recapture only re-grabs post text + localizes existing
|
||||
# files. Returns skipped_seen so the run-of-seen / counts stay consistent.
|
||||
if seen:
|
||||
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
|
||||
|
||||
post_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Pace real downloads only (the skips above already returned). plan #703.
|
||||
|
||||
Reference in New Issue
Block a user