feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 44s
CI / integration (push) Successful in 3m21s

A plain backfill gates post-body capture on the seen-ledger, so a post whose
media is already on disk AND whose post key is already seen never gets its body
recaptured (operator-flagged: Industrial Lust description missing). Recovery
recaptures unconditionally but re-downloads the whole source.

New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the
post-record gate so EVERY post's body + external links are re-captured
(detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and
surfaces already-present media via a separate non-deleting relink channel so the
importer backfills ImageRecord.source_filehash for inline-image localization.

- ingest_core: recapture mode + recapture_records gate bypass + relink collect
- patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path),
  never refetches seen-missing media, still downloads genuinely-new
- importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks
- download_service: mode derivation + phase-3 relink loop + lifecycle clear
- source_service/api: start_recapture + backfill_recapture field + action
- frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/
  Row/Card/SubscriptionsTab + sources store
- tests across ingester/downloader/importer/source_service/api/download_service

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-14 20:58:40 -04:00
parent 96c29c370b
commit 65ec29ba9b
18 changed files with 467 additions and 29 deletions
+27 -5
View File
@@ -105,18 +105,28 @@ class Ingester:
) -> DownloadResult:
"""Walk + download for one source, returning a gallery-dl-shaped result.
`mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
seen-ledger AND the dead-letter ledger (tier-2 disk still skips kept
files). The walk stops on:
`mode` is "tick" | "backfill" | "recovery" | "recapture". Recovery
bypasses the tier-1 seen-ledger AND the dead-letter ledger (tier-2 disk
still skips kept files). Recapture (#830) is the cheap "re-grab post
text" walk: it bypasses the post-record gate (so EVERY post's body +
external links are re-captured / detail-fetched) but KEEPS the media
seen-ledger — on-disk media is NOT re-downloaded, only surfaced so its
ImageRecord's source_filehash can be backfilled for inline-image
localization. The walk stops on:
- budget exhaustion (time_budget_seconds) → TIMEOUT / PARTIAL
- tick early-out (seen_threshold contiguous seen) → success
- reaching the bottom of the feed → success (rc 0)
A client-level failure (drift / auth / network) fails the whole run loud.
"""
bypass_seen = mode == "recovery"
recapture = mode == "recapture"
# Both recovery and recapture re-capture EVERY post's body + links — they
# bypass the post-record seen-gate (recovery via bypass_seen, recapture
# explicitly). A plain backfill stays gated (capture once per post).
recapture_records = bypass_seen or recapture
# Only deep walks checkpoint their cursor mid-flight (plan #705 #6); a
# tick has no resumable backfill state.
checkpoint = mode in ("backfill", "recovery")
checkpoint = mode in ("backfill", "recovery", "recapture")
ledger_key = self._ledger_key
# Optional seams (Patreon native ingester): capture pure-text posts that
# have NO downloadable media so the artist archive is complete. Absent on
@@ -129,6 +139,10 @@ class Ingester:
written: list[str] = []
post_records: list[str] = []
quarantined_paths: list[str] = []
# #830 recapture: (on-disk path, CDN source_url) pairs for already-present
# media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
# re-downloading or unlinking the file. Empty outside recapture mode.
relink: list[tuple[str, str]] = []
downloaded = 0
errors = 0
quarantined = 0
@@ -165,6 +179,7 @@ class Ingester:
quarantined_paths=list(quarantined_paths),
written_paths=written,
post_record_paths=list(post_records),
relink_source_paths=list(relink),
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
@@ -241,7 +256,7 @@ class Ingester:
if rk is not None:
pkey, ppid = rk
already = (
set() if bypass_seen
set() if recapture_records
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
@@ -275,6 +290,7 @@ class Ingester:
outcomes = self.downloader.download_post(
post, media, artist_slug, is_seen=_is_skip,
should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
recapture=recapture,
)
to_mark: list[tuple[str, str]] = []
@@ -300,6 +316,12 @@ class Ingester:
to_clear.append(key)
skipped_count += 1
consecutive_seen += 1
# #830 recapture: surface (on-disk path, CDN url) so phase
# 3 can backfill source_filehash for inline-image
# localization — a SEPARATE non-deleting channel, never the
# import list (which would unlink the file, per above).
if recapture and outcome.path is not None:
relink.append((str(outcome.path), media_item.url))
elif outcome.status == "skipped_seen":
skipped_count += 1
consecutive_seen += 1