feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)
A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -105,18 +105,28 @@ class Ingester:
|
||||
) -> DownloadResult:
|
||||
"""Walk + download for one source, returning a gallery-dl-shaped result.
|
||||
|
||||
`mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
|
||||
seen-ledger AND the dead-letter ledger (tier-2 disk still skips kept
|
||||
files). The walk stops on:
|
||||
`mode` is "tick" | "backfill" | "recovery" | "recapture". Recovery
|
||||
bypasses the tier-1 seen-ledger AND the dead-letter ledger (tier-2 disk
|
||||
still skips kept files). Recapture (#830) is the cheap "re-grab post
|
||||
text" walk: it bypasses the post-record gate (so EVERY post's body +
|
||||
external links are re-captured / detail-fetched) but KEEPS the media
|
||||
seen-ledger — on-disk media is NOT re-downloaded, only surfaced so its
|
||||
ImageRecord's source_filehash can be backfilled for inline-image
|
||||
localization. The walk stops on:
|
||||
- budget exhaustion (time_budget_seconds) → TIMEOUT / PARTIAL
|
||||
- tick early-out (seen_threshold contiguous seen) → success
|
||||
- reaching the bottom of the feed → success (rc 0)
|
||||
A client-level failure (drift / auth / network) fails the whole run loud.
|
||||
"""
|
||||
bypass_seen = mode == "recovery"
|
||||
recapture = mode == "recapture"
|
||||
# Both recovery and recapture re-capture EVERY post's body + links — they
|
||||
# bypass the post-record seen-gate (recovery via bypass_seen, recapture
|
||||
# explicitly). A plain backfill stays gated (capture once per post).
|
||||
recapture_records = bypass_seen or recapture
|
||||
# Only deep walks checkpoint their cursor mid-flight (plan #705 #6); a
|
||||
# tick has no resumable backfill state.
|
||||
checkpoint = mode in ("backfill", "recovery")
|
||||
checkpoint = mode in ("backfill", "recovery", "recapture")
|
||||
ledger_key = self._ledger_key
|
||||
# Optional seams (Patreon native ingester): capture pure-text posts that
|
||||
# have NO downloadable media so the artist archive is complete. Absent on
|
||||
@@ -129,6 +139,10 @@ class Ingester:
|
||||
written: list[str] = []
|
||||
post_records: list[str] = []
|
||||
quarantined_paths: list[str] = []
|
||||
# #830 recapture: (on-disk path, CDN source_url) pairs for already-present
|
||||
# media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
|
||||
# re-downloading or unlinking the file. Empty outside recapture mode.
|
||||
relink: list[tuple[str, str]] = []
|
||||
downloaded = 0
|
||||
errors = 0
|
||||
quarantined = 0
|
||||
@@ -165,6 +179,7 @@ class Ingester:
|
||||
quarantined_paths=list(quarantined_paths),
|
||||
written_paths=written,
|
||||
post_record_paths=list(post_records),
|
||||
relink_source_paths=list(relink),
|
||||
stdout="\n".join(log_lines),
|
||||
stderr="",
|
||||
return_code=return_code,
|
||||
@@ -241,7 +256,7 @@ class Ingester:
|
||||
if rk is not None:
|
||||
pkey, ppid = rk
|
||||
already = (
|
||||
set() if bypass_seen
|
||||
set() if recapture_records
|
||||
else self._seen_keys(source_id, [pkey])
|
||||
)
|
||||
if pkey not in already:
|
||||
@@ -275,6 +290,7 @@ class Ingester:
|
||||
outcomes = self.downloader.download_post(
|
||||
post, media, artist_slug, is_seen=_is_skip,
|
||||
should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
|
||||
recapture=recapture,
|
||||
)
|
||||
|
||||
to_mark: list[tuple[str, str]] = []
|
||||
@@ -300,6 +316,12 @@ class Ingester:
|
||||
to_clear.append(key)
|
||||
skipped_count += 1
|
||||
consecutive_seen += 1
|
||||
# #830 recapture: surface (on-disk path, CDN url) so phase
|
||||
# 3 can backfill source_filehash for inline-image
|
||||
# localization — a SEPARATE non-deleting channel, never the
|
||||
# import list (which would unlink the file, per above).
|
||||
if recapture and outcome.path is not None:
|
||||
relink.append((str(outcome.path), media_item.url))
|
||||
elif outcome.status == "skipped_seen":
|
||||
skipped_count += 1
|
||||
consecutive_seen += 1
|
||||
|
||||
Reference in New Issue
Block a user