feat(recovery): surgical re-fetch for deep posts via ExternalLink reset
CI / lint (push) Successful in 5s
CI / frontend-build (push) Successful in 47s
CI / backend-lint-and-test (push) Successful in 2m0s
CI / integration (push) Successful in 4m50s

Operator-flagged: the recovered defective files live DEEP in their artists'
back-catalogues — the normal download cadence (by design, via the seen-gates)
will never re-walk them, so recovery's source re-check alone can't bring them
back. The durable per-post handle is the ExternalLink row, which survives the
image delete:

- services/external_links.refetch_links_for_post: reset settled links to
  pending (fresh attempt budget, in-flight left alone) + dispatch their
  fetches; sha-dedupe at import discards payload files that still exist, so
  only the missing file lands.
- recover_defective_image now captures the image's post ids BEFORE the delete
  cascades provenance away and resets those posts' links — future recoveries
  are surgical automatically (response gains links_reset; source re-check
  stays for gallery-dl-native files within walk reach).
- POST /api/admin/posts/refetch-external {external_post_id, source_id?} — the
  manual tool for the three files recovered before this fix existed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
2026-07-02 21:07:21 -04:00
parent b1cfbcc06a
commit aa12a57f97
5 changed files with 213 additions and 11 deletions
+52
View File
@@ -0,0 +1,52 @@
"""Surgical re-fetch of a post's external file-host links.
The normal download cadence never re-walks deep back-catalogue posts (the
seen-gates exist precisely to keep old items from resurfacing), so when a
file that CAME from an ExternalLink is deleted — e.g. the failure-triage
recovery flow removing a corrupt original — a plain source re-check will
never bring it back. The link ROW is the durable, per-post handle: resetting
it to pending and dispatching the fetch re-downloads exactly that link's
payload, and sha-dedupe at import discards anything that still exists — so
only the missing file actually lands. (Operator 2026-07-03: recovery must
not require artist-wide deep scans.)
"""
import logging
from sqlalchemy import select
from sqlalchemy.orm import Session
from ..models import ExternalLink
log = logging.getLogger(__name__)
def refetch_links_for_post(session: Session, post_id: int) -> dict:
"""Reset every settled ExternalLink on a post to pending (fresh attempt
budget) and dispatch its fetch. In-flight ('downloading') links are left
alone. Commits. Returns {links_total, links_reset}."""
links = session.execute(
select(ExternalLink).where(ExternalLink.post_id == post_id)
).scalars().all()
reset_ids = []
for link in links:
if link.status == "downloading":
continue
link.status = "pending"
link.attempts = 0
link.last_error = None
link.completed_at = None
reset_ids.append(link.id)
session.commit()
if reset_ids:
# Lazy import (services -> tasks would cycle at module load). The
# 10-min extdl sweep would pick pending rows up anyway — dispatching
# directly just skips the wait.
from ..tasks.external import fetch_external_link
for lid in reset_ids:
fetch_external_link.delay(lid)
log.info(
"external refetch: post %s%d/%d link(s) reset + dispatched",
post_id, len(reset_ids), len(links),
)
return {"links_total": len(links), "links_reset": len(reset_ids)}
+32 -9
View File
@@ -121,13 +121,22 @@ def triage_errored_jobs(
def recover_defective_image(
session: Session, image_id: int, *, images_root: Path,
) -> dict:
"""Delete the defective copy + record and re-poll its subscription Source.
"""Delete the defective copy + record and queue its re-fetch — surgically
where possible.
Mirrors the Layer-2 import refetch: with the bad file gone, the source's
next gallery-dl run re-fetches a fresh copy, which re-imports as a new
record and re-enters the GPU pipeline. The record delete cascades the
error tombstones with it. 'no_source' when no enabled, real-URL Source is
reachable via the image's provenance — manual remediation there."""
Two re-fetch layers (operator 2026-07-03: deep back-catalogue items are
NEVER re-walked by the normal cadence, so recovery can't rely on it):
1. SURGICAL: any ExternalLink rows on the image's post(s) are reset +
re-dispatched — this is how external-host files (the common defect
case: big videos) come back regardless of post age. Sha-dedupe at
import discards payload files that still exist.
2. BROAD: a source re-check, which re-fetches gallery-dl-NATIVE files the
walk still reaches (recent posts). A native file deeper than the walk
needs a per-source backfill/deep scan — reported via links_reset=0 so
the caller can say so.
The record delete cascades the error tombstones with it. 'no_source' when
no enabled, real-URL Source resolves via provenance — manual there."""
rec = session.get(ImageRecord, image_id)
if rec is None:
return {"status": "not_found"}
@@ -143,14 +152,28 @@ def recover_defective_image(
).scalars().first()
if src_id is None:
return {"status": "no_source"}
# Capture the post linkage BEFORE the delete cascades provenance away.
post_ids = session.execute(
select(ImageProvenance.post_id)
.where(ImageProvenance.image_record_id == image_id)
).scalars().all()
path = rec.path
summary = delete_images(session, image_ids=[image_id], images_root=images_root)
from ..external_links import refetch_links_for_post
links_reset = 0
for pid in post_ids:
links_reset += refetch_links_for_post(session, pid)["links_reset"]
# Lazy import (services -> tasks would cycle at module load).
from ...tasks.download import download_source
download_source.delay(src_id)
log.warning(
"gpu triage recovery: deleted defective image %s (%s) and queued a "
"re-check of source %s to re-fetch it", image_id, path, src_id,
"gpu triage recovery: deleted defective image %s (%s); reset %d "
"external link(s) and queued a re-check of source %s",
image_id, path, links_reset, src_id,
)
return {"status": "refetch_queued", "source_id": src_id, **summary}
return {
"status": "refetch_queued", "source_id": src_id,
"links_reset": links_reset, **summary,
}