bvandeusenandClaude Opus 5.5
5259caa846
CI and images / lint (push) Successful in 1s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 20s
CI and images / extension-test (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m20s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 2m1s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
The daily recovery reset every dead link to a fresh three-attempt budget, so a link that can never succeed (an archive too big to import inside the fetch's hard limit, a file made private) was downloaded up to three times a day, every day. Recovery now gives a dead link one more attempt once it has been dead for 2**recoveries days (1, 2, 4, 8, 16), and stops after five. A recovered link that fails again goes straight back to dead. Retention prunes only links recovery has given up on, 30 days after their last attempt. A manual refetch of a post restores the full budgets. Migration 0118 adds external_link.recoveries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
54 lines
2.1 KiB
Python
54 lines
2.1 KiB
Python
"""Surgical re-fetch of a post's external file-host links.
|
|
|
|
The normal download cadence never re-walks deep back-catalogue posts (the
|
|
seen-gates exist precisely to keep old items from resurfacing), so when a
|
|
file that CAME from an ExternalLink is deleted — e.g. the failure-triage
|
|
recovery flow removing a corrupt original — a plain source re-check will
|
|
never bring it back. The link ROW is the durable, per-post handle: resetting
|
|
it to pending and dispatching the fetch re-downloads exactly that link's
|
|
payload, and sha-dedupe at import discards anything that still exists — so
|
|
only the missing file actually lands. (Operator 2026-07-03: recovery must
|
|
not require artist-wide deep scans.)
|
|
"""
|
|
import logging
|
|
|
|
from sqlalchemy import select
|
|
from sqlalchemy.orm import Session
|
|
|
|
from ..models import ExternalLink
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
|
|
def refetch_links_for_post(session: Session, post_id: int) -> dict:
|
|
"""Reset every settled ExternalLink on a post to pending (fresh attempt and
|
|
recovery budgets) and dispatch its fetch. In-flight ('downloading') links
|
|
are left alone. Commits. Returns {links_total, links_reset}."""
|
|
links = session.execute(
|
|
select(ExternalLink).where(ExternalLink.post_id == post_id)
|
|
).scalars().all()
|
|
reset_ids = []
|
|
for link in links:
|
|
if link.status == "downloading":
|
|
continue
|
|
link.status = "pending"
|
|
link.attempts = 0
|
|
link.recoveries = 0
|
|
link.last_error = None
|
|
link.completed_at = None
|
|
reset_ids.append(link.id)
|
|
session.commit()
|
|
if reset_ids:
|
|
# Lazy import (services -> tasks would cycle at module load). The
|
|
# 10-min extdl sweep would pick pending rows up anyway — dispatching
|
|
# directly just skips the wait.
|
|
from ..tasks.external import fetch_external_link
|
|
|
|
for lid in reset_ids:
|
|
fetch_external_link.delay(lid)
|
|
log.info(
|
|
"external refetch: post %s — %d/%d link(s) reset + dispatched",
|
|
post_id, len(reset_ids), len(links),
|
|
)
|
|
return {"links_total": len(links), "links_reset": len(reset_ids)}
|