feat(patreon): drift detection + error categorization — build step 4 (plan #697)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 28s
CI / integration (push) Successful in 2m58s

Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".

- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
  distinct from auth so the operator knows the fix is updating the
  ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
  for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
  expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
  PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
  update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
  transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.

Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 21:56:58 -04:00
parent 96c30eba13
commit 682beafbc5
7 changed files with 261 additions and 35 deletions
+30 -16
View File
@@ -51,6 +51,7 @@ from .gallery_dl import DownloadResult, ErrorType
from .patreon_client import (
MediaItem,
PatreonAPIError,
PatreonAuthError,
PatreonClient,
PatreonDriftError,
)
@@ -242,7 +243,9 @@ class PatreonIngester:
break
else:
reached_bottom = True
except (PatreonDriftError, PatreonAPIError) as exc:
except PatreonAPIError as exc:
# Base of PatreonAuthError + PatreonDriftError — catches every
# client-level failure; _failure_result maps it to a typed error.
return self._failure_result(exc, _result)
if errors:
@@ -294,24 +297,35 @@ class PatreonIngester:
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
"""Map a client-level exception to a loud, typed failed DownloadResult.
Step 3 keeps this deliberately coarse — auth-ish drift vs. everything
else — so the integration is correct end-to-end. Step 4 (drift detection
+ error categorization) refines the typed-error mapping and adds the
contract test. Either way we NEVER return a silent zero-download
"success": the whole point of the native ingester is to fail red when
Patreon's API shape or our auth changes.
We NEVER return a silent zero-download "success" — the whole point of the
native ingester is to fail RED when Patreon's API shape or our auth
changes. The typed mapping lets FailingSourcesCard render the right chip
and tells the operator what to do:
- PatreonAuthError → AUTH_ERROR (rotate cookies)
- PatreonDriftError → API_DRIFT (ingester field-set/parser needs update)
- HTTP 429 / 404 → RATE_LIMITED / NOT_FOUND
- other HTTP status → HTTP_ERROR; transport failure → NETWORK_ERROR
PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
they must be matched before the generic HTTP/transport fallthrough.
"""
message = str(exc)
lowered = message.lower()
if isinstance(exc, PatreonDriftError):
if "login" in lowered or "challenge" in lowered or "auth" in lowered:
error_type = ErrorType.AUTH_ERROR
else:
error_type = ErrorType.UNKNOWN_ERROR
if isinstance(exc, PatreonAuthError):
error_type = ErrorType.AUTH_ERROR
elif isinstance(exc, PatreonDriftError):
error_type = ErrorType.API_DRIFT
message = f"Patreon API changed — ingester needs update: {message}"
else:
error_type = ErrorType.NETWORK_ERROR
log.warning("Patreon ingest failed: %s", message)
else: # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
status = getattr(exc, "status_code", None)
if status == 429:
error_type = ErrorType.RATE_LIMITED
elif status == 404:
error_type = ErrorType.NOT_FOUND
elif status is not None:
error_type = ErrorType.HTTP_ERROR
else:
error_type = ErrorType.NETWORK_ERROR
log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
return _result(
success=False, return_code=1,
error_type=error_type, error_message=message,