feat(patreon): drift detection + error categorization — build step 4 (plan #697)
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -51,6 +51,7 @@ from .gallery_dl import DownloadResult, ErrorType
|
||||
from .patreon_client import (
|
||||
MediaItem,
|
||||
PatreonAPIError,
|
||||
PatreonAuthError,
|
||||
PatreonClient,
|
||||
PatreonDriftError,
|
||||
)
|
||||
@@ -242,7 +243,9 @@ class PatreonIngester:
|
||||
break
|
||||
else:
|
||||
reached_bottom = True
|
||||
except (PatreonDriftError, PatreonAPIError) as exc:
|
||||
except PatreonAPIError as exc:
|
||||
# Base of PatreonAuthError + PatreonDriftError — catches every
|
||||
# client-level failure; _failure_result maps it to a typed error.
|
||||
return self._failure_result(exc, _result)
|
||||
|
||||
if errors:
|
||||
@@ -294,24 +297,35 @@ class PatreonIngester:
|
||||
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
|
||||
"""Map a client-level exception to a loud, typed failed DownloadResult.
|
||||
|
||||
Step 3 keeps this deliberately coarse — auth-ish drift vs. everything
|
||||
else — so the integration is correct end-to-end. Step 4 (drift detection
|
||||
+ error categorization) refines the typed-error mapping and adds the
|
||||
contract test. Either way we NEVER return a silent zero-download
|
||||
"success": the whole point of the native ingester is to fail red when
|
||||
Patreon's API shape or our auth changes.
|
||||
We NEVER return a silent zero-download "success" — the whole point of the
|
||||
native ingester is to fail RED when Patreon's API shape or our auth
|
||||
changes. The typed mapping lets FailingSourcesCard render the right chip
|
||||
and tells the operator what to do:
|
||||
- PatreonAuthError → AUTH_ERROR (rotate cookies)
|
||||
- PatreonDriftError → API_DRIFT (ingester field-set/parser needs update)
|
||||
- HTTP 429 / 404 → RATE_LIMITED / NOT_FOUND
|
||||
- other HTTP status → HTTP_ERROR; transport failure → NETWORK_ERROR
|
||||
|
||||
PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
|
||||
they must be matched before the generic HTTP/transport fallthrough.
|
||||
"""
|
||||
message = str(exc)
|
||||
lowered = message.lower()
|
||||
if isinstance(exc, PatreonDriftError):
|
||||
if "login" in lowered or "challenge" in lowered or "auth" in lowered:
|
||||
error_type = ErrorType.AUTH_ERROR
|
||||
else:
|
||||
error_type = ErrorType.UNKNOWN_ERROR
|
||||
if isinstance(exc, PatreonAuthError):
|
||||
error_type = ErrorType.AUTH_ERROR
|
||||
elif isinstance(exc, PatreonDriftError):
|
||||
error_type = ErrorType.API_DRIFT
|
||||
message = f"Patreon API changed — ingester needs update: {message}"
|
||||
else:
|
||||
error_type = ErrorType.NETWORK_ERROR
|
||||
log.warning("Patreon ingest failed: %s", message)
|
||||
else: # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
|
||||
status = getattr(exc, "status_code", None)
|
||||
if status == 429:
|
||||
error_type = ErrorType.RATE_LIMITED
|
||||
elif status == 404:
|
||||
error_type = ErrorType.NOT_FOUND
|
||||
elif status is not None:
|
||||
error_type = ErrorType.HTTP_ERROR
|
||||
else:
|
||||
error_type = ErrorType.NETWORK_ERROR
|
||||
log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
|
||||
return _result(
|
||||
success=False, return_code=1,
|
||||
error_type=error_type, error_message=message,
|
||||
|
||||
Reference in New Issue
Block a user