feat(patreon): drift detection + error categorization — build step 4 (plan #697)
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,72 @@
|
||||
"""Patreon API contract test (native ingester build step 4).
|
||||
|
||||
The drift tripwire the plan calls for: if the field-set we SEND or the parser we
|
||||
resolve responses against drifts from what real Patreon returns, this build goes
|
||||
RED — instead of the ingester silently importing nothing in production (the exact
|
||||
failure mode the native ingester exists to escape).
|
||||
|
||||
Two halves:
|
||||
- the recorded `/api/posts` fixture must still parse end-to-end (no drift), and
|
||||
- the request params must still carry every field the parser depends on (the
|
||||
fixture already has the data, so trimming the request would NOT trip the
|
||||
parse half — this half guards the request shape directly).
|
||||
|
||||
Pure: no network, no DB.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from backend.app.services.patreon_client import MediaItem, PatreonClient
|
||||
|
||||
_FIXTURE = Path(__file__).parent / "fixtures" / "patreon_posts_page1.json"
|
||||
|
||||
# Pinned media total across the 4 fixture posts: 1001→2 (gallery, image_large
|
||||
# cover deduped), 1002→1 (attachment+postfile collapse), 1003→1 (inline content
|
||||
# img), 1004→1 (video postfile). A parser change that drops or doubles media
|
||||
# trips this.
|
||||
_EXPECTED_MEDIA_TOTAL = 5
|
||||
_KNOWN_KINDS = {"images", "image_large", "attachments", "postfile", "content"}
|
||||
|
||||
|
||||
def test_recorded_fixture_parses_end_to_end():
|
||||
response = json.loads(_FIXTURE.read_text())
|
||||
client = PatreonClient(cookies_path=None)
|
||||
|
||||
client._validate_response(response) # top-level shape must hold
|
||||
index = client._transform(response)
|
||||
|
||||
total = 0
|
||||
for post in response["data"]:
|
||||
items = client.extract_media(post, index) # must not raise drift
|
||||
for m in items:
|
||||
assert isinstance(m, MediaItem)
|
||||
assert m.url, f"empty url in post {post['id']}"
|
||||
assert m.filename, f"empty filename in post {post['id']}"
|
||||
assert m.kind in _KNOWN_KINDS, f"unknown kind {m.kind!r}"
|
||||
assert m.post_id == post["id"]
|
||||
total += len(items)
|
||||
|
||||
assert total == _EXPECTED_MEDIA_TOTAL
|
||||
|
||||
|
||||
def test_request_field_set_carries_parser_dependencies():
|
||||
"""The params we SEND must keep requesting the fields the parser resolves
|
||||
against. Trimming `fields[media]=file_name`, an `include` relationship, or a
|
||||
`fields[post]` key would make real responses parse to nothing — caught here,
|
||||
not by the fixture parse above."""
|
||||
params = PatreonClient(cookies_path=None)._params("5555", None)
|
||||
|
||||
media_fields = params["fields[media]"]
|
||||
assert "file_name" in media_fields
|
||||
assert "image_urls" in media_fields or "download_url" in media_fields
|
||||
|
||||
includes = params["include"]
|
||||
for rel in ("images", "attachments_media", "media"):
|
||||
assert rel in includes, f"missing include relationship {rel}"
|
||||
|
||||
post_fields = params["fields[post]"]
|
||||
for field in ("content", "post_file", "image"):
|
||||
assert field in post_fields, f"missing post field {field}"
|
||||
|
||||
assert params["filter[campaign_id]"] == "5555"
|
||||
Reference in New Issue
Block a user