bvandeusen
682beafbc5
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".
- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
distinct from auth so the operator knows the fix is updating the
ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.
Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
73 lines
2.9 KiB
Python
73 lines
2.9 KiB
Python
"""Patreon API contract test (native ingester build step 4).
|
|
|
|
The drift tripwire the plan calls for: if the field-set we SEND or the parser we
|
|
resolve responses against drifts from what real Patreon returns, this build goes
|
|
RED — instead of the ingester silently importing nothing in production (the exact
|
|
failure mode the native ingester exists to escape).
|
|
|
|
Two halves:
|
|
- the recorded `/api/posts` fixture must still parse end-to-end (no drift), and
|
|
- the request params must still carry every field the parser depends on (the
|
|
fixture already has the data, so trimming the request would NOT trip the
|
|
parse half — this half guards the request shape directly).
|
|
|
|
Pure: no network, no DB.
|
|
"""
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
from backend.app.services.patreon_client import MediaItem, PatreonClient
|
|
|
|
_FIXTURE = Path(__file__).parent / "fixtures" / "patreon_posts_page1.json"
|
|
|
|
# Pinned media total across the 4 fixture posts: 1001→2 (gallery, image_large
|
|
# cover deduped), 1002→1 (attachment+postfile collapse), 1003→1 (inline content
|
|
# img), 1004→1 (video postfile). A parser change that drops or doubles media
|
|
# trips this.
|
|
_EXPECTED_MEDIA_TOTAL = 5
|
|
_KNOWN_KINDS = {"images", "image_large", "attachments", "postfile", "content"}
|
|
|
|
|
|
def test_recorded_fixture_parses_end_to_end():
|
|
response = json.loads(_FIXTURE.read_text())
|
|
client = PatreonClient(cookies_path=None)
|
|
|
|
client._validate_response(response) # top-level shape must hold
|
|
index = client._transform(response)
|
|
|
|
total = 0
|
|
for post in response["data"]:
|
|
items = client.extract_media(post, index) # must not raise drift
|
|
for m in items:
|
|
assert isinstance(m, MediaItem)
|
|
assert m.url, f"empty url in post {post['id']}"
|
|
assert m.filename, f"empty filename in post {post['id']}"
|
|
assert m.kind in _KNOWN_KINDS, f"unknown kind {m.kind!r}"
|
|
assert m.post_id == post["id"]
|
|
total += len(items)
|
|
|
|
assert total == _EXPECTED_MEDIA_TOTAL
|
|
|
|
|
|
def test_request_field_set_carries_parser_dependencies():
|
|
"""The params we SEND must keep requesting the fields the parser resolves
|
|
against. Trimming `fields[media]=file_name`, an `include` relationship, or a
|
|
`fields[post]` key would make real responses parse to nothing — caught here,
|
|
not by the fixture parse above."""
|
|
params = PatreonClient(cookies_path=None)._params("5555", None)
|
|
|
|
media_fields = params["fields[media]"]
|
|
assert "file_name" in media_fields
|
|
assert "image_urls" in media_fields or "download_url" in media_fields
|
|
|
|
includes = params["include"]
|
|
for rel in ("images", "attachments_media", "media"):
|
|
assert rel in includes, f"missing include relationship {rel}"
|
|
|
|
post_fields = params["fields[post]"]
|
|
for field in ("content", "post_file", "image"):
|
|
assert field in post_fields, f"missing post field {field}"
|
|
|
|
assert params["filter[campaign_id]"] == "5555"
|