bvandeusen
976107bbe8
THE empty-body root cause. Patreon deprecated the flat `content` HTML field — it returns null on the feed AND the detail endpoint, for every post type (confirmed against the live API: all 135 StickySpoodge posts, text_only/ image_file/poll alike). The real body now lives in `content_json_string` (a ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse fields[post]=content request omits it. Not credential, not post_type: a request shape gone stale. - NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks bold/italic/underline/strike/code/link, hardBreak, inline images, lists, headings; unknown nodes degrade to children). post_body_html(attrs) = the one resolver: legacy content HTML else convert content_json_string. - patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post]) and resolve via post_body_html (replaces the wrong sparse req + full-fetch fallback). - patreon_downloader._write_sidecar_data: resolve body via post_body_html (feed content_json_string) before the detail-fetch; memoize resolved HTML. - tests: prosemirror converter unit tests; client legacy + content_json_string paths; contract pins content_json_string. Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
75 lines
3.1 KiB
Python
75 lines
3.1 KiB
Python
"""Patreon API contract test (native ingester build step 4).
|
|
|
|
The drift tripwire the plan calls for: if the field-set we SEND or the parser we
|
|
resolve responses against drifts from what real Patreon returns, this build goes
|
|
RED — instead of the ingester silently importing nothing in production (the exact
|
|
failure mode the native ingester exists to escape).
|
|
|
|
Two halves:
|
|
- the recorded `/api/posts` fixture must still parse end-to-end (no drift), and
|
|
- the request params must still carry every field the parser depends on (the
|
|
fixture already has the data, so trimming the request would NOT trip the
|
|
parse half — this half guards the request shape directly).
|
|
|
|
Pure: no network, no DB.
|
|
"""
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
from backend.app.services.patreon_client import MediaItem, PatreonClient
|
|
|
|
_FIXTURE = Path(__file__).parent / "fixtures" / "patreon_posts_page1.json"
|
|
|
|
# Pinned media total across the 4 fixture posts: 1001→2 (gallery, image_large
|
|
# cover deduped), 1002→1 (attachment+postfile collapse), 1003→1 (inline content
|
|
# img), 1004→1 (video postfile). A parser change that drops or doubles media
|
|
# trips this.
|
|
_EXPECTED_MEDIA_TOTAL = 5
|
|
_KNOWN_KINDS = {"images", "image_large", "attachments", "postfile", "content"}
|
|
|
|
|
|
def test_recorded_fixture_parses_end_to_end():
|
|
response = json.loads(_FIXTURE.read_text())
|
|
client = PatreonClient(cookies_path=None)
|
|
|
|
client._validate_response(response) # top-level shape must hold
|
|
index = client._transform(response)
|
|
|
|
total = 0
|
|
for post in response["data"]:
|
|
items = client.extract_media(post, index) # must not raise drift
|
|
for m in items:
|
|
assert isinstance(m, MediaItem)
|
|
assert m.url, f"empty url in post {post['id']}"
|
|
assert m.filename, f"empty filename in post {post['id']}"
|
|
assert m.kind in _KNOWN_KINDS, f"unknown kind {m.kind!r}"
|
|
assert m.post_id == post["id"]
|
|
total += len(items)
|
|
|
|
assert total == _EXPECTED_MEDIA_TOTAL
|
|
|
|
|
|
def test_request_field_set_carries_parser_dependencies():
|
|
"""The params we SEND must keep requesting the fields the parser resolves
|
|
against. Trimming `fields[media]=file_name`, an `include` relationship, or a
|
|
`fields[post]` key would make real responses parse to nothing — caught here,
|
|
not by the fixture parse above."""
|
|
params = PatreonClient(cookies_path=None)._params("5555", None)
|
|
|
|
media_fields = params["fields[media]"]
|
|
assert "file_name" in media_fields
|
|
assert "image_urls" in media_fields or "download_url" in media_fields
|
|
|
|
includes = params["include"]
|
|
for rel in ("images", "attachments_media", "media"):
|
|
assert rel in includes, f"missing include relationship {rel}"
|
|
|
|
post_fields = params["fields[post]"]
|
|
# content_json_string is the CURRENT body field (Patreon nulled flat `content`,
|
|
# #842); content kept for legacy posts. Both must stay requested.
|
|
for field in ("content", "content_json_string", "post_file", "image"):
|
|
assert field in post_fields, f"missing post field {field}"
|
|
|
|
assert params["filter[campaign_id]"] == "5555"
|