fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)
THE empty-body root cause. Patreon deprecated the flat `content` HTML field — it returns null on the feed AND the detail endpoint, for every post type (confirmed against the live API: all 135 StickySpoodge posts, text_only/ image_file/poll alike). The real body now lives in `content_json_string` (a ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse fields[post]=content request omits it. Not credential, not post_type: a request shape gone stale. - NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks bold/italic/underline/strike/code/link, hardBreak, inline images, lists, headings; unknown nodes degrade to children). post_body_html(attrs) = the one resolver: legacy content HTML else convert content_json_string. - patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post]) and resolve via post_body_html (replaces the wrong sparse req + full-fetch fallback). - patreon_downloader._write_sidecar_data: resolve body via post_body_html (feed content_json_string) before the detail-fetch; memoize resolved HTML. - tests: prosemirror converter unit tests; client legacy + content_json_string paths; contract pins content_json_string. Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -41,6 +41,7 @@ from urllib.parse import urlsplit
|
||||
|
||||
import requests
|
||||
|
||||
from ..utils.prosemirror import post_body_html
|
||||
from .file_validator import is_validatable, quarantine_file, validate_file
|
||||
from .patreon_client import (
|
||||
_BACKOFF_CAP_SECONDS,
|
||||
@@ -553,19 +554,22 @@ class PatreonDownloader:
|
||||
the per-media sidecar — a media-less post has no source file."""
|
||||
attrs = post.get("attributes") or {}
|
||||
title = attrs.get("title")
|
||||
content = attrs.get("content")
|
||||
# The feed/list endpoint frequently returns an empty `content`; the full
|
||||
# HTML body (formatting + inline <img> + external <a href> links) only
|
||||
# comes from the per-post detail endpoint. Enrich on first write for this
|
||||
# post and MEMOIZE by mutating the shared `post` dict — so a multi-image
|
||||
# post fetches detail exactly once, and a fully-seen post (no fresh
|
||||
# download → no sidecar write) never pays the extra GET.
|
||||
if (not isinstance(content, str) or not content.strip()) and self._content_fetcher:
|
||||
# Resolve the body HTML from the feed attrs: legacy flat `content`, else
|
||||
# convert the current `content_json_string` ProseMirror doc (#842).
|
||||
content = post_body_html(attrs)
|
||||
# The feed/list endpoint frequently returns an empty body; the full body
|
||||
# only comes from the per-post detail endpoint. Enrich on first write for
|
||||
# this post and MEMOIZE the RESOLVED HTML by mutating the shared `post`
|
||||
# dict — so a multi-image post fetches detail at most once, the post-record
|
||||
# body-length read reuses it, and a fully-seen post (no fresh download → no
|
||||
# sidecar write) never pays the extra GET.
|
||||
if (not content or not content.strip()) and self._content_fetcher:
|
||||
fetched = self._content_fetcher(str(post.get("id") or ""))
|
||||
if fetched:
|
||||
attrs["content"] = fetched
|
||||
post["attributes"] = attrs
|
||||
content = fetched
|
||||
if isinstance(content, str) and content.strip():
|
||||
attrs["content"] = content
|
||||
post["attributes"] = attrs
|
||||
published = attrs.get("published_at")
|
||||
url = attrs.get("url")
|
||||
data = {
|
||||
|
||||
Reference in New Issue
Block a user