fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m13s

THE empty-body root cause. Patreon deprecated the flat `content` HTML field —
it returns null on the feed AND the detail endpoint, for every post type
(confirmed against the live API: all 135 StickySpoodge posts, text_only/
image_file/poll alike). The real body now lives in `content_json_string` (a
ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse
fields[post]=content request omits it. Not credential, not post_type: a request
shape gone stale.

- NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks
  bold/italic/underline/strike/code/link, hardBreak, inline images, lists,
  headings; unknown nodes degrade to children). post_body_html(attrs) = the one
  resolver: legacy content HTML else convert content_json_string.
- patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite
  fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post])
  and resolve via post_body_html (replaces the wrong sparse req + full-fetch
  fallback).
- patreon_downloader._write_sidecar_data: resolve body via post_body_html
  (feed content_json_string) before the detail-fetch; memoize resolved HTML.
- tests: prosemirror converter unit tests; client legacy + content_json_string
  paths; contract pins content_json_string.

Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 00:14:52 -04:00
parent 0d51b93aa7
commit 976107bbe8
6 changed files with 302 additions and 89 deletions
+14 -10
View File
@@ -41,6 +41,7 @@ from urllib.parse import urlsplit
import requests
from ..utils.prosemirror import post_body_html
from .file_validator import is_validatable, quarantine_file, validate_file
from .patreon_client import (
_BACKOFF_CAP_SECONDS,
@@ -553,19 +554,22 @@ class PatreonDownloader:
the per-media sidecar — a media-less post has no source file."""
attrs = post.get("attributes") or {}
title = attrs.get("title")
content = attrs.get("content")
# The feed/list endpoint frequently returns an empty `content`; the full
# HTML body (formatting + inline <img> + external <a href> links) only
# comes from the per-post detail endpoint. Enrich on first write for this
# post and MEMOIZE by mutating the shared `post` dict — so a multi-image
# post fetches detail exactly once, and a fully-seen post (no fresh
# download → no sidecar write) never pays the extra GET.
if (not isinstance(content, str) or not content.strip()) and self._content_fetcher:
# Resolve the body HTML from the feed attrs: legacy flat `content`, else
# convert the current `content_json_string` ProseMirror doc (#842).
content = post_body_html(attrs)
# The feed/list endpoint frequently returns an empty body; the full body
# only comes from the per-post detail endpoint. Enrich on first write for
# this post and MEMOIZE the RESOLVED HTML by mutating the shared `post`
# dict — so a multi-image post fetches detail at most once, the post-record
# body-length read reuses it, and a fully-seen post (no fresh download → no
# sidecar write) never pays the extra GET.
if (not content or not content.strip()) and self._content_fetcher:
fetched = self._content_fetcher(str(post.get("id") or ""))
if fetched:
attrs["content"] = fetched
post["attributes"] = attrs
content = fetched
if isinstance(content, str) and content.strip():
attrs["content"] = content
post["attributes"] = attrs
published = attrs.get("published_at")
url = attrs.get("url")
data = {