feat(patreon): capture full post body via adaptive detail-fetch
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m24s

The feed endpoint (/api/posts) returns `content` empty for many posts, so post
bodies — their formatting, inline <img>, and external <a href> links — were
never captured (the post showed "(no description)"). Enrich an empty feed body
from the per-post detail endpoint (/api/posts/{id}) before writing the importer
sidecar, memoized by mutating the shared post dict so a multi-image post fetches
detail exactly once and fully-seen posts (no fresh download) pay nothing.
Best-effort by design: a body we can't fetch returns None and never fails the
walk. No-doubling and no-clobber-of-populated-body already hold (post upsert is
keyed on external_post_id; an empty body parses to None and isn't applied).

First slice of milestone #64 (rich post capture + faithful rendering +
external-host downloads). Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 12:29:06 -04:00
parent 7fcef53d5b
commit 2c67c27044
5 changed files with 161 additions and 1 deletions
@@ -168,10 +168,17 @@ class PatreonDownloader:
validate: bool = True,
rate_limit: float = 0.0,
session: requests.Session | None = None,
content_fetcher: Callable[[str], str | None] | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
self._validate = validate
# Best-effort enrichment seam: (post_id) -> full HTML body, or None. The
# feed endpoint often omits `content`; the adapter wires this to
# PatreonClient.fetch_post_detail_content so the sidecar captures the
# real body (formatting + inline <img> + external <a href> links).
# None in unit tests / when enrichment isn't wanted.
self._content_fetcher = content_fetcher
# Politeness: seconds to sleep before each actual media download (paces
# the CDN; honors ImportSettings.download_rate_limit_seconds, the same
# value gallery-dl used as its between-downloads `sleep`). 0 = no pacing.
@@ -495,6 +502,18 @@ class PatreonDownloader:
attrs = post.get("attributes") or {}
title = attrs.get("title")
content = attrs.get("content")
# The feed/list endpoint frequently returns an empty `content`; the full
# HTML body (formatting + inline <img> + external <a href> links) only
# comes from the per-post detail endpoint. Enrich on first write for this
# post and MEMOIZE by mutating the shared `post` dict — so a multi-image
# post fetches detail exactly once, and a fully-seen post (no fresh
# download → no sidecar write) never pays the extra GET.
if (not isinstance(content, str) or not content.strip()) and self._content_fetcher:
fetched = self._content_fetcher(str(post.get("id") or ""))
if fetched:
attrs["content"] = fetched
post["attributes"] = attrs
content = fetched
published = attrs.get("published_at")
url = attrs.get("url")
data = {