fix(subscribestar): port gallery-dl's content + preview-skip extraction faithfully

Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body regex either returned empty or over-captured into sibling upload divs and the 'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact _data_from_post rule — content between the post_content-text wrapper and the youtube-uploads div (literal markers), then strip the trix editor's <html><body>…</body></html> document wrapper to its inner. Verified against the live cheunart sample: clean per-post bodies, empty for genuinely text-less posts. Also port gallery-dl's _media_from_post preview guard: skip gallery items whose URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of the Patreon gated-preview bug (#874); this is why a locked post yields no media. Tests: body must not bleed into the pagination counter; trix html-document wrapper stripped; /previews items skipped. Fixture now includes the youtube- uploads close marker present in real markup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:06:52 -04:00
parent 559d29fe1c
commit 8771364cee
2 changed files with 88 additions and 9 deletions
@@ -87,12 +87,15 @@ _GDL_HEADERS = {
 _POST_OPEN = '<div class="post '
 _POST_ID_RE = re.compile(r'data-id="(\d+)"')
 _POST_DATE_RE = re.compile(r'<div class="post-date">([^<]+)</div>')
-# The body: inner of `.post-content .trix-content`. Non-greedy to the LAST
-# </div></div> in the chunk would over-capture; instead capture from trix-content
-# to the close of the post-content wrapper, which the post-body block ends with.
-_BODY_RE = re.compile(
-    r'<div class="post-content"[^>]*>(.*?)</div>\s*</div>\s*</div>', re.DOTALL
-)
+# The body lives between these two LITERAL markers — exactly gallery-dl's
+# `_data_from_post` extraction: from the post_content-text wrapper open to the
+# youtube-uploads div that always follows post-content. A balanced-</div> regex
+# either returned empty or over-captured into sibling upload divs AND the
+# "View next posts (N / M)" pagination counter (the "264 / 265" body bug,
+# cheunart 2026-06-17). The trix editor wraps rich bodies in a full
+# `<html><body>…</body></html>` document, so strip to the body inner when present.
+_CONTENT_OPEN = '<div class="post-content" data-role="post_content-text">'
+_CONTENT_CLOSE = '</div><div class="post-uploads for-youtube"'
 _GALLERY_RE = re.compile(r'data-gallery="([^"]*)"')
 _NEXT_PAGE_RE = re.compile(
    r'data-role="infinite_scroll-next_page"\s+href="([^"]+)"'
@@ -165,6 +168,29 @@ class MediaItem:
    media_id: str


+def _extr(text: str, start: str, end: str) -> str:
+    """Substring between the first `start` and the next `end` after it (gallery-
+    dl's `text.extr`); '' when either marker is absent."""
+    i = text.find(start)
+    if i < 0:
+        return ""
+    i += len(start)
+    j = text.find(end, i)
+    if j < 0:
+        return ""
+    return text[i:j]
+
+
+def _extract_content(chunk: str) -> str:
+    """The post body HTML — gallery-dl's `_data_from_post` content rule: between
+    the post_content-text wrapper and the youtube-uploads div, with the trix
+    editor's `<html><body>…</body></html>` wrapper stripped to its inner."""
+    content = _extr(chunk, _CONTENT_OPEN, _CONTENT_CLOSE)
+    if "<html><body>" in content:
+        content = _extr(content, "<body>", "</body>")
+    return content.strip()
+
+
 def _split_creator_url(campaign_id: str) -> tuple[str, str]:
    """`campaign_id` is the creator URL → (base, slug).

@@ -316,8 +342,7 @@ class SubscribeStarClient:
        post_id = m.group(1)
        date_m = _POST_DATE_RE.search(chunk)
        published = _parse_ss_datetime(date_m.group(1)) if date_m else None
-        body_m = _BODY_RE.search(chunk)
-        content = body_m.group(1).strip() if body_m else ""
+        content = _extract_content(chunk)
        return {
            "id": post_id,
            "attributes": {
@@ -368,6 +393,12 @@ class SubscribeStarClient:
                rel = it.get("url")
                if not isinstance(rel, str) or not rel:
                    continue
+                # gallery-dl's _media_from_post: a gallery item whose URL is under
+                # /previews is a locked/blurred TEASER, not the real file — skip it
+                # (the SubscribeStar analog of the Patreon gated-preview bug #874).
+                # This is why a locked post yields no downloadable media.
+                if "/previews" in rel:
+                    continue
                url = urljoin(base + "/", rel)
                media_id = str(it.get("id") or "")
                name = it.get("original_filename")