fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m18s

After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_
subscriber even with valid .adult cookies (confirmed present: _personalization_id
+ _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's
ENTIRE flow — including the base Extractor._init_session I'd not read — the
divergence is the HTTP request profile, not the cookies or parser.

gallery-dl's default (cookies-only) mode sends, on EVERY request including the
first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same-
site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint
is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling
looked unlike a browser → SubscribeStar gated the adult-creator page.

Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept-
Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the
per-request XHR headers (both feed and load-more now use the shared profile).

Test updated to assert the gallery-dl-parity profile instead of the old
navigation-vs-XHR split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-17 15:28:33 -04:00
parent 204d341a99
commit 559d29fe1c
2 changed files with 42 additions and 29 deletions
+15 -15
View File
@@ -79,29 +79,29 @@ def test_parse_ss_datetime_variants():
# -- client: request headers (regression: live-fetch drift) -----------------
def test_initial_fetch_is_navigation_loadmore_is_xhr(monkeypatch):
# The initial creator-page GET must NOT be flagged XHR (Rails content-
# negotiates an XHR full-page request to a non-HTML body → drift; first live
# run, cheunart 2026-06-17). Only the "load more" endpoint is a real XHR.
def test_request_profile_matches_gallery_dl(monkeypatch):
# gallery-dl's default (cookies-only) mode: Firefox UA, Accept: */*, a same-
# site Referer (root/), and NO X-Requested-With on EITHER the creator page or
# the "load more" JSON endpoint. That profile is what avoids SubscribeStar's
# /verify_subscriber 302 with valid cookies (cheunart, 2026-06-17).
client = SubscribeStarClient(None)
seen = []
assert "Firefox" in client._session.headers["User-Agent"]
assert client._session.headers["Accept"] == "*/*"
assert "X-Requested-With" not in client._session.headers
class _R:
status_code = 200
text = '<div data-role="posts_container-list"></div>'
history: list = []
url = "https://subscribestar.adult/x"
text = '<div class="post " data-id="1"></div>'
def json(self):
return {"html": ""}
def fake_get(url, *, headers=None):
seen.append(headers)
return _R()
monkeypatch.setattr(client, "_get", fake_get)
client._feed_html("https://subscribestar.adult/x")
client._loadmore_html("https://subscribestar.adult/posts?page=2")
assert seen[0] is None # navigation — no per-request XHR header
assert seen[1]["X-Requested-With"] == "XMLHttpRequest"
monkeypatch.setattr(client, "_get", lambda url, *, headers=None: _R())
list(client.iter_posts("https://subscribestar.adult/x"))
# Referer is stamped to the creator base for the walk.
assert client._session.headers["Referer"] == "https://subscribestar.adult/"
# -- client: iteration + pagination -----------------------------------------