fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate)
After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_ subscriber even with valid .adult cookies (confirmed present: _personalization_id + _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's ENTIRE flow — including the base Extractor._init_session I'd not read — the divergence is the HTTP request profile, not the cookies or parser. gallery-dl's default (cookies-only) mode sends, on EVERY request including the first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same- site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling looked unlike a browser → SubscribeStar gated the adult-creator page. Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept- Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the per-request XHR headers (both feed and load-more now use the shared profile). Test updated to assert the gallery-dl-parity profile instead of the old navigation-vs-XHR split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -79,29 +79,29 @@ def test_parse_ss_datetime_variants():
|
||||
|
||||
# -- client: request headers (regression: live-fetch drift) -----------------
|
||||
|
||||
def test_initial_fetch_is_navigation_loadmore_is_xhr(monkeypatch):
|
||||
# The initial creator-page GET must NOT be flagged XHR (Rails content-
|
||||
# negotiates an XHR full-page request to a non-HTML body → drift; first live
|
||||
# run, cheunart 2026-06-17). Only the "load more" endpoint is a real XHR.
|
||||
def test_request_profile_matches_gallery_dl(monkeypatch):
|
||||
# gallery-dl's default (cookies-only) mode: Firefox UA, Accept: */*, a same-
|
||||
# site Referer (root/), and NO X-Requested-With on EITHER the creator page or
|
||||
# the "load more" JSON endpoint. That profile is what avoids SubscribeStar's
|
||||
# /verify_subscriber 302 with valid cookies (cheunart, 2026-06-17).
|
||||
client = SubscribeStarClient(None)
|
||||
seen = []
|
||||
assert "Firefox" in client._session.headers["User-Agent"]
|
||||
assert client._session.headers["Accept"] == "*/*"
|
||||
assert "X-Requested-With" not in client._session.headers
|
||||
|
||||
class _R:
|
||||
status_code = 200
|
||||
text = '<div data-role="posts_container-list"></div>'
|
||||
history: list = []
|
||||
url = "https://subscribestar.adult/x"
|
||||
text = '<div class="post " data-id="1"></div>'
|
||||
|
||||
def json(self):
|
||||
return {"html": ""}
|
||||
|
||||
def fake_get(url, *, headers=None):
|
||||
seen.append(headers)
|
||||
return _R()
|
||||
|
||||
monkeypatch.setattr(client, "_get", fake_get)
|
||||
client._feed_html("https://subscribestar.adult/x")
|
||||
client._loadmore_html("https://subscribestar.adult/posts?page=2")
|
||||
assert seen[0] is None # navigation — no per-request XHR header
|
||||
assert seen[1]["X-Requested-With"] == "XMLHttpRequest"
|
||||
monkeypatch.setattr(client, "_get", lambda url, *, headers=None: _R())
|
||||
list(client.iter_posts("https://subscribestar.adult/x"))
|
||||
# Referer is stamped to the creator base for the walk.
|
||||
assert client._session.headers["Referer"] == "https://subscribestar.adult/"
|
||||
|
||||
|
||||
# -- client: iteration + pagination -----------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user