fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift) #119

Merged

bvandeusen merged 2 commits from dev into main 2026-06-17 15:07:59 -04:00

Conversation 0 Commits 2 Files Changed 2

+105 -7

bvandeusen commented 2026-06-17 15:07:54 -04:00

Owner

Copy Link

Copy Source

What

Fixes the SubscribeStar native live-feed SubscribeStarDriftError ("no posts and no recognizable feed container") seen on creator cheunart.

Root cause (from reading gallery-dl's actual extractor)

gallery-dl's _pagination splits the feed on the generic delimiter <div class="post (trailing space). Our client keyed on <div class="post is-shown — but is-shown is added by SubscribeStar's infinite-scroll JS when a post scrolls into view. It's present in a browser-saved page (what the Step-0 characterization used) but absent from the raw server HTML we and gallery-dl actually fetch → the live feed parsed to zero posts → false drift.

Verified the cookie domain was a red herring: gallery-dl switches cookies_domain to .subscribestar.adult for .adult creators, exactly what our augment_cookies already does.

Changes

• _POST_OPEN → <div class="post (gallery-dl parity). The trailing space rules out the hyphenated post-content/post-date/post-body/post-uploads siblings.
• Mirror gallery-dl's redirect-based gating detection in _get: a redirect to /verify_subscriber or /age_confirmation_warning now raises SubscribeStarAuthError (rotate cookies / age cookie) instead of surfacing as drift.
• Regression tests: raw server markup without is-shown now parses; an age-wall redirect raises auth.

Also carries the prior diagnostic commit (_describe_page naming interstitials in drift errors).

CI green on dev (run 1230).

🤖 Generated with Claude Code

## What Fixes the SubscribeStar native live-feed `SubscribeStarDriftError` ("no posts and no recognizable feed container") seen on creator **cheunart**. ## Root cause (from reading gallery-dl's actual extractor) gallery-dl's `_pagination` splits the feed on the **generic** delimiter `<div class="post ` (trailing space). Our client keyed on `<div class="post is-shown` — but `is-shown` is added by SubscribeStar's **infinite-scroll JS** when a post scrolls into view. It's present in a browser-**saved** page (what the Step-0 characterization used) but **absent** from the raw server HTML we and gallery-dl actually fetch → the live feed parsed to zero posts → false drift. Verified the cookie domain was a red herring: gallery-dl switches `cookies_domain` to `.subscribestar.adult` for `.adult` creators, exactly what our `augment_cookies` already does. ## Changes - `_POST_OPEN` → `<div class="post ` (gallery-dl parity). The trailing space rules out the hyphenated `post-content`/`post-date`/`post-body`/`post-uploads` siblings. - Mirror gallery-dl's redirect-based gating detection in `_get`: a redirect to `/verify_subscriber` or `/age_confirmation_warning` now raises `SubscribeStarAuthError` (rotate cookies / age cookie) instead of surfacing as drift. - Regression tests: raw server markup without `is-shown` now parses; an age-wall redirect raises auth. Also carries the prior diagnostic commit (`_describe_page` naming interstitials in drift errors). CI green on dev (run 1230). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-17 15:07:54 -04:00

diag(subscribestar): name the interstitial in the drift error (title + type) ... 9201b7b539

The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart
still drifts — we're being served a full HTML page that isn't the feed. Add
_describe_page(): the drift error now reports the page <title> + which known
interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha),
so the next run names the actual cause instead of "markup changed". Strong
suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is
UA-locked and our hardcoded UA likely differs from the cookie-capturing browser).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift) ... 204d341a99

The native client split the feed on `<div class="post is-shown`, but `is-shown`
is added by SubscribeStar's infinite-scroll JS when a post scrolls into view —
present in a browser-SAVED page (what the Step-0 characterization used) but
ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live
feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError.

Align with gallery-dl's proven `_pagination`: split on the generic
`<div class="post ` (trailing space rules out the hyphenated post-content/
post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating
detection (/verify_subscriber, /age_confirmation_warning => auth, not drift).

Regression tests: raw server markup without is-shown now parses; an age-wall
redirect raises SubscribeStarAuthError.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit 26827ff38f into main 2026-06-17 15:07:59 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#119