feat: SubscribeStar joins the membership roster (387 D1)
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m41s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m41s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
The second platform through the seam note 3970 contracted, characterized first from a live capture of the account's /subscriptions page (note 3989). The capture lives in the gitignored captures dir; the committed fixture is hand-built with invented values and was verified tag-for-tag against it - card wrappers, both table heads, and every distinct row shape - before any code depended on it. What the page is, and the three decisions it forced: The table IS the status. SubscribeStar has no per-row status word: a creator is either in the active_subscriptions card or the cancelled_subscriptions one. The card's data-identifier is stored verbatim as Membership.status and mapped in MEMBERSHIP_STATUS, keyed on the identifier rather than the table class because the cancelled table's class names the same list differently (for-unsubscribed_users). The creator's numeric data-user-id is the key, not the slug. A slug re-keys when a creator renames; the old row stops appearing; and a disappearance is exactly what reconciliation reads as a lapse. Keyed on the slug, a rename would have told a paying subscriber they had cancelled. The slug rides as vanity, where the identity join already looks for a handle. Price is kept as text, never parsed into amount_cents. A bare $ names no currency and a page price is not proven to be the charge - 3970 finding 4. Tier names live behind a per-row modal and are not fetched. Refusals, because SubscribeStar offers nothing like Patreon's meta.pagination.total and every conclusion downstream is drawn from absence. The parser raises when: the active card is missing (auth error on a login/age wall, drift otherwise); a row lacks a numeric creator id or a creator link; anything renders after a card's table; or the page carries a page= link. Both cards are paginatable (app#embed_pagination) and the captured account was too small to show what pagination looks like, so possible pagination is a roster FC cannot prove complete. A loud error on a larger account beats a quiet half-list. A missing cancelled card is not drift, and a creator in both tables is reported once, as active. Fetched from subscribestar.adult, not the .art the capture came from: FC's requests never clear the .art age wall with the 18+ cookie (1259, 1284). Whether /subscriptions on .adult authenticates exactly as .art did in the browser is untested - if not, the sweep records a visible error and C6 shows its unavailable rung. The seam leak D1 found. Note 3970 promised a second platform would be one builders line plus the client method. The sweep instead called current_user_id() on every client, which only Patreon's has, so SubscribeStar would have raised AttributeError on the first sweep. roster_user_id probes it with getattr, the same way the sweep already probes iter_memberships. Two existing tests were passing for the wrong reason and now can fail: - "a platform that has never been characterised says nothing" named SubscribeStar, and stayed green only because active_patron is not a SubscribeStar word. Now uses hentaifoundry, with a positive SubscribeStar test beside it. - the freshness test gave SubscribeStar a Patreon word, so the vocabulary excluded it and deleting the freshness gate outright would have left it green. It now uses cancelled_subscriptions, making the gate the only thing that excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
This commit is contained in:
@@ -43,6 +43,7 @@ import requests
|
||||
from ..utils.paths import filehash_from_url
|
||||
from .native_ingest_common import (
|
||||
_MAX_429_RETRIES,
|
||||
Membership,
|
||||
NativeAuthError,
|
||||
NativeDriftError,
|
||||
NativeIngestError,
|
||||
@@ -297,6 +298,196 @@ def _extract_creator_name(html: str) -> str | None:
|
||||
return name or None
|
||||
|
||||
|
||||
# -- membership roster (#387 D1) ------------------------------------------
|
||||
#
|
||||
# Characterized from a live operator capture of the account's /subscriptions
|
||||
# page, 2026-09-13 — Scribe note #3989. Read that note before changing any of
|
||||
# this; each constant below is a finding from it, not a guess.
|
||||
|
||||
# The account page is fetched from `.adult`. The `.art` age wall never clears
|
||||
# with the 18+ cookie for FC's requests (see _normalize_ss_host, issues #1259 /
|
||||
# #1284). The capture itself came from `.art` only because a human had clicked
|
||||
# through the gate in the browser.
|
||||
_ROSTER_BASE = "https://subscribestar.adult"
|
||||
_ROSTER_URL = f"{_ROSTER_BASE}/subscriptions"
|
||||
|
||||
# Two tables, and WHICH table a creator sits in is the only status the page
|
||||
# gives — there is no per-row status word. Keyed on each card's
|
||||
# `data-identifier`, the one vocabulary that names a state: the table class
|
||||
# inside the cancelled card says `for-unsubscribed_users`, a different word for
|
||||
# the same list (note #3989, CORRECTION 1). The identifier is stored verbatim as
|
||||
# Membership.status and mapped in membership_roster.MEMBERSHIP_STATUS.
|
||||
_ROSTER_ACTIVE = "active_subscriptions"
|
||||
_ROSTER_CANCELLED = "cancelled_subscriptions"
|
||||
|
||||
_ROSTER_ROW_OPEN = '<td class="for-name">'
|
||||
# Active rows nest a second `<tr class="for-actions">` INSIDE the row's own
|
||||
# <tr> — a narrow-screen duplicate of the actions cell. Its <td>s are not
|
||||
# columns, so every row is cut here before its cells are read.
|
||||
_ROSTER_NESTED_ROW = '<tr class="for-actions"'
|
||||
_ROSTER_HREF_RE = re.compile(r'<a href="/([^"/?#]+)"')
|
||||
_ROSTER_USER_ID_RE = re.compile(r'data-user-id="([^"]*)"')
|
||||
_ROSTER_NAME_RE = re.compile(r"<img [^>]*>([^<]*)</div>")
|
||||
_ROSTER_HEAD_RE = re.compile(r'<th class="[^"]*"[^>]*>(.*?)</th>', re.DOTALL)
|
||||
_ROSTER_CELL_RE = re.compile(r'<td class="[^"]*"[^>]*>(.*?)</td>', re.DOTALL)
|
||||
_ROSTER_PAGE_LINK_RE = re.compile(r'href="[^"]*[?&]page=\d')
|
||||
_TAG_RE = re.compile(r"<[^>]+>")
|
||||
|
||||
# Columns that hold identity or controls rather than facts about the
|
||||
# subscription, so they stay out of `details`. Matched on the header's own text,
|
||||
# lowercased — the page's words, not ours.
|
||||
_ROSTER_SKIP_COLUMNS = frozenset({"profile", "updates", "actions"})
|
||||
|
||||
|
||||
def _cell_text(fragment: str) -> str:
|
||||
"""Visible text of a cell: tags dropped, entities decoded, whitespace folded.
|
||||
|
||||
Decoding matters here specifically: an active row with no Discord link
|
||||
renders its cell as the entity `—`, not as an empty cell.
|
||||
"""
|
||||
return " ".join(unescape(_TAG_RE.sub(" ", fragment)).split())
|
||||
|
||||
|
||||
def _roster_table(html: str, identifier: str) -> tuple[str, str] | None:
|
||||
"""One roster card: (its table markup, whatever trails `</table>` inside it).
|
||||
|
||||
None when the card is absent. The trailing part is returned rather than
|
||||
discarded because it is the pagination check: in the characterized page a
|
||||
card closes the moment its table does.
|
||||
"""
|
||||
start = html.find(f'data-identifier="{identifier}"')
|
||||
if start < 0:
|
||||
return None
|
||||
end = html.find("</table>", start)
|
||||
if end < 0:
|
||||
raise SubscribeStarDriftError(
|
||||
f"SubscribeStar roster card {identifier!r} has no table"
|
||||
)
|
||||
close = html.find("</div>", end)
|
||||
trailing = html[end + len("</table>"): close if close >= 0 else len(html)]
|
||||
return html[start:end], trailing
|
||||
|
||||
|
||||
def _roster_rows(table: str, identifier: str, base: str) -> list[Membership]:
|
||||
labels = [_cell_text(h).lower() for h in _ROSTER_HEAD_RE.findall(table)]
|
||||
body = table[table.find("<tbody>"):] if "<tbody>" in table else ""
|
||||
starts = [m.start() for m in re.finditer(re.escape(_ROSTER_ROW_OPEN), body)]
|
||||
rows = []
|
||||
for n, start in enumerate(starts):
|
||||
row = body[start: starts[n + 1] if n + 1 < len(starts) else len(body)]
|
||||
row = row.split(_ROSTER_NESTED_ROW, 1)[0]
|
||||
|
||||
href = _ROSTER_HREF_RE.search(row)
|
||||
if href is None:
|
||||
raise SubscribeStarDriftError(
|
||||
f"SubscribeStar roster row in {identifier!r} has no creator link"
|
||||
)
|
||||
# The creator's numeric id, NOT the slug, is the key (note #3989,
|
||||
# CORRECTION 2). A slug re-keys when a creator renames; the old row then
|
||||
# stops appearing, and a disappearance is exactly what reconciliation
|
||||
# reads as a lapse. The id survives a rename.
|
||||
user_id = _ROSTER_USER_ID_RE.search(row)
|
||||
if user_id is None or not user_id.group(1).isdigit():
|
||||
raise SubscribeStarDriftError(
|
||||
f"SubscribeStar roster row in {identifier!r} has no numeric "
|
||||
f"data-user-id — a membership that cannot be attributed to a "
|
||||
f"creator is not usable"
|
||||
)
|
||||
name = _ROSTER_NAME_RE.search(row)
|
||||
slug = unescape(href.group(1))
|
||||
cells = _ROSTER_CELL_RE.findall(row)
|
||||
rows.append(Membership(
|
||||
campaign_id=user_id.group(1),
|
||||
display_name=(_cell_text(name.group(1)) if name else "") or None,
|
||||
url=f"{base}/{slug}",
|
||||
vanity=slug,
|
||||
status=identifier,
|
||||
# No free-follow concept on this page (#3970 §2: False when a
|
||||
# platform has none).
|
||||
is_free_member=False,
|
||||
# Tier names live behind a per-row modal, not inline. Fetching every
|
||||
# modal would be N authenticated requests for a field nothing reads.
|
||||
tier_names=[],
|
||||
# Deliberately NOT parsed from the price cell: a bare `$` names no
|
||||
# currency, and a page price is not proven to be the charge (#3970
|
||||
# finding 4). None keeps "unknown" distinct from zero. The raw text
|
||||
# is kept in `details`.
|
||||
amount_cents=None,
|
||||
currency=None,
|
||||
details={
|
||||
# Paired with the header text by POSITION: two columns share the
|
||||
# `for-date` class, and the updates column's <td> does not carry
|
||||
# its <th>'s class at all.
|
||||
"columns": {
|
||||
label: _cell_text(cell)
|
||||
for label, cell in zip(labels, cells)
|
||||
if label not in _ROSTER_SKIP_COLUMNS
|
||||
},
|
||||
},
|
||||
))
|
||||
return rows
|
||||
|
||||
|
||||
def parse_subscriptions_page(html: str, *, base: str = _ROSTER_BASE) -> list[Membership]:
|
||||
"""Every membership on the account's /subscriptions page.
|
||||
|
||||
Refuses rather than guessing, because every conclusion downstream is drawn
|
||||
from ABSENCE — a roster that comes back short reads as "you cancelled
|
||||
those". So this raises when:
|
||||
|
||||
* the active card is missing — as SubscribeStarAuthError if the page is a
|
||||
login or age wall (the fix is credentials), otherwise as drift;
|
||||
* a row has no creator link or no numeric creator id;
|
||||
* anything renders after a card's table, or the page carries a `page=` link.
|
||||
Both cards are paginatable (`data-view="app#embed_pagination"`), and the
|
||||
characterized account was too small to show what pagination looks like —
|
||||
so possible pagination is treated as a roster FC cannot prove complete.
|
||||
|
||||
A missing cancelled card is NOT drift: an account that has never cancelled
|
||||
plausibly has no such table. A creator present in both tables is reported
|
||||
once, as active — a current subscription is the fact that matters.
|
||||
"""
|
||||
active = _roster_table(html, _ROSTER_ACTIVE)
|
||||
if active is None:
|
||||
if any(marker in html for marker in _LOGIN_MARKERS):
|
||||
raise SubscribeStarAuthError(
|
||||
"SubscribeStar served a login/age wall instead of the "
|
||||
"subscriptions page (cookies expired or age cookie missing)"
|
||||
)
|
||||
raise SubscribeStarDriftError(
|
||||
f"SubscribeStar subscriptions page has no {_ROSTER_ACTIVE!r} card "
|
||||
f"— {_describe_page(html)}"
|
||||
)
|
||||
roster_region = html[html.find(f'data-identifier="{_ROSTER_ACTIVE}"'):]
|
||||
if _ROSTER_PAGE_LINK_RE.search(roster_region):
|
||||
raise SubscribeStarDriftError(
|
||||
"SubscribeStar subscriptions page carries a page= link — the roster "
|
||||
"may be paginated, and FC cannot prove it is complete (note #3989)"
|
||||
)
|
||||
|
||||
memberships: list[Membership] = []
|
||||
seen: set[str] = set()
|
||||
for identifier, found in (
|
||||
(_ROSTER_ACTIVE, active),
|
||||
(_ROSTER_CANCELLED, _roster_table(html, _ROSTER_CANCELLED)),
|
||||
):
|
||||
if found is None:
|
||||
continue
|
||||
table, trailing = found
|
||||
if trailing.strip():
|
||||
raise SubscribeStarDriftError(
|
||||
f"SubscribeStar roster card {identifier!r} renders content after "
|
||||
f"its table — possibly pagination, so the roster cannot be "
|
||||
f"proven complete (note #3989)"
|
||||
)
|
||||
for membership in _roster_rows(table, identifier, base):
|
||||
if membership.campaign_id in seen:
|
||||
continue
|
||||
seen.add(membership.campaign_id)
|
||||
memberships.append(membership)
|
||||
return memberships
|
||||
|
||||
|
||||
class SubscribeStarClient:
|
||||
"""Synchronous SubscribeStar HTML-scrape read client. Construct with a path
|
||||
to a Netscape cookies.txt (the same file CredentialService.get_cookies_path
|
||||
@@ -645,6 +836,23 @@ class SubscribeStarClient:
|
||||
return None
|
||||
return _extract_creator_name(html)
|
||||
|
||||
# -- membership roster (#387 D1) ----------------------------------------
|
||||
|
||||
def iter_memberships(self, user_id: str | None = None) -> Iterator[Membership]:
|
||||
"""Yield every subscription the account holds (note #3989).
|
||||
|
||||
`user_id` exists for the seam's signature (note #3970) and is ignored:
|
||||
the page is the logged-in account's own, so there is nothing to resolve.
|
||||
The sweep only resolves an id for a client that exposes
|
||||
`current_user_id`, which this one does not.
|
||||
|
||||
One request, and the whole page is parsed before anything is yielded, so
|
||||
a drift error can never leave a caller holding part of a roster.
|
||||
"""
|
||||
self._session.headers["Referer"] = f"{_ROSTER_BASE}/"
|
||||
resp = self._get(_ROSTER_URL)
|
||||
yield from parse_subscriptions_page(resp.text or "", base=_ROSTER_BASE)
|
||||
|
||||
# -- verify ------------------------------------------------------------
|
||||
|
||||
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
|
||||
|
||||
Reference in New Issue
Block a user