feat: SubscribeStar joins the membership roster (387 D1)
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m41s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m41s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
The second platform through the seam note 3970 contracted, characterized first from a live capture of the account's /subscriptions page (note 3989). The capture lives in the gitignored captures dir; the committed fixture is hand-built with invented values and was verified tag-for-tag against it - card wrappers, both table heads, and every distinct row shape - before any code depended on it. What the page is, and the three decisions it forced: The table IS the status. SubscribeStar has no per-row status word: a creator is either in the active_subscriptions card or the cancelled_subscriptions one. The card's data-identifier is stored verbatim as Membership.status and mapped in MEMBERSHIP_STATUS, keyed on the identifier rather than the table class because the cancelled table's class names the same list differently (for-unsubscribed_users). The creator's numeric data-user-id is the key, not the slug. A slug re-keys when a creator renames; the old row stops appearing; and a disappearance is exactly what reconciliation reads as a lapse. Keyed on the slug, a rename would have told a paying subscriber they had cancelled. The slug rides as vanity, where the identity join already looks for a handle. Price is kept as text, never parsed into amount_cents. A bare $ names no currency and a page price is not proven to be the charge - 3970 finding 4. Tier names live behind a per-row modal and are not fetched. Refusals, because SubscribeStar offers nothing like Patreon's meta.pagination.total and every conclusion downstream is drawn from absence. The parser raises when: the active card is missing (auth error on a login/age wall, drift otherwise); a row lacks a numeric creator id or a creator link; anything renders after a card's table; or the page carries a page= link. Both cards are paginatable (app#embed_pagination) and the captured account was too small to show what pagination looks like, so possible pagination is a roster FC cannot prove complete. A loud error on a larger account beats a quiet half-list. A missing cancelled card is not drift, and a creator in both tables is reported once, as active. Fetched from subscribestar.adult, not the .art the capture came from: FC's requests never clear the .art age wall with the 18+ cookie (1259, 1284). Whether /subscriptions on .adult authenticates exactly as .art did in the browser is untested - if not, the sweep records a visible error and C6 shows its unavailable rung. The seam leak D1 found. Note 3970 promised a second platform would be one builders line plus the client method. The sweep instead called current_user_id() on every client, which only Patreon's has, so SubscribeStar would have raised AttributeError on the first sweep. roster_user_id probes it with getattr, the same way the sweep already probes iter_memberships. Two existing tests were passing for the wrong reason and now can fail: - "a platform that has never been characterised says nothing" named SubscribeStar, and stayed green only because active_patron is not a SubscribeStar word. Now uses hentaifoundry, with a positive SubscribeStar test beside it. - the freshness test gave SubscribeStar a Patreon word, so the vocabulary excluded it and deleting the freshness gate outright would have left it green. It now uses cancelled_subscriptions, making the gate the only thing that excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
This commit is contained in:
@@ -60,11 +60,21 @@ log = logging.getLogger(__name__)
|
|||||||
# Unknown words are NOT an error: an unrecognised status means the roster
|
# Unknown words are NOT an error: an unrecognised status means the roster
|
||||||
# records evidence it cannot yet interpret, which is a better state than
|
# records evidence it cannot yet interpret, which is a better state than
|
||||||
# dropping the row or asserting a meaning for it.
|
# dropping the row or asserting a meaning for it.
|
||||||
|
#
|
||||||
|
# subscribestar: from a live capture of the account's /subscriptions page,
|
||||||
|
# 2026-09-13 (Scribe note #3989). SubscribeStar gives NO per-row status word —
|
||||||
|
# a membership's state is which of two tables it sits in — so the "word" stored
|
||||||
|
# is the table card's own `data-identifier`, verbatim. Those two identifiers are
|
||||||
|
# the whole vocabulary; there is nothing further to characterise later.
|
||||||
MEMBERSHIP_STATUS: dict[str, dict[str, bool]] = {
|
MEMBERSHIP_STATUS: dict[str, dict[str, bool]] = {
|
||||||
"patreon": {
|
"patreon": {
|
||||||
"active_patron": True,
|
"active_patron": True,
|
||||||
"former_patron": False,
|
"former_patron": False,
|
||||||
},
|
},
|
||||||
|
"subscribestar": {
|
||||||
|
"active_subscriptions": True,
|
||||||
|
"cancelled_subscriptions": False,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -194,6 +204,24 @@ async def _record_sync(session: AsyncSession, platform: str, **values) -> None:
|
|||||||
))
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def roster_user_id(client) -> str | None:
|
||||||
|
"""The account id a client's roster walk needs, if that client needs one.
|
||||||
|
|
||||||
|
Patreon's members endpoint filters on the account's own user id, so the
|
||||||
|
sweep has to resolve it first. SubscribeStar's /subscriptions page is simply
|
||||||
|
the logged-in account's, with nothing to resolve. Probed with `getattr`,
|
||||||
|
the same way the sweep probes `iter_memberships` itself (rule #169), rather
|
||||||
|
than called unconditionally.
|
||||||
|
|
||||||
|
Calling `current_user_id()` unconditionally was the one place the membership
|
||||||
|
seam was still Patreon-shaped: note #3970 promised a second platform would be
|
||||||
|
one `builders` line plus the client method, and D1 found the sweep would
|
||||||
|
instead have crashed on the first client without that method.
|
||||||
|
"""
|
||||||
|
resolve = getattr(client, "current_user_id", None)
|
||||||
|
return resolve() if resolve is not None else None
|
||||||
|
|
||||||
|
|
||||||
async def sync_platform(
|
async def sync_platform(
|
||||||
session: AsyncSession,
|
session: AsyncSession,
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ import requests
|
|||||||
from ..utils.paths import filehash_from_url
|
from ..utils.paths import filehash_from_url
|
||||||
from .native_ingest_common import (
|
from .native_ingest_common import (
|
||||||
_MAX_429_RETRIES,
|
_MAX_429_RETRIES,
|
||||||
|
Membership,
|
||||||
NativeAuthError,
|
NativeAuthError,
|
||||||
NativeDriftError,
|
NativeDriftError,
|
||||||
NativeIngestError,
|
NativeIngestError,
|
||||||
@@ -297,6 +298,196 @@ def _extract_creator_name(html: str) -> str | None:
|
|||||||
return name or None
|
return name or None
|
||||||
|
|
||||||
|
|
||||||
|
# -- membership roster (#387 D1) ------------------------------------------
|
||||||
|
#
|
||||||
|
# Characterized from a live operator capture of the account's /subscriptions
|
||||||
|
# page, 2026-09-13 — Scribe note #3989. Read that note before changing any of
|
||||||
|
# this; each constant below is a finding from it, not a guess.
|
||||||
|
|
||||||
|
# The account page is fetched from `.adult`. The `.art` age wall never clears
|
||||||
|
# with the 18+ cookie for FC's requests (see _normalize_ss_host, issues #1259 /
|
||||||
|
# #1284). The capture itself came from `.art` only because a human had clicked
|
||||||
|
# through the gate in the browser.
|
||||||
|
_ROSTER_BASE = "https://subscribestar.adult"
|
||||||
|
_ROSTER_URL = f"{_ROSTER_BASE}/subscriptions"
|
||||||
|
|
||||||
|
# Two tables, and WHICH table a creator sits in is the only status the page
|
||||||
|
# gives — there is no per-row status word. Keyed on each card's
|
||||||
|
# `data-identifier`, the one vocabulary that names a state: the table class
|
||||||
|
# inside the cancelled card says `for-unsubscribed_users`, a different word for
|
||||||
|
# the same list (note #3989, CORRECTION 1). The identifier is stored verbatim as
|
||||||
|
# Membership.status and mapped in membership_roster.MEMBERSHIP_STATUS.
|
||||||
|
_ROSTER_ACTIVE = "active_subscriptions"
|
||||||
|
_ROSTER_CANCELLED = "cancelled_subscriptions"
|
||||||
|
|
||||||
|
_ROSTER_ROW_OPEN = '<td class="for-name">'
|
||||||
|
# Active rows nest a second `<tr class="for-actions">` INSIDE the row's own
|
||||||
|
# <tr> — a narrow-screen duplicate of the actions cell. Its <td>s are not
|
||||||
|
# columns, so every row is cut here before its cells are read.
|
||||||
|
_ROSTER_NESTED_ROW = '<tr class="for-actions"'
|
||||||
|
_ROSTER_HREF_RE = re.compile(r'<a href="/([^"/?#]+)"')
|
||||||
|
_ROSTER_USER_ID_RE = re.compile(r'data-user-id="([^"]*)"')
|
||||||
|
_ROSTER_NAME_RE = re.compile(r"<img [^>]*>([^<]*)</div>")
|
||||||
|
_ROSTER_HEAD_RE = re.compile(r'<th class="[^"]*"[^>]*>(.*?)</th>', re.DOTALL)
|
||||||
|
_ROSTER_CELL_RE = re.compile(r'<td class="[^"]*"[^>]*>(.*?)</td>', re.DOTALL)
|
||||||
|
_ROSTER_PAGE_LINK_RE = re.compile(r'href="[^"]*[?&]page=\d')
|
||||||
|
_TAG_RE = re.compile(r"<[^>]+>")
|
||||||
|
|
||||||
|
# Columns that hold identity or controls rather than facts about the
|
||||||
|
# subscription, so they stay out of `details`. Matched on the header's own text,
|
||||||
|
# lowercased — the page's words, not ours.
|
||||||
|
_ROSTER_SKIP_COLUMNS = frozenset({"profile", "updates", "actions"})
|
||||||
|
|
||||||
|
|
||||||
|
def _cell_text(fragment: str) -> str:
|
||||||
|
"""Visible text of a cell: tags dropped, entities decoded, whitespace folded.
|
||||||
|
|
||||||
|
Decoding matters here specifically: an active row with no Discord link
|
||||||
|
renders its cell as the entity `—`, not as an empty cell.
|
||||||
|
"""
|
||||||
|
return " ".join(unescape(_TAG_RE.sub(" ", fragment)).split())
|
||||||
|
|
||||||
|
|
||||||
|
def _roster_table(html: str, identifier: str) -> tuple[str, str] | None:
|
||||||
|
"""One roster card: (its table markup, whatever trails `</table>` inside it).
|
||||||
|
|
||||||
|
None when the card is absent. The trailing part is returned rather than
|
||||||
|
discarded because it is the pagination check: in the characterized page a
|
||||||
|
card closes the moment its table does.
|
||||||
|
"""
|
||||||
|
start = html.find(f'data-identifier="{identifier}"')
|
||||||
|
if start < 0:
|
||||||
|
return None
|
||||||
|
end = html.find("</table>", start)
|
||||||
|
if end < 0:
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
f"SubscribeStar roster card {identifier!r} has no table"
|
||||||
|
)
|
||||||
|
close = html.find("</div>", end)
|
||||||
|
trailing = html[end + len("</table>"): close if close >= 0 else len(html)]
|
||||||
|
return html[start:end], trailing
|
||||||
|
|
||||||
|
|
||||||
|
def _roster_rows(table: str, identifier: str, base: str) -> list[Membership]:
|
||||||
|
labels = [_cell_text(h).lower() for h in _ROSTER_HEAD_RE.findall(table)]
|
||||||
|
body = table[table.find("<tbody>"):] if "<tbody>" in table else ""
|
||||||
|
starts = [m.start() for m in re.finditer(re.escape(_ROSTER_ROW_OPEN), body)]
|
||||||
|
rows = []
|
||||||
|
for n, start in enumerate(starts):
|
||||||
|
row = body[start: starts[n + 1] if n + 1 < len(starts) else len(body)]
|
||||||
|
row = row.split(_ROSTER_NESTED_ROW, 1)[0]
|
||||||
|
|
||||||
|
href = _ROSTER_HREF_RE.search(row)
|
||||||
|
if href is None:
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
f"SubscribeStar roster row in {identifier!r} has no creator link"
|
||||||
|
)
|
||||||
|
# The creator's numeric id, NOT the slug, is the key (note #3989,
|
||||||
|
# CORRECTION 2). A slug re-keys when a creator renames; the old row then
|
||||||
|
# stops appearing, and a disappearance is exactly what reconciliation
|
||||||
|
# reads as a lapse. The id survives a rename.
|
||||||
|
user_id = _ROSTER_USER_ID_RE.search(row)
|
||||||
|
if user_id is None or not user_id.group(1).isdigit():
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
f"SubscribeStar roster row in {identifier!r} has no numeric "
|
||||||
|
f"data-user-id — a membership that cannot be attributed to a "
|
||||||
|
f"creator is not usable"
|
||||||
|
)
|
||||||
|
name = _ROSTER_NAME_RE.search(row)
|
||||||
|
slug = unescape(href.group(1))
|
||||||
|
cells = _ROSTER_CELL_RE.findall(row)
|
||||||
|
rows.append(Membership(
|
||||||
|
campaign_id=user_id.group(1),
|
||||||
|
display_name=(_cell_text(name.group(1)) if name else "") or None,
|
||||||
|
url=f"{base}/{slug}",
|
||||||
|
vanity=slug,
|
||||||
|
status=identifier,
|
||||||
|
# No free-follow concept on this page (#3970 §2: False when a
|
||||||
|
# platform has none).
|
||||||
|
is_free_member=False,
|
||||||
|
# Tier names live behind a per-row modal, not inline. Fetching every
|
||||||
|
# modal would be N authenticated requests for a field nothing reads.
|
||||||
|
tier_names=[],
|
||||||
|
# Deliberately NOT parsed from the price cell: a bare `$` names no
|
||||||
|
# currency, and a page price is not proven to be the charge (#3970
|
||||||
|
# finding 4). None keeps "unknown" distinct from zero. The raw text
|
||||||
|
# is kept in `details`.
|
||||||
|
amount_cents=None,
|
||||||
|
currency=None,
|
||||||
|
details={
|
||||||
|
# Paired with the header text by POSITION: two columns share the
|
||||||
|
# `for-date` class, and the updates column's <td> does not carry
|
||||||
|
# its <th>'s class at all.
|
||||||
|
"columns": {
|
||||||
|
label: _cell_text(cell)
|
||||||
|
for label, cell in zip(labels, cells)
|
||||||
|
if label not in _ROSTER_SKIP_COLUMNS
|
||||||
|
},
|
||||||
|
},
|
||||||
|
))
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def parse_subscriptions_page(html: str, *, base: str = _ROSTER_BASE) -> list[Membership]:
|
||||||
|
"""Every membership on the account's /subscriptions page.
|
||||||
|
|
||||||
|
Refuses rather than guessing, because every conclusion downstream is drawn
|
||||||
|
from ABSENCE — a roster that comes back short reads as "you cancelled
|
||||||
|
those". So this raises when:
|
||||||
|
|
||||||
|
* the active card is missing — as SubscribeStarAuthError if the page is a
|
||||||
|
login or age wall (the fix is credentials), otherwise as drift;
|
||||||
|
* a row has no creator link or no numeric creator id;
|
||||||
|
* anything renders after a card's table, or the page carries a `page=` link.
|
||||||
|
Both cards are paginatable (`data-view="app#embed_pagination"`), and the
|
||||||
|
characterized account was too small to show what pagination looks like —
|
||||||
|
so possible pagination is treated as a roster FC cannot prove complete.
|
||||||
|
|
||||||
|
A missing cancelled card is NOT drift: an account that has never cancelled
|
||||||
|
plausibly has no such table. A creator present in both tables is reported
|
||||||
|
once, as active — a current subscription is the fact that matters.
|
||||||
|
"""
|
||||||
|
active = _roster_table(html, _ROSTER_ACTIVE)
|
||||||
|
if active is None:
|
||||||
|
if any(marker in html for marker in _LOGIN_MARKERS):
|
||||||
|
raise SubscribeStarAuthError(
|
||||||
|
"SubscribeStar served a login/age wall instead of the "
|
||||||
|
"subscriptions page (cookies expired or age cookie missing)"
|
||||||
|
)
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
f"SubscribeStar subscriptions page has no {_ROSTER_ACTIVE!r} card "
|
||||||
|
f"— {_describe_page(html)}"
|
||||||
|
)
|
||||||
|
roster_region = html[html.find(f'data-identifier="{_ROSTER_ACTIVE}"'):]
|
||||||
|
if _ROSTER_PAGE_LINK_RE.search(roster_region):
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
"SubscribeStar subscriptions page carries a page= link — the roster "
|
||||||
|
"may be paginated, and FC cannot prove it is complete (note #3989)"
|
||||||
|
)
|
||||||
|
|
||||||
|
memberships: list[Membership] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for identifier, found in (
|
||||||
|
(_ROSTER_ACTIVE, active),
|
||||||
|
(_ROSTER_CANCELLED, _roster_table(html, _ROSTER_CANCELLED)),
|
||||||
|
):
|
||||||
|
if found is None:
|
||||||
|
continue
|
||||||
|
table, trailing = found
|
||||||
|
if trailing.strip():
|
||||||
|
raise SubscribeStarDriftError(
|
||||||
|
f"SubscribeStar roster card {identifier!r} renders content after "
|
||||||
|
f"its table — possibly pagination, so the roster cannot be "
|
||||||
|
f"proven complete (note #3989)"
|
||||||
|
)
|
||||||
|
for membership in _roster_rows(table, identifier, base):
|
||||||
|
if membership.campaign_id in seen:
|
||||||
|
continue
|
||||||
|
seen.add(membership.campaign_id)
|
||||||
|
memberships.append(membership)
|
||||||
|
return memberships
|
||||||
|
|
||||||
|
|
||||||
class SubscribeStarClient:
|
class SubscribeStarClient:
|
||||||
"""Synchronous SubscribeStar HTML-scrape read client. Construct with a path
|
"""Synchronous SubscribeStar HTML-scrape read client. Construct with a path
|
||||||
to a Netscape cookies.txt (the same file CredentialService.get_cookies_path
|
to a Netscape cookies.txt (the same file CredentialService.get_cookies_path
|
||||||
@@ -645,6 +836,23 @@ class SubscribeStarClient:
|
|||||||
return None
|
return None
|
||||||
return _extract_creator_name(html)
|
return _extract_creator_name(html)
|
||||||
|
|
||||||
|
# -- membership roster (#387 D1) ----------------------------------------
|
||||||
|
|
||||||
|
def iter_memberships(self, user_id: str | None = None) -> Iterator[Membership]:
|
||||||
|
"""Yield every subscription the account holds (note #3989).
|
||||||
|
|
||||||
|
`user_id` exists for the seam's signature (note #3970) and is ignored:
|
||||||
|
the page is the logged-in account's own, so there is nothing to resolve.
|
||||||
|
The sweep only resolves an id for a client that exposes
|
||||||
|
`current_user_id`, which this one does not.
|
||||||
|
|
||||||
|
One request, and the whole page is parsed before anything is yielded, so
|
||||||
|
a drift error can never leave a caller holding part of a roster.
|
||||||
|
"""
|
||||||
|
self._session.headers["Referer"] = f"{_ROSTER_BASE}/"
|
||||||
|
resp = self._get(_ROSTER_URL)
|
||||||
|
yield from parse_subscriptions_page(resp.text or "", base=_ROSTER_BASE)
|
||||||
|
|
||||||
# -- verify ------------------------------------------------------------
|
# -- verify ------------------------------------------------------------
|
||||||
|
|
||||||
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
|
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
|
||||||
|
|||||||
@@ -1247,17 +1247,20 @@ def sync_memberships() -> str:
|
|||||||
from ..services.artist_membership_service import rescan as membership_rescan
|
from ..services.artist_membership_service import rescan as membership_rescan
|
||||||
from ..services.credential_crypto import CredentialCrypto
|
from ..services.credential_crypto import CredentialCrypto
|
||||||
from ..services.credential_service import CredentialService
|
from ..services.credential_service import CredentialService
|
||||||
from ..services.membership_roster import sync_platform
|
from ..services.membership_roster import roster_user_id, sync_platform
|
||||||
from ..services.patreon_client import PatreonClient
|
from ..services.patreon_client import PatreonClient
|
||||||
|
from ..services.subscribestar_client import SubscribeStarClient
|
||||||
from ._async_session import async_session_factory
|
from ._async_session import async_session_factory
|
||||||
|
|
||||||
key_path = IMAGES_ROOT / "secrets" / "credential_key.b64"
|
key_path = IMAGES_ROOT / "secrets" / "credential_key.b64"
|
||||||
|
|
||||||
# platform -> how to build a client from a cookies path. A platform is in
|
# platform -> how to build a client from a cookies path. A platform is in
|
||||||
# the sweep only if it is here AND its client exposes `iter_memberships`
|
# the sweep only if it is here AND its client exposes `iter_memberships`
|
||||||
# AND a credential exists — three independent gates, each silent, so
|
# AND a credential exists — three independent gates, each silent, so a
|
||||||
# adding SubscribeStar (D1) is one line here and nothing else.
|
# platform is added with one line here and nothing else. SubscribeStar (D1)
|
||||||
builders = {"patreon": PatreonClient}
|
# was the second; the only other change it needed was `roster_user_id`
|
||||||
|
# replacing an unconditional Patreon-only call below.
|
||||||
|
builders = {"patreon": PatreonClient, "subscribestar": SubscribeStarClient}
|
||||||
|
|
||||||
async def _run() -> dict:
|
async def _run() -> dict:
|
||||||
async_factory, engine = async_session_factory()
|
async_factory, engine = async_session_factory()
|
||||||
@@ -1287,8 +1290,7 @@ def sync_memberships() -> str:
|
|||||||
# slow roster does not block the event loop, and bound the
|
# slow roster does not block the event loop, and bound the
|
||||||
# whole walk rather than only its individual requests.
|
# whole walk rather than only its individual requests.
|
||||||
def _walk():
|
def _walk():
|
||||||
user_id = _client.current_user_id()
|
return list(_client.iter_memberships(roster_user_id(_client)))
|
||||||
return list(_client.iter_memberships(user_id))
|
|
||||||
|
|
||||||
return await asyncio.wait_for(
|
return await asyncio.wait_for(
|
||||||
asyncio.to_thread(_walk),
|
asyncio.to_thread(_walk),
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
<!DOCTYPE html><html><head><title>My Subscriptions | SubscribeStar.adult</title></head><body><div class="layout"><div class="section-body"><div class="warnings-wrapper"></div><div class="card for-table" data-identifier="active_subscriptions" data-view="app#embed_pagination" id="subscriptions_active"><h2 class="card-title">Active Subscriptions</h2><table class="details_table for-active_subscriptions"><thead><tr><th class="for-name">Profile</th><th class="for-updates">Updates</th><th class="for-date">Subscribed</th><th class="for-renewal">Renewed</th><th class="for-date">Paused</th><th class="for-earnings for-subscription_price">Price</th><th class="for-discord">Discord</th><th class="for-actions for-large_screen">Actions</th></tr></thead><tbody><tr><td class="for-name"><a href="/creator-alpha"><div class="inline_user_name"><img data-view="app#avatar" data-type="avatar" data-user-id="1001" alt="Creator Alpha" src="https://cdn.example.invalid/avatars/1001.jpg" />Creator Alpha</div> </a></td><td class="for-actions has-icons"><button class="details_table-link for-updates-unsubscribe" data-modal="/creator-alpha/unsubscribe" title="Unsubscribe from post updates">Mute updates</button></td><td class="for-date">Jan 2026</td><td class="for-renewal">Mar 2026</td><td class="for-date">-</td><td class="for-earnings for-subscription_price"><span class="subscription_cost">$5</span><span class="for-tiers-list" data-modal="/creator-alpha/tier_details" title="Show tiers"><svg class="icon"><use></use></svg>
|
||||||
|
</span></td><td class="for-discord"><button class="details_table-link" data-modal="/creator-alpha/discord"><i class="md_icon is-fa_discord"><svg class="icon"><use></use></svg> </i><span>Joined</span></button></td><td class="for-actions is-multiple for-large_screen"><button class="details_table-link" data-modal="/creator-alpha/manage">Manage plan</button></td><tr class="for-actions"><td class="is-inline" colspan="100"><small>Manage options</small><button class="details_table-link" data-modal="/creator-alpha/manage">Manage plan</button></td></tr></tr><tr><td class="for-name"><a href="/creator-beta"><div class="inline_user_name"><img data-view="app#avatar" data-type="avatar" data-user-id="2002002" alt="Creator Beta Studio" src="https://cdn.example.invalid/avatars/2002002.jpg" />Creator Beta Studio</div> </a></td><td class="for-actions has-icons"><button class="details_table-link for-updates-unsubscribe" data-modal="/creator-beta/unsubscribe" title="Unsubscribe from post updates">Mute updates</button></td><td class="for-date">Feb 2025</td><td class="for-renewal">Mar 2026</td><td class="for-date">-</td><td class="for-earnings for-subscription_price"><span class="subscription_cost">$12</span><span class="for-tiers-list" data-modal="/creator-beta/tier_details" title="Show tiers"><svg class="icon"><use></use></svg>
|
||||||
|
</span></td><td class="for-discord">—</td><td class="for-actions is-multiple for-large_screen"><button class="details_table-link" data-modal="/creator-beta/manage">Manage plan</button></td><tr class="for-actions"><td class="is-inline" colspan="100"><small>Manage options</small><button class="details_table-link" data-modal="/creator-beta/manage">Manage plan</button></td></tr></tr></tbody></table></div><div class="card for-table" data-identifier="cancelled_subscriptions" data-view="app#embed_pagination" id="subscriptions_cancelled"><h2 class="card-title">Cancelled subscriptions</h2><table class="details_table for-unsubscribed_users"><thead><tr><th class="for-name">Profile</th><th class="for-date">Unsubscribed</th><th class="for-earnings">Price</th><th class="for-actions">Actions</th></tr></thead><tbody><tr><td class="for-name"><a href="/creator-gamma"><div class="inline_user_name"><img data-view="app#avatar" data-type="avatar" data-user-id="3003003" alt="Creator Gamma" src="https://cdn.example.invalid/avatars/3003003.jpg" />Creator Gamma</div> </a></td><td class="for-date">Dec 2025</td><td class="for-earnings">$3</td><td class="for-actions"><button class="details_table-link" data-modal="/creator-gamma/resubscribe">Renew</button></td></tr><tr><td class="for-name"><a href="/creator-delta"><div class="inline_user_name"><img data-view="app#avatar" data-type="avatar" data-user-id="4004" alt="Creator Delta" src="https://cdn.example.invalid/avatars/4004.jpg" />Creator Delta</div> </a></td><td class="for-date">Aug 2025</td><td class="for-earnings">$10</td><td class="for-actions"><button class="details_table-link" data-modal="/creator-delta/resubscribe">Renew</button></td></tr></tbody></table></div></div></div></body></html>
|
||||||
@@ -74,8 +74,22 @@ def test_an_unrecognised_status_says_nothing(status):
|
|||||||
|
|
||||||
|
|
||||||
def test_a_platform_that_has_never_been_characterised_says_nothing():
|
def test_a_platform_that_has_never_been_characterised_says_nothing():
|
||||||
"""SubscribeStar and FANBOX (D1) inherit silence, not a Patreon guess."""
|
"""An uncharacterised platform inherits silence, not a Patreon guess.
|
||||||
assert gated_reason("subscribestar", "active_patron") is None
|
|
||||||
|
This named SubscribeStar until D1 characterised it (note #3989). Left as it
|
||||||
|
was, it would have kept passing only because `active_patron` is not a
|
||||||
|
SubscribeStar word, testing nothing its name claims.
|
||||||
|
"""
|
||||||
|
assert gated_reason("hentaifoundry", "active_patron") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_subscribestar_explains_the_gate_in_its_own_words():
|
||||||
|
"""SubscribeStar's status is which table the creator sits in, and those
|
||||||
|
identifiers reach the same three reasons as Patreon's words."""
|
||||||
|
assert gated_reason("subscribestar", "active_subscriptions") == GATED_TIER
|
||||||
|
assert gated_reason("subscribestar", "cancelled_subscriptions") == GATED_LAPSED
|
||||||
|
# Patreon's vocabulary does not leak across platforms.
|
||||||
|
assert gated_reason("subscribestar", "former_patron") is None
|
||||||
|
|
||||||
|
|
||||||
# --- the join, against the database ----------------------------------------
|
# --- the join, against the database ----------------------------------------
|
||||||
@@ -178,8 +192,11 @@ async def test_one_platforms_fresh_sweep_does_not_vouch_for_anothers(db):
|
|||||||
url="https://subscribestar.adult/maewix",
|
url="https://subscribestar.adult/maewix",
|
||||||
)
|
)
|
||||||
await _membership(db, status="former_patron")
|
await _membership(db, status="former_patron")
|
||||||
|
# SubscribeStar's REAL word, so that the freshness gate is the only thing
|
||||||
|
# excluding this membership. With a Patreon word (as this test first had),
|
||||||
|
# the vocabulary excluded it and removing the gate would have left it green.
|
||||||
await _membership(db, platform="subscribestar", campaign="s1",
|
await _membership(db, platform="subscribestar", campaign="s1",
|
||||||
status="former_patron",
|
status="cancelled_subscriptions",
|
||||||
url="https://subscribestar.adult/maewix",
|
url="https://subscribestar.adult/maewix",
|
||||||
details={"campaign": {"vanity": "maewix"}})
|
details={"campaign": {"vanity": "maewix"}})
|
||||||
await _synced(db) # patreon only
|
await _synced(db) # patreon only
|
||||||
|
|||||||
@@ -170,8 +170,12 @@ def test_the_status_map_contains_only_characterised_values():
|
|||||||
contain. Adding it because it "obviously" belongs is precisely the guess
|
contain. Adding it because it "obviously" belongs is precisely the guess
|
||||||
this test exists to stop.
|
this test exists to stop.
|
||||||
"""
|
"""
|
||||||
|
# subscribestar added with its own capture (note #3989, 2026-09-13). Its
|
||||||
|
# "words" are the two table identifiers, because the page carries no
|
||||||
|
# per-row status at all.
|
||||||
assert MEMBERSHIP_STATUS == {
|
assert MEMBERSHIP_STATUS == {
|
||||||
"patreon": {"active_patron": True, "former_patron": False},
|
"patreon": {"active_patron": True, "former_patron": False},
|
||||||
|
"subscribestar": {"active_subscriptions": True, "cancelled_subscriptions": False},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,261 @@
|
|||||||
|
"""SubscribeStarClient.iter_memberships — parsing only, no network (#387 D1).
|
||||||
|
|
||||||
|
The fixture mirrors a REAL capture of the operator's /subscriptions page
|
||||||
|
(Scribe note #3989) with every value invented. Its structure was checked
|
||||||
|
against the capture tag for tag before this file was written: the card
|
||||||
|
wrappers, both table heads, and every distinct row shape — including the two
|
||||||
|
forms the Discord cell takes, and the narrow-screen `<tr class="for-actions">`
|
||||||
|
that active rows nest INSIDE their own row.
|
||||||
|
|
||||||
|
Most of what follows pins refusals. SubscribeStar gives no completeness signal
|
||||||
|
the way Patreon's `meta.pagination.total` does, and every conclusion drawn from
|
||||||
|
this roster is drawn from absence, so the parser's job is to raise whenever it
|
||||||
|
cannot vouch for the whole list.
|
||||||
|
|
||||||
|
`_get` is stubbed rather than mocked at the socket: these tests are about what
|
||||||
|
the client does with a page, and the HTTP path is covered by
|
||||||
|
test_subscribestar_client.py.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from backend.app.services.membership_roster import has_paid_access, roster_user_id
|
||||||
|
from backend.app.services.native_ingest_common import Membership
|
||||||
|
from backend.app.services.subscribestar_client import (
|
||||||
|
SubscribeStarAuthError,
|
||||||
|
SubscribeStarClient,
|
||||||
|
SubscribeStarDriftError,
|
||||||
|
parse_subscriptions_page,
|
||||||
|
)
|
||||||
|
|
||||||
|
_FIXTURE = Path(__file__).parent / "fixtures" / "subscribestar_subscriptions_page1.html"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def page():
|
||||||
|
return _FIXTURE.read_text()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client():
|
||||||
|
return SubscribeStarClient(cookies_path=None)
|
||||||
|
|
||||||
|
|
||||||
|
def _serve(client, html):
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake(url, *, headers=None):
|
||||||
|
calls.append(url)
|
||||||
|
return SimpleNamespace(text=html)
|
||||||
|
|
||||||
|
client._get = fake
|
||||||
|
return calls
|
||||||
|
|
||||||
|
|
||||||
|
def _by_id(memberships):
|
||||||
|
return {m.campaign_id: m for m in memberships}
|
||||||
|
|
||||||
|
|
||||||
|
# --- the request -----------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_roster_is_fetched_from_adult_not_art(client, page):
|
||||||
|
"""The capture came from `.art`, but FC's requests never clear the `.art`
|
||||||
|
age wall with the 18+ cookie (#1259, #1284). Copying the browser's host is
|
||||||
|
the obvious move and would turn every sweep into an auth error."""
|
||||||
|
calls = _serve(client, page)
|
||||||
|
list(client.iter_memberships())
|
||||||
|
assert calls == ["https://subscribestar.adult/subscriptions"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_request_for_the_whole_roster(client, page):
|
||||||
|
calls = _serve(client, page)
|
||||||
|
list(client.iter_memberships(user_id="ignored"))
|
||||||
|
assert len(calls) == 1
|
||||||
|
|
||||||
|
|
||||||
|
# --- what a membership contains --------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_both_tables_are_read(page):
|
||||||
|
rows = parse_subscriptions_page(page)
|
||||||
|
assert all(isinstance(m, Membership) for m in rows)
|
||||||
|
assert len(rows) == 4
|
||||||
|
statuses = sorted(m.status for m in rows)
|
||||||
|
assert statuses == [
|
||||||
|
"active_subscriptions", "active_subscriptions",
|
||||||
|
"cancelled_subscriptions", "cancelled_subscriptions",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_creator_id_is_the_key_not_the_slug(page):
|
||||||
|
"""A slug re-keys when a creator renames; the old row then stops appearing,
|
||||||
|
and reconciliation reads a disappearance as a lapse. The numeric id does not
|
||||||
|
change, so a rename can never surface as "you cancelled this creator"
|
||||||
|
(note #3989, CORRECTION 2)."""
|
||||||
|
rows = _by_id(parse_subscriptions_page(page))
|
||||||
|
assert set(rows) == {"1001", "2002002", "3003003", "4004"}
|
||||||
|
alpha = rows["1001"]
|
||||||
|
assert alpha.vanity == "creator-alpha"
|
||||||
|
# The handle is carried as a URL so `match_kind`'s URL-tail fallback reaches
|
||||||
|
# it — which is what a SubscribeStar Source.url ends in.
|
||||||
|
assert alpha.url == "https://subscribestar.adult/creator-alpha"
|
||||||
|
assert alpha.display_name == "Creator Alpha"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_table_is_the_status(page):
|
||||||
|
"""No per-row status word exists. The card's `data-identifier` is stored
|
||||||
|
verbatim — and is exactly what MEMBERSHIP_STATUS maps, so a renamed
|
||||||
|
identifier fails here rather than silently reading as unknown."""
|
||||||
|
rows = _by_id(parse_subscriptions_page(page))
|
||||||
|
assert rows["1001"].status == "active_subscriptions"
|
||||||
|
assert rows["3003003"].status == "cancelled_subscriptions"
|
||||||
|
assert has_paid_access("subscribestar", rows["1001"].status) is True
|
||||||
|
assert has_paid_access("subscribestar", rows["3003003"].status) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_price_is_kept_as_text_and_never_parsed_into_money(page):
|
||||||
|
"""A bare `$` names no currency and a page price is not proven to be the
|
||||||
|
charge. None keeps "unknown" distinct from zero (#3970 finding 4)."""
|
||||||
|
rows = _by_id(parse_subscriptions_page(page))
|
||||||
|
alpha = rows["1001"]
|
||||||
|
assert alpha.amount_cents is None
|
||||||
|
assert alpha.currency is None
|
||||||
|
assert alpha.details["columns"]["price"] == "$5"
|
||||||
|
assert alpha.tier_names == []
|
||||||
|
assert alpha.is_free_member is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_columns_are_paired_by_position_not_by_class(page):
|
||||||
|
"""Two active columns share the `for-date` class, and the updates column's
|
||||||
|
<td> does not carry its <th>'s class — pairing by class would merge or drop
|
||||||
|
them. Controls and identity stay out of details."""
|
||||||
|
cols = _by_id(parse_subscriptions_page(page))["1001"].details["columns"]
|
||||||
|
assert cols == {
|
||||||
|
"subscribed": "Jan 2026",
|
||||||
|
"renewed": "Mar 2026",
|
||||||
|
"paused": "-",
|
||||||
|
"price": "$5",
|
||||||
|
"discord": "Joined",
|
||||||
|
}
|
||||||
|
cancelled = _by_id(parse_subscriptions_page(page))["3003003"].details["columns"]
|
||||||
|
assert cancelled == {"unsubscribed": "Dec 2025", "price": "$3"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_both_discord_cell_forms_read_as_text(page):
|
||||||
|
"""The real page renders a linked Discord as a button with an icon, and an
|
||||||
|
unlinked one as the ENTITY `—`. Undecoded, the second would store the
|
||||||
|
literal string "—"."""
|
||||||
|
rows = _by_id(parse_subscriptions_page(page))
|
||||||
|
assert rows["1001"].details["columns"]["discord"] == "Joined"
|
||||||
|
assert rows["2002002"].details["columns"]["discord"] == "—"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_nested_actions_row_is_not_read_as_columns(page):
|
||||||
|
"""Active rows nest a narrow-screen `<tr class="for-actions">` inside their
|
||||||
|
own row. Its cells would otherwise zip onto the header as extra columns."""
|
||||||
|
cols = _by_id(parse_subscriptions_page(page))["2002002"].details["columns"]
|
||||||
|
assert "Manage options" not in cols.values()
|
||||||
|
assert set(cols) == {"subscribed", "renewed", "paused", "price", "discord"}
|
||||||
|
|
||||||
|
|
||||||
|
# --- refusals --------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_creator_in_both_tables_is_reported_once_as_active(page):
|
||||||
|
"""A current subscription is the fact that matters. Written the other way
|
||||||
|
round, the cancelled row would overwrite the active one in the upsert and
|
||||||
|
tell a paying subscriber they had lapsed."""
|
||||||
|
duplicated = page.replace('data-user-id="3003003"', 'data-user-id="1001"')
|
||||||
|
rows = parse_subscriptions_page(duplicated)
|
||||||
|
matches = [m for m in rows if m.campaign_id == "1001"]
|
||||||
|
assert len(matches) == 1
|
||||||
|
assert matches[0].status == "active_subscriptions"
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_account_that_never_cancelled_has_no_cancelled_card(page):
|
||||||
|
"""Not drift — such an account plausibly renders no cancelled table."""
|
||||||
|
start = page.index('<div class="card for-table" data-identifier="cancelled_subscriptions"')
|
||||||
|
end = page.index("</table></div>", start) + len("</table></div>")
|
||||||
|
rows = parse_subscriptions_page(page[:start] + page[end:])
|
||||||
|
assert {m.status for m in rows} == {"active_subscriptions"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_login_wall_is_auth_not_drift():
|
||||||
|
"""The fix for this is a fresh credential, not a scraper change — and the
|
||||||
|
distinction survives into membership_sync.last_error_type, where it is what
|
||||||
|
makes the UI's advice correct."""
|
||||||
|
wall = '<html><body><form action="/session/new" data-role="sign_in"></form></body></html>'
|
||||||
|
with pytest.raises(SubscribeStarAuthError):
|
||||||
|
parse_subscriptions_page(wall)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unrecognised_page_is_drift():
|
||||||
|
with pytest.raises(SubscribeStarDriftError):
|
||||||
|
parse_subscriptions_page("<html><title>Something else</title><body></body></html>")
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_row_without_a_creator_id_refuses_the_whole_roster(page):
|
||||||
|
"""Identity is the point (#3970 §4). Skipping the row instead would return a
|
||||||
|
roster one creator short, which downstream reads as a cancellation."""
|
||||||
|
with pytest.raises(SubscribeStarDriftError, match="data-user-id"):
|
||||||
|
parse_subscriptions_page(page.replace(' data-user-id="2002002"', ""))
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_non_numeric_creator_id_is_drift(page):
|
||||||
|
with pytest.raises(SubscribeStarDriftError, match="data-user-id"):
|
||||||
|
parse_subscriptions_page(page.replace('data-user-id="2002002"', 'data-user-id="beta"'))
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_row_without_a_creator_link_is_drift(page):
|
||||||
|
with pytest.raises(SubscribeStarDriftError, match="creator link"):
|
||||||
|
parse_subscriptions_page(page.replace('<a href="/creator-gamma">', "<a>"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("card", ["active_subscriptions", "cancelled_subscriptions"])
|
||||||
|
def test_content_after_a_table_is_treated_as_possible_pagination(page, card):
|
||||||
|
"""Both cards are paginatable (`app#embed_pagination`), and the capture was
|
||||||
|
too short to show what pagination looks like. Anything rendered after a
|
||||||
|
table is therefore a roster FC cannot prove complete — raised, never
|
||||||
|
returned short."""
|
||||||
|
start = page.index(f'data-identifier="{card}"')
|
||||||
|
close = page.index("</table>", start) + len("</table>")
|
||||||
|
paginated = page[:close] + '<nav class="pagination"><a>2</a></nav>' + page[close:]
|
||||||
|
with pytest.raises(SubscribeStarDriftError, match="cannot be proven complete"):
|
||||||
|
parse_subscriptions_page(paginated)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_page_link_anywhere_in_the_roster_is_drift(page):
|
||||||
|
"""The same concern for a paginator rendered outside the card."""
|
||||||
|
paged = page.replace("</body>", '<a href="/subscriptions?page=2">Next</a></body>')
|
||||||
|
with pytest.raises(SubscribeStarDriftError, match="page="):
|
||||||
|
parse_subscriptions_page(paged)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_drift_error_mid_page_yields_nothing(client, page):
|
||||||
|
"""The whole page is parsed before anything is yielded, so a caller can never
|
||||||
|
be left holding the rows that came before the bad one."""
|
||||||
|
_serve(client, page.replace(' data-user-id="4004"', ""))
|
||||||
|
got = []
|
||||||
|
with pytest.raises(SubscribeStarDriftError):
|
||||||
|
for m in client.iter_memberships():
|
||||||
|
got.append(m)
|
||||||
|
assert got == []
|
||||||
|
|
||||||
|
|
||||||
|
# --- the sweep's user-id probe ---------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_client_without_current_user_id_needs_no_id():
|
||||||
|
"""The seam leak D1 found: the sweep called `current_user_id()` on every
|
||||||
|
client, which exists only on Patreon's. Note #3970 promised a second platform
|
||||||
|
would be one `builders` line and the client method; without this probe it
|
||||||
|
would have raised AttributeError on the first sweep."""
|
||||||
|
assert roster_user_id(SubscribeStarClient(cookies_path=None)) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_client_with_current_user_id_is_asked_for_it():
|
||||||
|
assert roster_user_id(SimpleNamespace(current_user_id=lambda: "248453")) == "248453"
|
||||||
Reference in New Issue
Block a user