16 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	4b4e532c56	fix: the membership status vocabulary moves beside Membership, so the Patreon client never imports the roster ... 240f11c made PatreonClient._membership import has_paid_access from membership_roster. test_gated_reason::test_no_fetch_path_can_read_the_roster failed on it, correctly: native_ingest_common is a fetch root, patreon_client is reachable from it, and no fetch path may be able to reach the roster. The roster is allowed to explain a skip, never to cause one. MEMBERSHIP_STATUS and has_paid_access are pure platform knowledge with no database behind them. They move to native_ingest_common, next to the Membership type they interpret (the same move C7 made for Membership itself). membership_roster, membership_reconcile, patreon_client and the tests import them from there. There is no re-export from membership_roster. The guard is unchanged. The lapsed-orphan skip from 240f11c stays as it was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 16:07:26 -04:00
bvandeusenandClaude Opus 5	0835da8a91	fix: give the roster's column zip an explicit strict=False (387 D1, B905) ... ef91fcf failed ruff's B905 lane on one zip(labels, cells) without strict=. Tests, integration and the frontend were already green on that SHA. strict=False is the deliberate side, not the quiet one. strict=True raises a bare ValueError - not SubscribeStarDriftError - and would fail the whole roster sync over a column mismatch in `details`, which nothing reads yet. That would take down reconciliation and the gated-post reasons over a cosmetic markup change, while creator identity (id, slug) never depended on the columns at all. But a shifted column would mislabel details silently (a price filed under "discord"), so a count mismatch now logs a canary warning, mirroring the feed parser's existing parse canary: diagnosable from the worker log, never fatal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 10:56:18 -04:00
bvandeusenandClaude Opus 5	ef91fcfd26	feat: SubscribeStar joins the membership roster (387 D1) ... The second platform through the seam note 3970 contracted, characterized first from a live capture of the account's /subscriptions page (note 3989). The capture lives in the gitignored captures dir; the committed fixture is hand-built with invented values and was verified tag-for-tag against it - card wrappers, both table heads, and every distinct row shape - before any code depended on it. What the page is, and the three decisions it forced: The table IS the status. SubscribeStar has no per-row status word: a creator is either in the active_subscriptions card or the cancelled_subscriptions one. The card's data-identifier is stored verbatim as Membership.status and mapped in MEMBERSHIP_STATUS, keyed on the identifier rather than the table class because the cancelled table's class names the same list differently (for-unsubscribed_users). The creator's numeric data-user-id is the key, not the slug. A slug re-keys when a creator renames; the old row stops appearing; and a disappearance is exactly what reconciliation reads as a lapse. Keyed on the slug, a rename would have told a paying subscriber they had cancelled. The slug rides as vanity, where the identity join already looks for a handle. Price is kept as text, never parsed into amount_cents. A bare $ names no currency and a page price is not proven to be the charge - 3970 finding 4. Tier names live behind a per-row modal and are not fetched. Refusals, because SubscribeStar offers nothing like Patreon's meta.pagination.total and every conclusion downstream is drawn from absence. The parser raises when: the active card is missing (auth error on a login/age wall, drift otherwise); a row lacks a numeric creator id or a creator link; anything renders after a card's table; or the page carries a page= link. Both cards are paginatable (app#embed_pagination) and the captured account was too small to show what pagination looks like, so possible pagination is a roster FC cannot prove complete. A loud error on a larger account beats a quiet half-list. A missing cancelled card is not drift, and a creator in both tables is reported once, as active. Fetched from subscribestar.adult, not the .art the capture came from: FC's requests never clear the .art age wall with the 18+ cookie (1259, 1284). Whether /subscriptions on .adult authenticates exactly as .art did in the browser is untested - if not, the sweep records a visible error and C6 shows its unavailable rung. The seam leak D1 found. Note 3970 promised a second platform would be one builders line plus the client method. The sweep instead called current_user_id() on every client, which only Patreon's has, so SubscribeStar would have raised AttributeError on the first sweep. roster_user_id probes it with getattr, the same way the sweep already probes iter_memberships. Two existing tests were passing for the wrong reason and now can fail: - "a platform that has never been characterised says nothing" named SubscribeStar, and stayed green only because active_patron is not a SubscribeStar word. Now uses hentaifoundry, with a positive SubscribeStar test beside it. - the freshness test gave SubscribeStar a Patreon word, so the vocabulary excluded it and deleting the freshness gate outright would have left it green. It now uses cancelled_subscriptions, making the gate the only thing that excludes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 10:53:01 -04:00
bvandeusenandClaude Opus 4.8	0963bf0db3	feat(artist): resolve patreon + subscribestar display names at add-time (#130 step 5) ... Parity with pixiv (operator ask): the extension add now resolves the real display name for our other native platforms too, not just the URL handle. patreon_resolver.resolve_display_name reads the campaigns API's attributes.name; SubscribeStarClient.resolve_display_name pulls the creator name off the profile page (og:title, else the <title> stripped of the SubscribeStar suffix). extension_service._resolve_artist_name dispatches per platform (pixiv=token, patreon/subscribestar=cookies via get_cookies_path), best-effort in an executor, falling back to the readable URL handle on any failure. Still all curator core — the extension is unchanged (sends only the URL). gallery-dl platforms keep the handle (readable, no native client). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 22:37:50 -04:00
bvandeusenandClaude Opus 4.8	c9696a2faf	fix(subscribestar): route .art creators to .adult; clear source failure on disable ... Two pre-merge fixes: 1. SubscribeStar .art age wall: the 18+ cookie doesn't clear the age gate on the .art domain (keeps 302'ing to /age_confirmation_warning even with the cookie — Elasid #54116), but the same creator is reachable on .adult where the cookie works. _normalize_ss_host rewrites subscribestar.art → subscribestar.adult at request time (stored Source.url untouched), logged so it's visible in walk logs. .com/.adult pass through. 2. Disabling a source now clears its failure state (last_error, error_type, consecutive_failures) so subs you pause (not paying for) stop lingering as 'failing'. Only the explicit disable clears — an unrelated edit to an already-disabled source leaves state alone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-03 14:43:02 -04:00
bvandeusenandClaude Opus 4.8	7f6345dccf	fix(subscribestar): port gallery-dl date extraction (wrapped dates) + parse canary ... Image posts wrap the post date in an <a> permalink (<div class="post-date"><a href="/posts/ID">DATE</a></div>); text-only posts don't. Our hand-written <div class="post-date">([^<]+)</div> regex matched ONLY the unwrapped case, so every image post got a null published_at and sorted to the top of the feed looking broken (cheunart 2026-06-17). Port gallery-dl's _data_from_post method: text up to the first </, then after the last > — handles both. Verified against the live raw feed (all 6 dates now parse). Robust logging (operator request): _parse_posts now logs per-page parse stats (posts / dated / with-body) and a WARNING canary when posts parse but NONE get a date or body while the raw markers are present — i.e. our extraction diverged from the live markup. Makes this failure class diagnosable from the worker log alone, no authed re-fetch needed. Test: a permalink-wrapped date parses to ISO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 20:51:18 -04:00
bvandeusenandClaude Opus 4.8	479b7b54da	style(subscribestar): drop quoted forward-ref annotation (UP037) ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 16:19:12 -04:00
bvandeusenandClaude Opus 4.8	976f581aa2	feat(subscribestar): port gallery-dl doc + audio attachment extraction ... Some SubscribeStar posts deliver content only through document/audio attachments, which live OUTSIDE data-gallery. Port gallery-dl's _media_from_post for them: - docs: scope uploads-docs..post-edit_form, split on doc_preview blocks, take the href URL + doc_preview-title + data-upload-id (kind=attachment). - audio: scope uploads-audios..post-edit_form, split on audio_preview-data blocks, take the src URL + audio_preview-title + data-upload-id (kind=audio). The existing downloader handles them unchanged (plain streaming GET; the file validator only inspects image/video extensions via is_validatable, so PDFs/zips/ audio pass straight through, no quarantine). Test covers doc + audio extraction (the cheunart sample has none, so this pins gallery-dl's documented markup shape). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 16:18:10 -04:00
bvandeusenandClaude Opus 4.8	8771364cee	fix(subscribestar): port gallery-dl's content + preview-skip extraction faithfully ... Body rendered as a bogus '264 / 265' on every post: our balanced-</div> body regex either returned empty or over-captured into sibling upload divs and the 'View next posts (N / M)' pagination counter. Replace it with gallery-dl's exact _data_from_post rule — content between the post_content-text wrapper and the youtube-uploads div (literal markers), then strip the trix editor's <html><body>…</body></html> document wrapper to its inner. Verified against the live cheunart sample: clean per-post bodies, empty for genuinely text-less posts. Also port gallery-dl's _media_from_post preview guard: skip gallery items whose URL is under /previews (locked/blurred teasers) — the SubscribeStar analog of the Patreon gated-preview bug (#874); this is why a locked post yields no media. Tests: body must not bleed into the pagination counter; trix html-document wrapper stripped; /previews items skipped. Fixture now includes the youtube- uploads close marker present in real markup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 16:06:52 -04:00
bvandeusenandClaude Opus 4.8	559d29fe1c	fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate) ... After the delimiter fix, the live cheunart fetch 302'd to /cheunart/verify_ subscriber even with valid .adult cookies (confirmed present: _personalization_id + _subscribestar_session on .subscribestar.adult, logged in). Walking gallery-dl's ENTIRE flow — including the base Extractor._init_session I'd not read — the divergence is the HTTP request profile, not the cookies or parser. gallery-dl's default (cookies-only) mode sends, on EVERY request including the first creator-page GET: a Firefox UA, Accept: */*, Accept-Language, and a same- site Referer (root/), with NO X-Requested-With anywhere (the load-more endpoint is a plain GET parsed as JSON). Our Chrome UA + missing Referer + XHR toggling looked unlike a browser → SubscribeStar gated the adult-creator page. Make our SubscribeStar session identical: Firefox UA + Accept */* + Accept- Language via make_session extra_headers; stamp Referer=<base>/ per walk; drop the per-request XHR headers (both feed and load-more now use the shared profile). Test updated to assert the gallery-dl-parity profile instead of the old navigation-vs-XHR split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:28:33 -04:00
bvandeusenandClaude Opus 4.8	204d341a99	fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift) ... The native client split the feed on `<div class="post is-shown`, but `is-shown` is added by SubscribeStar's infinite-scroll JS when a post scrolls into view — present in a browser-SAVED page (what the Step-0 characterization used) but ABSENT from the raw server HTML we and gallery-dl actually fetch. So the live feed (cheunart) parsed to zero posts and raised a false SubscribeStarDriftError. Align with gallery-dl's proven `_pagination`: split on the generic `<div class="post ` (trailing space rules out the hyphenated post-content/ post-date/post-body siblings). Also mirror gallery-dl's redirect-based gating detection (/verify_subscriber, /age_confirmation_warning => auth, not drift). Regression tests: raw server markup without is-shown now parses; an age-wall redirect raises SubscribeStarAuthError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 15:04:05 -04:00
bvandeusenandClaude Opus 4.8	9201b7b539	diag(subscribestar): name the interstitial in the drift error (title + type) ... The XHR fix worked (we now get a real 93KB HTML page, not JSON) but cheunart still drifts — we're being served a full HTML page that isn't the feed. Add _describe_page(): the drift error now reports the page <title> + which known interstitial it resembles (cloudflare/bot-challenge, age-gate, login, captcha), so the next run names the actual cause instead of "markup changed". Strong suspicion: a Cloudflare challenge (python-requests has no JS; cf_clearance is UA-locked and our hardcoded UA likely differs from the cookie-capturing browser). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 14:52:28 -04:00
bvandeusenandClaude Opus 4.8	78a3977f8a	fix(subscribestar): initial feed GET is a navigation, not XHR (first-run drift) ... First live run (cheunart) tripped the drift guard: "no posts and no recognizable feed container". The browser-saved page was normal (6 posts + posts_container-list), so the parser was fine — our live HTTP fetch got a different response. Cause: the client set X-Requested-With: XMLHttpRequest (+ a JSON Accept) session-wide, so the initial creator-page GET was sent as an XHR. SubscribeStar (Rails) content- negotiates an XHR full-page request to a non-HTML body → no container → drift. Fix: the session now uses browser-like navigation headers (Accept: html, NO X-Requested-With); the XHR header + JSON Accept are applied PER-REQUEST only on the "load more" endpoint (which is a genuine XHR). Drift message now reports the response length + a JSON hint so a recurrence is self-explaining. Regression test pins the header split (navigation initial GET, XHR load-more). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 14:25:40 -04:00
bvandeusenandClaude Opus 4.8	ebe6ab9741	refactor(native-ingest): shared exception trio + base _failure_result (#899 DRY 2/3) ... DRY pass commit 2. The two adapters re-implemented the same auth→drift→429→404 →http→network mapping in _failure_result; only the exception classes + drift phrasing differed (divergence-bug risk: a new error_type handled in one and not the other). - native_ingest_common gains NativeIngestError / NativeAuthError / NativeDriftError (status_code + retry_after on the base). Patreon{API,Auth,Drift}Error and SubscribeStar{API,Auth,Drift}Error now subclass them via multiple inheritance, keeping their isinstance-distinct platform names. - Ingester._failure_result (base) does the whole mapping via the shared NativeAuthError/NativeDriftError taxonomy + status_code; a new platform gets it free. New drift_label kwarg supplies the per-platform API_DRIFT phrasing ("Patreon API" / "SubscribeStar markup"), preserving the existing message (test asserts "Patreon API changed"). - Both adapters drop their near-identical _failure_result overrides and their now -unused DownloadResult/ErrorType/*Auth/*Drift imports. Verified at every consumer (rule 93/§8b): test_patreon_ingester (auth/drift/429/ 404/network) and test_subscribestar_native (_failure_result mapping) both exercise the base method now. Remaining: ingest_core L1/L3 logging (3/3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 11:40:36 -04:00
bvandeusenandClaude Opus 4.8	7ac5c7e522	refactor(native-ingest): extract native_ingest_common + BaseNativeDownloader (#899 DRY 1/3) ... DRY pass commit 1 (process #594). Consolidate the helpers + download plumbing the Patreon and SubscribeStar adapters had duplicated (SubscribeStar was importing patreon privates — wrong owner). New backend/app/services/ native_ingest_common.py is the neutral home for: - make_session (was _load_session ×2), retry_after_seconds + 429 constants, sanitize_segment, basename_from_url, post_dir_name, MediaOutcome / PostRecordOutcome. - BaseNativeDownloader: the shared streaming GET (transient-retry + Range-resume) and validation/quarantine. Patreon + SubscribeStar downloaders now subclass it; each keeps only what differs (Patreon's Mux/yt-dlp video branch + detail-fetch enrichment; SubscribeStar nothing extra). Behavior preserved exactly; the divergence-bug risk (a fix to one _fetch_to_file not reaching the other) is gone. - Folds in #899 L2: a quarantine now log.warning's path+reason (was counted only). post_dir_name merges both date handlers (accepts trailing-Z and pre-parsed ISO). Tests repointed to the single source at every consumer (rule 93 / §8b parity): patreon_client/downloader, subscribestar_native. Exception-trio consolidation + base _failure_result (2/3) and the remaining ingest_core logging (3/3) follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 11:32:28 -04:00
bvandeusenandClaude Opus 4.8	817a002c2b	feat(subscribestar): native client + downloader + ingester (post-first) (#890/#891/#892) ... Phase-1 steps 2-4 of moving SubscribeStar off gallery-dl onto the native core ingester. SubscribeStar has no JSON:API, so the client scrapes HTML; the platform-agnostic core (ingest_core) is unchanged. - subscribestar_client.py: HTML-scrape read path. iter_posts pages via the creator page → infinite_scroll-next_page href → JSON {html} fragments (campaign_id = creator URL; no resolver). extract_media reads the per-post data-gallery JSON manifest (id/original_filename/type/url). post_record_key, post_meta, and post_is_gated (best-effort locked-teaser marker, pending a live locked sample). Loud auth/drift taxonomy (SubscribeStar{API,Auth,Drift}Error). Parser validated against the real Step-0 fixtures. - subscribestar_downloader.py: mirrors PatreonDownloader minus the Mux/yt-dlp branch (SubscribeStar serves files directly via /post_uploads). gallery-dl on-disk layout so existing downloads dedup on disk at cutover. Post-first: _post.json owns the body/links; per-media sidecar carries image identity only. - subscribestar_ingester.py: thin adapter wiring client/downloader/the SubscribeStar ledgers into the core; ledger_key = filehash else post_id:media_id; SubscribeStar failure mapping. verify_subscribestar_credential. - tests: client parsing/pagination/media/gating/record-key/dates, downloader layout/sidecar/post-record/skip-seen, ingester ledger_key + failure mapping. Not yet wired into dispatch (Step 5) — these modules are inert until then. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-17 11:06:02 -04:00