11 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	e3fd8c67d4	feat: switch pixiv off — unregistered, unreachable, and refused at dispatch (406 phase 1) ... Milestone 406 retires pixiv (rule 171) in two phases at the operator's explicit ask: switch it off, then later delete its code. This is the switch-off. Steps 2 and 3 ship together because each is a half-state of the other: unregistered but still in the extension, pixiv creator pages would offer a button the backend then refuses. Reachability removed, never gated (rule 22 - no flag, no `if platform == "pixiv"`): - platforms registry: pixiv unregistered, so /api/platforms, the source validator and quick-add all refuse it through their existing unknown-platform paths. - NATIVE_INGESTER_PLATFORMS: pixiv removed. - extension_service: pixiv's quick-add URL pattern removed (the Python half of the JS mirror). - extension: pixiv's host permissions, content-script match, platform entry and artist pattern removed; popup's pixiv branches removed; and the whole pixiv PKCE OAuth flow cut out of background.js. That last one could not wait for phase 2 - a webRequest listener on a host the manifest no longer grants is at best dead and at worst a startup failure for the entire background script. On startup the extension now also removes any pixiv refresh token a browser still holds in storage, for the same reason as the server-side credential cleanup (3980). - frontend: the extension card stops listing pixiv; SourceActions' copy of the native list drops it. platformColor keeps rendering a pixiv key so existing pixiv posts do not look broken. The guard, and why a registry change alone was not enough. A source outlives its platform: the live instance still had one ENABLED pixiv source (step 1). Tracing it: the scheduler only selects enabled rows and every platform lookup uses .get(), so a disabled row is inert - but re-enabling it and pressing Check would have routed pixiv, no longer native, straight into the gallery-dl branch, which still has a pixiv extractor. And a worker can pick up a still-enabled row before a deploy's migration runs. So run_download and verify_source_credential - the two functions every download and credential probe pass through - now refuse any platform not in the registry: an unsupported_url failure for downloads, and an inconclusive (None, not False) verify, since nothing was probed so nothing was rejected. Generic by registration, so it covers deviantart's leftovers too. Positive-controlled: a supported gallery-dl platform must still reach gallery-dl, or a guard that refused everything would pass (rule 167). Migration 0097 disables sources on retired platforms (pixiv, deviantart) and clears their failure state exactly as disabling through the app does (1285), so the stale row stops being scheduled and stops showing as failing. Nothing is deleted: removing a source can collide with uq_post_artist_external_id_null_source on real data, which is phase 2's step 6 to check. No post or image is touched. Tests: the known-platform lists drop pixiv and gain retirement assertions beside deviantart's; pixiv's positive extension cases become negative guards; the pixiv sidecar post-URL test is deleted with the behaviour it tested; quick-add rejects a pixiv URL. The pixiv client/downloader/ingester suites stay - that code stays until phase 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9	2026-09-13 11:45:49 -04:00
bvandeusenandClaude Opus 5	ddf896078c	refactor(platforms): retire deviantart end-to-end (#3069) ... Executes the 2026-07-05 product decision (FC downloaders = art-dedicated services only), which removed Twitter/X and Bluesky but left deviantart fully wired for seven weeks — the half-retired state rule 22 exists to prevent. Removed: the PlatformInfo module and its registry entry, the gallery-dl extractor block, extension_service's artist-page pattern, the extension's PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission and content-script match, the frontend icon/colour/label, and the operator- facing "supported platforms" list that still advertised it. Two judgment calls, both recorded in migration 0088: * existing `source` rows are DISABLED, not deleted. The row is the only record of the artist's DeviantArt URL. Disabling is also required for correctness rather than tidiness: with the platform unregistered the download path falls through to gallery-dl, which carries its OWN deviantart extractor, so an enabled row would have kept downloading from a dropped platform. * the `credential` row IS deleted — a live session cookie for a site FC will never call again. Adds the invariant whose absence is why manifest.json drifted in the first place: nothing tied its domain lists back to the platform table. The extension suite now asserts both directions, plus that no host permission belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at an operator-chosen URL the extension cannot enumerate). Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged extension change without a bump. No release is cut — build.yml's sign-extension job only runs on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:24:08 -04:00
bvandeusenandClaude Opus 4.8	69b5637bd6	fix(extension): show Add-to-FC button on subscribed Patreon creators (#1485) ... Symptom (operator-flagged): the extension injected the Add-as-source button on a Patreon creator you had NOT subscribed to, but it disappeared once you were subscribed — the opposite of when it's useful. Root cause (extension logic, not Patreon security): Patreon serves a creator under three URL shapes — bare patreon.com/Atole, patreon.com/c/Atole, and patreon.com/cw/Atole (the 'creator workspace' URL you land on once subscribed; documented in patreon_resolver._VANITY_RE). The button's artist-page gate (PLATFORM_ARTIST_PATTERNS.patreon in platforms.js) and its byte-mirror probe pattern (_PLATFORM_PATTERNS in extension_service._derive) only matched the bare single-segment form and explicitly excluded c/. So the subscribed-view URL failed the gate → no button. The ingestion resolver already handled all three; only these two gates were too narrow. Fix: both regexes now accept optional cw/ and c/ prefixes and drop the strict single-segment end-anchor, so a creator's inner page (/cw/Atole/posts, /Atole/membership) also matches — robust to whatever exact shape the subscribed view uses. Nav-page exclusions (home/search/messages/notifications/library/ settings/posts + post permalinks) preserved. New unit test covers all three prefixes, sub-paths, and nav-page rejection (both regexes validated identically). Bump extension 1.0.7→1.0.8 so a fresh signed XPI ships the fix (also exercises batch-5 web-ext-10's AMO sign path end-to-end on the main build). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 17:50:07 -04:00
bvandeusenandClaude Opus 4.8	0963bf0db3	feat(artist): resolve patreon + subscribestar display names at add-time (#130 step 5) ... Parity with pixiv (operator ask): the extension add now resolves the real display name for our other native platforms too, not just the URL handle. patreon_resolver.resolve_display_name reads the campaigns API's attributes.name; SubscribeStarClient.resolve_display_name pulls the creator name off the profile page (og:title, else the <title> stripped of the SubscribeStar suffix). extension_service._resolve_artist_name dispatches per platform (pixiv=token, patreon/subscribestar=cookies via get_cookies_path), best-effort in an executor, falling back to the readable URL handle on any failure. Still all curator core — the extension is unchanged (sends only the URL). gallery-dl platforms keep the handle (readable, no native client). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 22:37:50 -04:00
bvandeusenandClaude Opus 4.8	0c4b8aef8c	feat(artist): pixiv display-name at add-time + identity-by-source (#130 steps 2+3) ... Final piece of the artist decoupling. (1) Identity-by-source: quick_add_source resolves the artist by an existing (platform, url) Source first, so a re-add reuses the artist even after it was renamed (its frozen slug no longer matches the name) — a slug-based lookup would have duplicated it. (2) Pixiv naming: a new pixiv source resolves the real display name via the app API (PixivClient .resolve_display_name → /v1/user/detail) using the stored token, so the artist is 'Kurotsuchi Machi' not '12345678' — and its name-derived slug matches what a native download produces, unifying them. Falls back to the numeric id when no token/crypto. ExtensionService gains the crypto seam; the endpoint passes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 22:24:25 -04:00
bvandeusenandClaude Opus 4.8	7b2a2051e9	refactor(services): shared race-safe get_or_create helper (DRY backend sweep) ... The find-or-create dance — SELECT, then a SAVEPOINT INSERT that recovers (not a full rollback) on IntegrityError when a concurrent worker inserted first — was hand-rolled identically in 4 async sites: ArtistService.find_or_create, TagService.find_or_create, ExtensionService._find_or_create_artist and ._find_or_create_source. Divergent copies of exactly this pattern are how the duplicate-row/race bugs in reference_scalar_one_or_none_duplicates crept in, so it now lives once in services/db_helpers.get_or_create (returns (row, created); factory adds+flushes+returns the row; caller owns the outer commit). Over-DRY guard: SourceService's IntegrityError sites RAISE DuplicateSourceError (reject-on-conflict, a different concept) — left alone. Importer._get_or_create is the lone SYNC consumer (already shared by 2 callers) — stays separate, can't cross the sync/async boundary. §8b: no hand-rolled async find-or-create remains. Test: get_or_create creates then returns existing without re-invoking the factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-09 23:46:01 -04:00
bvandeusenandClaude Opus 4.8	96fffaff64	feat(download): smarter backfill — time-boxed chunks, run-until-done (backend) ... Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout wall and dies as an error each run. Builds on the cursor checkpoint (#689). - Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600), far under the 1350 soft limit. Hitting it = normal chunk boundary (the TimeoutExpired path already captures partial output + the cursor), not a near-wall death. - Run-until-done state machine driven by config_overrides[_backfill_state] (running/complete/stalled). A running backfill auto-continues in chunks across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom → 'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard pause a pathological walk as 'stalled'. Replaces the N-runs counter (backfill_runs_remaining repurposed as the cap countdown). - Progress, not error: a chunk that timed out but advanced (cursor moved and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok'). - Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838). - API: POST /sources/{id}/backfill now takes {action: start|stop}; service start_backfill/stop_backfill; new enabled sources auto-arm run-until-done; source dict exposes backfill_state + backfill_chunks. Frontend (Start/Stop control + state badge) lands in the next push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 15:02:46 -04:00
bvandeusen	3898ce7be4	fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit ... - BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout 30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat TimeoutExpired (matched the tick 870s/900s rationale). Backfill runs that hit the cap let the next tick continue via the archive. - recover_interrupted_tasks orphan UPDATE now stamps finished_at; without it cleanup_old_tasks' WHERE finished_at<cutoff never reaped orphan-swept rows. recover_stalled_task_runs also now sets duration_ms (matches celery_signals.finalize's millisecond math). - ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3 on Source creation, mirroring SourceService.create. Without it, Firefox quick-add on a creator with >20 unsynced posts walked the full feed until subprocess timeout. Renamed the constant from _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module. - gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside NO_NEW_CONTENT — the download path (line 712) already does, and TIER_LIMITED proves auth reached the post and was told it was tier-gated. Verify endpoint previously showed red on this and prompted operators to rotate working cookies. - prune_unused_tags now runs a single DELETE with the NOT-IN predicate find_unused_tags uses, instead of SELECT-ids → DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that would have surfaced on a tag explosion (>65k unused tags). - credentials.upload() reflects the returned record into the store cache (`.set(platform, rec)`) instead of evicting it; previously the card briefly rendered "no credential" between upload and loadAll().	2026-06-02 11:24:05 -04:00
bvandeusen	d65f0b2091	feat(extension): probe shows current state before click; v1.0.6 ... Operator-asked 2026-05-31 (during sidecar synthetic anchor cleanup): "the add source/subscription button idea to the firefox extension so it can tell me if a source/artist is added or not and offer an option to add it if it isn't." Plan tracked in Scribe task #507. ## Backend - `ExtensionService.probe(url)` — read-only resolution. Reuses `_derive` for platform+slug, then 2 SELECTs. Returns one of: - `source_match` (exact (artist, platform, url) Source exists) - `artist_match` (artist exists, this URL isn't a Source yet; collapses the sidecar-synthetic-only case from v26.06.01.0) - `new` (neither exists) - `unknown_platform` (URL didn't match any artist-page regex) - `GET /api/extension/probe?url=...` route with `X-Extension-Key` auth posture matching `/quick-add-source`. Read-only, side-effect free. - 6 backend tests in tests/test_api_extension.py covering each state + auth + invalid URL. ## Extension - `api.js`: `probeSource(url)` mirroring `quickAddSource` shape. - `background.js`: `PROBE_SOURCE` + `OPEN_ARTIST_PAGE` handlers. The latter strips the `/api` suffix from configured `apiUrl` (placeholder format per options.html) and opens `${base}/artist/{slug}` in a new tab via `browser.tabs.create`. - `content-script.js`: probe-first render — on page-load and SPA navigation, asks the backend for the URL's state and renders the chip in the matching color/copy on FIRST paint instead of flashing generic "Add" and updating after. Click handler branches: `source_match` → OPEN_ARTIST_PAGE; `artist_match`/`new` → existing ADD_AS_SOURCE flow (then re-probes so the chip flips green immediately, no wait for next nav). - `content-script.css`: three state-color modifiers (--new, --artist-match, --source-match) on the FC parchment-on-slate palette. Sage for already-added, amber for artist-exists, accent orange for new. ## Versioning - `extension/manifest.json` + `extension/package.json` → 1.0.6. build.yml's sign-extension job will fire on push to main since no `ext-1.0.6` Forgejo/Gitea release exists yet — exercises the regenerated AMO keys end-to-end. ## Behavior on the sidecar-synthetic case Filesystem-imported "Dymkens"-style artist with only a sidecar synthetic Source: probe returns `artist_match` (not `new`), so the chip reads "+ Add Patreon source to Dymkens" rather than offering to recreate the artist. Clicking adds the real Source; existing `_source_for_sidecar` preference logic (v26.06.01.0) routes future gallery-dl Posts to the real one.	2026-06-01 00:41:17 -04:00
bvandeusenandClaude Opus 4.7	fb41b90110	fix(extension): _find_or_create_artist + _find_or_create_source race-safe via savepoint + IntegrityError recovery — same pattern as importer's helpers. Two concurrent quick-add-source calls on the same artist/url would have 500'd on uq_artist_slug / uq_source_artist_platform_url; now the second one rolls the savepoint back and returns the existing row. ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 16:27:51 -04:00
bvandeusenandClaude Opus 4.7	46d199450d	fc3g: ExtensionService.quick_add_source — derive platform+slug, find-or-create Artist+Source ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-23 23:37:39 -04:00