6 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	24b10d0ffa	feat: retire pixiv entirely — delete its code, its ledgers, its credential (3977, 3978, 3979) ... Milestone #406 phase 2, with issue #3980 folded in. Phase 1 (2026-09-13) unregistered pixiv so nothing could reach it; the code has sat in the tree uncalled since. DeviantArt is why the second half is not left for later — #3069 retired it in code on 2026-08-27 and its stored session was still in the database seven weeks on. Step 5 — the code. Deletes pixiv_client, pixiv_downloader, pixiv_ingester, platforms/pixiv and their three test modules and fixture, then edits out every remaining reference: the dispatch entry, the campaign-id and verify branches in download_backends, the display-name branch in extension_service, and the comments that still described pixiv as live. The consolidation check the step asked for comes back negative: native_ingest_common has seven non-pixiv callers (patreon, subscribestar, membership_reconcile, membership_roster, ingest_core), so nothing there drops to a single user. Step 6 — the data, alembic 0102. Drops pixiv_seen_media and pixiv_failed_media, and deletes credential rows whose platform is not registered. Written as "not registered" rather than "pixiv" at the step's explicit ask, which is what makes one migration cover two retirements: the pixiv OAuth refresh token and DeviantArt's leftover session (#3980). It is also the only way either row can go — the credentials UI renders one card per platform from /api/platforms and looks the credential up by key, so an unregistered platform's row has no card and no Remove button. Pixiv's Source rows are KEPT, changing the milestone's original data table on the operator's call. `platform` is stored only on Source; neither Post nor ImageRecord carries it. Both FKs are ON DELETE SET NULL, so a delete would not lose the art — but it would drop every pixiv image into the gallery's __unsourced__ bucket and strip the platform chip off every pixiv post. The rows stay disabled (0097) and unregistered, so nothing schedules or downloads through them. Keeping them costs nothing and keeps the attribution that "the art already downloaded from pixiv stays" is about. Step 7 — the guard. test_pixiv_code_and_tables_are_gone asserts absence from the module table and from Base.metadata, not from prose (snippet #3352's trap). The extension and registry negative assertions were already in place from phase 1. The final sweep found one real residue step 4 missed: extension/README.md still advertised pixiv support and carried a "Pixiv OAuth" manual-test item. Also replaces the two deleted dispatch tests with one over the whole NATIVE_INGESTER_PLATFORMS set, so adding a platform and forgetting its ingester class now fails at unit level rather than as a mid-download KeyError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 20:46:15 -04:00
bvandeusenandClaude Opus 5	ddf896078c	refactor(platforms): retire deviantart end-to-end (#3069) ... Executes the 2026-07-05 product decision (FC downloaders = art-dedicated services only), which removed Twitter/X and Bluesky but left deviantart fully wired for seven weeks — the half-retired state rule 22 exists to prevent. Removed: the PlatformInfo module and its registry entry, the gallery-dl extractor block, extension_service's artist-page pattern, the extension's PLATFORMS + PLATFORM_ARTIST_PATTERNS entries, its manifest host permission and content-script match, the frontend icon/colour/label, and the operator- facing "supported platforms" list that still advertised it. Two judgment calls, both recorded in migration 0088: * existing `source` rows are DISABLED, not deleted. The row is the only record of the artist's DeviantArt URL. Disabling is also required for correctness rather than tidiness: with the platform unregistered the download path falls through to gallery-dl, which carries its OWN deviantart extractor, so an enabled row would have kept downloading from a dropped platform. * the `credential` row IS deleted — a live session cookie for a site FC will never call again. Adds the invariant whose absence is why manifest.json drifted in the first place: nothing tied its domain lists back to the platform table. The extension suite now asserts both directions, plus that no host permission belongs to an unclaimed domain (`*://*/*` exempted — FC is self-hosted at an operator-chosen URL the extension cannot enumerate). Extension version 1.0.10 -> 1.0.11: ci.yml's guard hard-fails a packaged extension change without a bump. No release is cut — build.yml's sign-extension job only runs on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 07:24:08 -04:00
bvandeusen	19aece1fc4	feat(download): tick/backfill modes + partial-success classifier (plan #544) ... Routine subscription polls walked the entire post history every tick even when nothing had changed, because gallery-dl's default `skip: True` continues iterating archived posts. A creator with ~550 archived posts (Knuxy patreon) saturates the 870s wall-clock cap before completing, even with zero downloads needed. Plus, a tier-limited run that downloaded hundreds of files but ran out the clock should be a warning, not an error. Two coupled changes, both operator-flagged 2026-06-01: * **Tick mode (default, cron polls).** New `TICK_SKIP_VALUE = "exit:20"` asks gallery-dl to exit after 20 contiguous archived items. Fresh subscriptions + new-content cases still walk normally; established subscription with zero new content exits in ~30s of HEAD requests instead of pegging the timeout. 20 (not 5) gives headroom against paywall warnings interleaving with archived items. * **Backfill mode (explicit, operator-triggered).** Sticky for N runs via new `Source.backfill_runs_remaining` (alembic 0031). While > 0, downloads use `skip: True` + 1800s timeout. Auto-decrements per run with early-reset to 0 when a clean run finds zero files (queue drained). N defaults to 3 — multiple runs give the system enough budget to finish a deep walk across timeout boundaries. New `POST /api/sources/{id}/backfill` arms the source; "Deep scan" button on each SourceRow (chip shows remaining count) wires it. Plus partial-success classifier: non-zero gallery-dl exit + ≥1 file downloaded + no source-level error fires `ErrorType.PARTIAL`, which download_service maps to `status=\"ok\"`. The run did real work; the next tick continues via gallery-dl's archive. No more red events for "timed out mid-walk after downloading 300 files." Retires `SourceConfig.skip_existing` — skip value is now derived from the source state and passed as a separate `skip_value` parameter through download() / _build_config_for_source(). `GD_DEFAULTS` drops the now-dead key (was inert data after this refactor). Tests cover: * tick + backfill skip-value emission in _build_config_for_source * PARTIAL classifier branch + TIER_LIMITED-wins-over-PARTIAL ordering * SourceService.set_backfill_runs validation + persistence * /api/sources/{id}/backfill 200/400/404 paths * download_service auto-decrement / auto-reset / tick-mode-no-touch * PARTIAL → status=ok in the orchestrator (no consecutive_failures bump)	2026-06-01 18:23:28 -04:00
bvandeusen	6d7116c090	fix(platforms): ruff I001 in base.py — one blank line between imports and module-level constant (was two)	2026-05-27 20:37:06 -04:00
bvandeusen	b447c42853	fix(platforms): ruff I001 — drop unused __future__ import; switch __init__ to per-module imports for clean isort ordering	2026-05-27 19:52:50 -04:00
bvandeusen	abafc3265e	refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation ... Operator-requested 2026-05-27: centralize the per-platform quirks that had been accumulating across credential_service, sidecar, and platforms into a single per-platform module so adding/updating quirks becomes "edit one file." **Layout** services/platforms/ base.py PlatformInfo dataclass + module-default key chains + shared helpers (str_id_value, str_field) __init__.py PLATFORMS dict + public API (auth_type_for, known_platform_keys, to_dict, external_post_id_keys_for, description_keys_for) patreon.py metadata only — the reference platform, no quirks subscribestar.py metadata + augment_cookies (18+ agreement) + derive_post_url (synthetic /posts/<post_id>) hentaifoundry.py metadata + augment_cookies (host-only PHPSESSID duplicate) + derive_post_url (/pictures/user/...) pixiv.py metadata + derive_post_url (/artworks/<id>) discord.py metadata + derive_post_url (channels/<server>/<channel>/<message>) deviantart.py metadata only — un-audited; quirks to be added when an operator first exercises DA **PlatformInfo extensions** Existing fields preserved. Four new optional fields: external_post_id_keys: tuple[str, ...] | None Override the sidecar external_post_id lookup chain. None falls back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py ("post_id", "id", "index", "message_id") — covers every current platform. description_keys: tuple[str, ...] | None Override the description body lookup chain. None falls back to DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption", "message") — Discord's "message" body field is covered by the default's trailing entry. derive_post_url: Callable[[dict], str | None] | None Synthesize the post permalink from sidecar metadata. None = trust the bare `url` / `post_url` field (patreon, deviantart). subscribestar/pixiv/hf/discord override this because their `url` is the file CDN URL. augment_cookies: Callable[[str], str] | None Post-process the materialized cookies.txt before gallery-dl consumes it. None = no-op. Used by subscribestar (age cookie) and hentaifoundry (host-only PHPSESSID duplicate). **Consumer changes** - credential_service._augment_cookies(platform, netscape) shrunk from a per-platform-conditional dispatcher (~80 lines of inlined helpers) to a 5-line lookup: `info.augment_cookies(netscape) if info and info.augment_cookies else netscape`. The platform-specific helper bodies moved verbatim into the per-platform modules. - sidecar.parse_sidecar similarly delegates: external_post_id chain via external_post_id_keys_for(category), description chain via description_keys_for(category), post_url via PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set and inline _derive_post_url body both gone. Added a shared `_first_id` helper for bool-safe id coercion. **Public API preserved** PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict are all re-exported from the package's __init__.py. test_platforms_registry test_credential_service, and test_sidecar_util pass without changes because the behavior is identical; only the implementation moved. **Adding a new platform** 1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)` and any of the four optional hooks. 2. Import it in services/platforms/__init__.py + add to the PLATFORMS tuple-comprehension. 3. Done. sidecar parsing, cookie materialization, /api/platforms all pick it up automatically.	2026-05-27 19:46:05 -04:00