Compare commits

...
53 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 4.8 3b2f7a41c3 feat(patreon): ingester rate-limit resilience — #703 step 1
CI / frontend-build (push) Successful in 21s
CI / integration (push) Failing after 3m0s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 25s
A single 429 mid-walk used to fail the run → RATE_LIMITED → platform-wide
cooldown → every Patreon source dark ("testing dead"). The native path also
ignored the operator's existing politeness setting. Fixed both:

- Pacing (avoid 429s): honor download_rate_limit_seconds (gallery-dl's
  `rate_limit`, read off self.gdl) as a pre-download sleep on real media
  downloads only (skips don't pace); pace /api/posts page fetches with the
  per-source sleep_request override, defaulting to max(0.5, rate_limit/4) —
  the same API-pacing default gallery-dl used for `sleep-request`.
- 429 backoff (ride out transient limits): PatreonClient._fetch retries a
  429 with backoff (honor Retry-After, else exponential 2·2^(n-1), capped
  30s, ≤3 tries); only a PERSISTENT 429 propagates as terminal
  RATE_LIMITED. Light 2-retry on a media-GET 429 too.

Threaded via PatreonIngester(rate_limit=, request_sleep=) →
PatreonClient/PatreonDownloader; download_service sources them. Injected
test client/downloader are unaffected (carry their own pacing).

Tests mock time.sleep (no real sleeping): retry-then-success, persistent
429 raises after N, Retry-After honored, request_sleep paces, media pacing
per real download, skips don't pace, media 429 retried.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 23:13:46 -04:00
bvandeusenandClaude Opus 4.8 218bfebb92 feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 19s
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 2m59s
The credential Verify button still ran gallery-dl --simulate for Patreon
after the cutover — testing the wrong path (and prone to the vanity
"Failed to extract campaign ID" the native resolver fixes). Wire it to the
native ingester, behind a DRY dispatch so callers never branch on platform.

- services/download_backends.py (new): the ONE place that knows which
  platforms are native vs gallery-dl. `uses_native_ingester(platform)` is
  the shared predicate; `verify_source_credential(...)` is the uniform
  probe (same (ok|None, message) contract for both backends). As a platform
  migrates, it moves into NATIVE_INGESTER_PLATFORMS here and BOTH download
  routing and verify switch together.
- PatreonClient.verify_auth(campaign_id): one authenticated /api/posts
  fetch → True (valid) / False (401/403/HTML-login) / None (drift or
  network — inconclusive, not a credential verdict).
- patreon_ingester.verify_patreon_credential(): resolve campaign id, then
  verify_auth — the verify counterpart to the download path.
- patreon_resolver.resolve_campaign_id_for_source(): extracted the
  override / id:-URL / vanity resolution into ONE helper now shared by the
  download ingester and verify (download_service no longer carries its own
  copy + regex; −`import re`).
- download_service: routes on uses_native_ingester() instead of inline
  `== "patreon"` (3 sites); uses the shared resolver.
- api/credentials: calls verify_source_credential — no platform branch.

Tests: verify_auth mapping, resolve_campaign_id_for_source (override/id:/
vanity/none), the dispatch predicate, verify_patreon_credential glue,
credentials endpoint proves Patreon uses the native path (gallery-dl verify
asserted not-called); repointed the gallery-dl verify test to subscribestar.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 22:49:43 -04:00
bvandeusenandClaude Opus 4.8 ec43e823e1 feat(patreon): recovery UI + gallery-dl cutover — build step 5 (plan #697)
CI / frontend-build (push) Successful in 23s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 10s
CI / integration (push) Successful in 2m59s
Final step of the native Patreon ingester: a first-class Recovery action,
and removal of the now-dead gallery-dl Patreon path.

Recovery (rules #23/#24/#27 — full product, with UI):
- source_service.start_recovery arms the #693 backfill state machine PLUS
  `_backfill_bypass_seen`, flipping download mode to recovery (bypass the
  seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate
  under the current pHash threshold). Stop via the shared stop_backfill.
- SourceRecord exposes backfill_bypass_seen; POST /sources/{id}/backfill
  gains action="recover".
- Frontend: Recovery button (Patreon-only, mdi-backup-restore) on SourceRow
  + SourceCard; the running badge labels "Recovering (N)" vs "Backfilling
  (N)"; the Stop tooltip says "Stop recovery". sources.js recoverSource +
  SubscriptionsTab onRecover.

Cutover (rule #22 — no legacy):
- gallery_dl: removed PLATFORM_DEFAULTS["patreon"], the patreon
  files/cursor branch in _build_config_for_source, and the patreon/Mux
  yt-dlp Referer/Origin block (was patreon-specific and wrongly tagged the
  other platforms' yt-dlp fetches; native ingester owns it now).
- download_service: removed the dead campaign-id-retry helpers
  (_looks_like_campaign_id_failure / _CAMPAIGN_ID_FAILURE_PATTERN) and
  _effective_url. Vanity→campaign resolution + resume_cursor + the
  cursor/PARTIAL lifecycle stay — they serve the native ingester.

Tests: removed the two obsolete patreon-gallery-dl config tests (yt-dlp
Referer, resume-cursor); repointed the generic skip-value config tests to
subscribestar; added start_recovery + recover-endpoint coverage
(backfill_bypass_seen). gallery-dl stays for the other 5 platforms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 22:18:03 -04:00
bvandeusenandClaude Opus 4.8 682beafbc5 feat(patreon): drift detection + error categorization — build step 4 (plan #697)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 28s
CI / integration (push) Successful in 2m58s
Typed, loud failure mapping for the native Patreon ingester so a changed
API shape or expired auth never silently zero-downloads as "success".

- New ErrorType.API_DRIFT (free varchar error_type col → no migration):
  distinct from auth so the operator knows the fix is updating the
  ingester, not rotating cookies.
- patreon_client: PatreonAPIError carries status_code; new PatreonAuthError
  for 401/403 + HTML-login/non-JSON bodies (reclassified from drift —
  expired-session is auth, actionable as "rotate cookies").
- patreon_ingester._failure_result maps: PatreonAuthError→AUTH_ERROR,
  PatreonDriftError→API_DRIFT ("Patreon API changed — ingester needs
  update"), HTTP 429→RATE_LIMITED, 404→NOT_FOUND, other HTTP→HTTP_ERROR,
  transport→NETWORK_ERROR. (429 thus drives the platform cooldown.)
- FailingSourcesCard: api_drift chip (red) + hint.

Contract test (new test_patreon_contract.py): the recorded /api/posts
fixture must parse end-to-end (no drift, 5 media across 4 posts) AND the
request params must still carry every field the parser depends on
(file_name, image_urls/download_url, images/attachments_media/media
includes, content/post_file/image post fields) — a trim of either trips a
red build. Plus client HTTP-status classification tests and ingester
error-type mapping tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 21:56:58 -04:00
bvandeusenandClaude Opus 4.8 96c30eba13 feat(patreon): phase-2 ingester integration — build step 3 (plan #697)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 31s
CI / integration (push) Successful in 2m59s
Branch download_service phase 2 by platform: Patreon now routes to the
native PatreonIngester (zero per-file HEADs, native cursor/resume, loud
drift detection) instead of gallery-dl; the other 5 platforms are
unchanged. The ingester returns a DownloadResult-shaped object so phase 1
(DB setup) and phase 3 (import → pHash → thumbs → ML) are untouched.

Three modes wired from config_overrides state:
  - tick: skip seen (tier-1 ledger + tier-2 disk), early-out after N
    contiguous already-have-it items.
  - backfill: full-history time-boxed chunk, cursor checkpoint via
    gallery-dl-style "Cursor: <token>" lines in stdout (reuses the #693
    lifecycle + parse_last_cursor verbatim).
  - recovery: backfill that BYPASSES the tier-1 seen-ledger so
    dropped-and-deleted near-dups get re-fetched and re-evaluated under
    the current pHash threshold. Rides the #693 state machine via a
    _backfill_bypass_seen flag, cleared on completion / stop.

The seen-ledger uses short-lived sync sessions (injected sessionmaker),
never held across the walk (avoids the connection-reaping trap). Campaign
id resolves from override, an id: URL, or a vanity lookup; unresolvable =
loud NOT_FOUND, never a silent empty success.

Tests: new test_patreon_ingester.py (modes, ledger skip/idempotency,
budget→PARTIAL, recovery bypass, tier-2 disk, drift). The patreon-oriented
download_service tests now drive the ingester branch via a stub; the
gallery-dl campaign-retry test is replaced by resolution/caching coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 21:38:42 -04:00
bvandeusenandClaude Opus 4.8 2ec7d86a3b feat(patreon): native media downloader — ingester build step 2b (plan #697)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / integration (push) Successful in 2m58s
CI / frontend-build (push) Successful in 27s
PatreonDownloader.download_post: writes resolved MediaItems to gallery-dl's
exact on-disk layout (<slug>/patreon/<date>_<id>_<title40>/<NN>_<file>) +
a sidecar the importer's find_sidecar/parse_sidecar consume unchanged. Two-tier
skip (injected seen predicate, then disk). Streamed GET (.part→rename) +
file_validator quarantine; Mux/m3u8 video shells out to yt-dlp with Patreon
Referer/Origin. Pure (no DB) — ledger + orchestration land in step 3. Unit
tests stub the session + yt-dlp seams (no network/subprocess).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 19:39:12 -04:00
bvandeusenandClaude Opus 4.8 6222928746 feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)
CI / integration (push) Successful in 2m57s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 28s
patreon_seen_media(source_id, filehash, post_id, seen_at), UNIQUE(source_id,
filehash) — our own queryable replacement for gallery-dl's archive.sqlite3.
Routine walks skip seen media; recovery mode bypasses the ledger. filehash is
a 32-hex CDN MD5 or a video:<post>:<media> sentinel (String(128)). alembic
0037 (← 0036). Integration test covers dedup + savepoint recovery.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 19:26:57 -04:00
bvandeusenandClaude Opus 4.8 1bdaa04aa2 test(patreon): fix self-contradictory attachment/postfile dedup assertion
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 29s
CI / integration (push) Successful in 2m57s
The fixture gives the attachment and post_file the same filehash, so they
correctly collapse to one item; the test asserted both survival and collapse.
Rewrite to verify the cross-kind dedup (postfile kind covered by video test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 19:17:56 -04:00
bvandeusenandClaude Opus 4.8 1c2dc7659a feat(patreon): native JSON-API client — ingester build step 1 (plan #697)
CI / frontend-build (push) Successful in 19s
CI / integration (push) Successful in 2m58s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Failing after 12s
PatreonClient: cookie-auth requests session, /api/posts cursor pagination,
JSON:API included flattening, per-post media extraction (images/image_large/
attachments/postfile/content) with filehash dedup, loud drift detection.
Zero per-file HEADs — every media URL+file_name comes from the API. Not yet
wired into download_service (later step). Pure-parsing unit tests + fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 19:16:20 -04:00
bvandeusenandClaude Opus 4.8 618dafde85 feat(subscriptions): smarter-backfill UI — Start/Stop + state badge
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 18s
CI / integration (push) Successful in 2m58s
Plan #693 (frontend). Backend landed in 96fffaf.

- sources store: setBackfill(runs) → startBackfill/stopBackfill ({action}).
- SubscriptionsTab: the deep-scan window.prompt for N runs becomes a
  Start/Stop toggle keyed on source.backfill_state.
- SourceRow + SourceCard: the 'backfill (N×)' chip becomes a state badge —
  Backfilling (chunk N) / Backfilled / Stalled — and the scan button
  toggles between Backfill (mdi-magnify-scan) and Stop (mdi-stop).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 15:06:30 -04:00
bvandeusenandClaude Opus 4.8 96fffaff64 feat(download): smarter backfill — time-boxed chunks, run-until-done (backend)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / integration (push) Successful in 2m58s
Plan #693. Large-catalog backfill (Anduo) no longer sprints to the timeout
wall and dies as an error each run. Builds on the cursor checkpoint (#689).

- Time-boxed chunks: BACKFILL_TIMEOUT_SECONDS(1170)→BACKFILL_CHUNK_SECONDS(600),
  far under the 1350 soft limit. Hitting it = normal chunk boundary (the
  TimeoutExpired path already captures partial output + the cursor), not a
  near-wall death.
- Run-until-done state machine driven by config_overrides[_backfill_state]
  (running/complete/stalled). A running backfill auto-continues in chunks
  across ticks until gallery-dl exits cleanly (rc=0 = reached the bottom →
  'complete'); a safety-cap (BACKFILL_MAX_CHUNKS=200) + the #689 stall-guard
  pause a pathological walk as 'stalled'. Replaces the N-runs counter
  (backfill_runs_remaining repurposed as the cap countdown).
- Progress, not error: a chunk that timed out but advanced (cursor moved
  and/or files written) is reclassified TIMEOUT→PARTIAL (status 'ok').
- Retry storm tamed: gallery-dl retries 3→2, downloader timeout 120→60s, so
  one stuck CDN file fails in ~1-2 min not ~10 (Anduo #40838).
- API: POST /sources/{id}/backfill now takes {action: start|stop}; service
  start_backfill/stop_backfill; new enabled sources auto-arm run-until-done;
  source dict exposes backfill_state + backfill_chunks.

Frontend (Start/Stop control + state badge) lands in the next push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 15:02:46 -04:00
bvandeusenandClaude Opus 4.8 add1c1ad14 fix(modal): mobile — no tag autofocus + sticky image over scrolling panel
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 24s
CI / integration (push) Successful in 2m58s
Operator-flagged 2026-06-05 (mobile):
- TagAutocomplete no longer autofocuses on the ≤900px stacked layout —
  focusing popped the soft keyboard, shrinking the viewport and shoving the
  pinned image + nav/close controls out of view. matchMedia gate (safe on
  plain HTTP); desktop autofocus unchanged.
- ImageViewer stacked layout: the body now scrolls and the media pane is
  sticky (height:55vh, top:0), so the image + prev/next/close + integrity
  badge stay pinned while the metadata panel scrolls beneath. Replaces the
  old fixed image + 40vh internally-scrolled panel that could push the
  controls off. Prev/next re-centered over the image band; opaque obsidian
  bg so the scrolling panel doesn't bleed through.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 08:35:22 -04:00
bvandeusenandClaude Opus 4.8 593f65c9cc feat(download): cursor-paged Patreon backfill for large catalogs
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 25s
CI / integration (push) Successful in 2m57s
Large Patreon creators (Anduo: weekly 50-120-image Reports back months =
thousands of files) couldn't backfill: each run re-walked newest→oldest
from the top, and gallery-dl's polite ~0.75s/request HEAD walk alone
exceeded the 1170s subprocess budget, so the run died during enumeration
with 0 files written and NO forward progress — re-stranding every time
(event #40411).

Checkpoint gallery-dl's pagination cursor so each backfill window advances
the frontier:

- gallery_dl.py: SourceConfig.resume_cursor; _build_config_for_source sets
  extractor.patreon.cursor=<resume> (PLATFORM_DEFAULTS leave log-only True
  for a fresh run); parse_last_cursor() pulls the last emitted
  'Cursor: <token>' from stdout+stderr — survives a timed-out run since the
  TimeoutExpired path returns partial output.
- download_service.py: phase2 stays in BACKFILL mode while a cursor is
  pending (even after the run budget drains) and threads resume_cursor;
  _apply_backfill_lifecycle() checkpoints the advancing cursor each
  non-completing run, completes on a clean rc=0 finish (walk reached
  bottom), and a stuck-guard clears the cursor after 2 non-advancing runs
  so a wedged walk can't re-strand forever.

patreon-only (sole platform with a resumable cursor); other platforms keep
the simple counter semantics. Cursor state lives in config_overrides JSON
(patreon_campaign_id precedent) — no migration. Time-budget ladder
(1170/1350/1500) unchanged.

Plan #689.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 23:44:31 -04:00
bvandeusenandClaude Opus 4.8 86efbf7f2c fix(modal): kebab menus open via explicit v-model, not activator click
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / integration (push) Successful in 2m56s
Operator-confirmed on a fresh build: both the tag-chip and suggestion
kebabs still never opened. The prior 8326e54 'fix' only wrapped them in a
<span @click.stop> — inert for SuggestionItem (no parent capture) — and
never addressed why the `#activator`/`v-bind="props"` click failed to
toggle the menu inside the teleported ImageViewer modal. The dialogs in
that same modal open via v-model and work, so drive the menus the same way:

- The activator (v-btn / v-icon) toggles a reactive flag with @click.stop
  (which also shields the chip's close button / any parent).
- The v-menu binds that flag (v-model / :model-value) and uses
  activator="parent" with :open-on-click="false" purely for positioning,
  so opening no longer depends on Vuetify's activator-click path.
- TagPanel tracks a single openTagId (one chip menu open at a time).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 23:01:55 -04:00
bvandeusenandClaude Opus 4.8 3a0cca5aca fix(tags): allow creating a character with no fandom
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 18s
CI / integration (push) Successful in 3m4s
Not all characters belong to a fandom (original characters, unsorted).
The create flow forced every new character through FandomPicker, whose
only outcomes were 'Use this fandom' (disabled until one is picked) or
Cancel (which aborts the whole creation) — there was no way to confirm a
character with no fandom.

- FandomPicker: add a 'No fandom' action that emits confirm(null).
- TagAutocomplete.onFandomChosen: pass fandom_id: null when null is
  emitted.

Backend already supported this end to end (Tag.fandom_id nullable, the
CHECK only forbids fandom_id on non-character kinds, tag_service
find_or_create defaults fandom_id=None, API reads body.get). A fandom can
still be assigned later from the chip kebab's 'Set fandom…'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:50:19 -04:00
bvandeusenandClaude Opus 4.8 a5b3702863 feat(settings): surface the near-duplicate (pHash) control + reorder import tab
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 10s
CI / integration (push) Successful in 3m6s
The phash_threshold knob (controls whether edits/variants of an image are
dropped as near-duplicates on import) was buried at the bottom of the import
filters form and labelled opaquely, so it read as 'missing'. Hoist it to the
TOP of the form as a 'Near-duplicate sensitivity' section: a labelled slider
(Exact / Strict / Default / Loose stops, 0-16) for the gist + the precise
number field, both bound to phash_threshold, with copy that says plainly to
lower it if variants are being dropped.

Also swap the import-tab order to filters → trigger → recent-tasks (filters on
top per operator); the task list stays directly under the trigger for hit/miss
feedback adjacency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 17:42:32 -04:00
bvandeusenandClaude Opus 4.8 509a7958cf feat(posts): images open the modal; only text expands in place
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 27s
CI / integration (push) Successful in 3m2s
Post cards no longer expand the whole card on click (the old two-click path
to the images). The card is compact-only now:
- Hero / rail thumbs / the +N tile are buttons that open the post-scoped image
  modal (modal.open(id, { postImageIds })) so you view big + arrow through ALL
  the post's images. The feed caps thumbnails at 6, so for posts with more we
  lazily getPostFull to get the complete id list; +N opens at the first hidden
  image.
- The description is the ONLY in-place expansion: a Show more / Show less toggle
  shown only when the text is actually truncated (server description_truncated
  flag OR a measured CSS-clamp overflow, ResizeObserver-guarded). Expanding
  loads description_full when server-truncated and renders it unclamped.
- Attachments: download chips now render inline in the compact card (the feed
  already carries download_url), since the expanded view is gone.

Removes PostImageGrid.vue (the mosaic, now unused). Tests cover show-more
visibility + image-click opening the scoped modal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 17:29:19 -04:00
bvandeusenandClaude Opus 4.8 5a6a95682d fix(cleanup): library scans survive navigation, reconnect on return
CI / frontend-build (push) Successful in 24s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / integration (push) Successful in 2m56s
The transparency / single-color audit cards held the run + poll timer in
local component state, so navigating away destroyed both and onMounted never
reconnected — the Celery scan kept running and writing LibraryAuditRun, but
the UI forgot it. Now each card, on mount, fetches its rule's latest run
(GET /api/cleanup/audit?rule=<rule>&limit=1) and rehydrates: shows progress +
resumes polling if still running, or shows the completed result (ready/applied/
error) so the operator can act on it after returning. Adds the ?rule= filter
to the audit-history endpoint + cleanup store latestAuditForRule().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 16:55:20 -04:00
bvandeusenandClaude Opus 4.8 91b0145bc8 feat(tags): 'Reset content tagging' admin action
CI / backend-lint-and-test (push) Successful in 11s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 30s
CI / integration (push) Successful in 2m57s
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.

Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.

Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 16:47:36 -04:00
bvandeusenandClaude Opus 4.8 26e47a86cb fix(gallery): render similar-mode results (flat list when no date groups)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 22s
CI / integration (push) Successful in 2m58s
The 'See all similar' takeover fetched /api/gallery/similar fine (200, ~100
results) but the grid showed nothing: GalleryGrid renders ONLY by iterating
store.dateGroups, and similar-mode returns date_groups=[] (results are ranked
by cosine distance, not chronological). Zero groups → zero tiles despite
store.images being full. Add a flat fallback: when there are no date groups
but images exist, render them as one ungrouped list in ranked order (no date
headers). The modal Related strip was unaffected (it renders its images
directly). Test locks both the flat and grouped paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 16:24:52 -04:00
bvandeusenandClaude Opus 4.8 928e3037f0 fix(ui): purpose-built mobile layout for subscriptions hub
CI / frontend-build (push) Successful in 59s
CI / integration (push) Successful in 2m56s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 10s
Vuetify's auto card-stack was too verbose (one subscription filled the whole
phone screen) and the expanded sources still needed lateral scroll. Replace it
below 600px (useDisplay) with a custom compact-card list: each subscription is
a 2-line card (name + health + expand chevron, then platform chips + sources
count + last activity) so several fit per screen. Expanding shows the action
row + each source as a STACKED SourceCard (new) — platform/url/enabled/last/
next/errors/actions laid out vertically, no horizontal scroll. The mobile
cards drive the same selected/expanded key arrays as the desktop data-table,
so selection and bulk actions are unchanged. Desktop keeps the v-data-table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 13:40:25 -04:00
bvandeusenandClaude Opus 4.8 b08b12eb8f fix(ui): subscriptions table → card layout on mobile
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 1m8s
CI / integration (push) Successful in 2m56s
The subscriptions v-data-table (select + expand + 6 cols + a nested 8-col
sources sub-table) horizontally-scrolled on phones. Set mobile-breakpoint=600
so Vuetify stacks each subscription row into a label:value card below 600px;
the custom item slots (platform chips, health dot, action buttons) render as
card rows. The expanded sources detail reclaims its desktop indent on mobile
and keeps its own horizontal scroll for the wide source columns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 12:53:40 -04:00
bvandeusenandClaude Opus 4.8 4fd6d4cc29 fix(ui): mobile pass 2 — Posts & Subscriptions filter bars
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 34s
CI / integration (push) Successful in 2m58s
PostsFilterBar didn't wrap and its artist/platform fields had inline
min-widths (240/180px) a media query can't override → horizontal overflow on
phones. Moved widths to classes, added flex-wrap, and <600px each field takes
a full-width row. SubscriptionsTab's status/search inline max-widths likewise
moved to classes; <600px they go full-width and the v-spacer is dropped so the
search isn't shoved around.

Verified as already-fine (sweep false positives, no change): PostCard (default
body is a stacked column; only goes row at container >=800px), SeriesReaderView
(already has a <=768px block: nav drawer 150px, quick-nav stacks). The
subscriptions v-data-table scrolls horizontally within its own wrapper, so it
doesn't widen the page — a true mobile card layout is a larger follow-up if
wanted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 09:26:56 -04:00
bvandeusenandClaude Opus 4.8 304e8aa878 fix(ui): mobile responsiveness — nav hamburger + primary-path fixes
CI / backend-lint-and-test (push) Successful in 12s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 29s
CI / integration (push) Successful in 2m57s
The top nav packed brand + health + pipeline chip + ~7 inline links + an
action slot into one flex row, colliding/overflowing on phones (operator:
'almost unusable'). Below 768px the links now fold into a hamburger v-menu;
below 480px the brand text hides (glyph still brands). Plus the primary
browsing path:
- BulkEditorPanel: fixed 320px -> min(320px, 90vw) so it can't swallow the screen.
- GalleryFilterBar: <600px gives search its own full-width row (its 200px
  min-width was jamming the wrapping bar); sort grows.
- GalleryFacetPanel: <480px wraps groups + lets the side-by-side date inputs
  grow full-width.
- ArtistsView grid: minmax(min(440px,100%),1fr) so a card never overflows
  (single column on phones).
- GalleryView: hide the year/month timeline strip <600px.
ImageViewer already stacks its side panel below the image <900px (left as-is).

Secondary surfaces (Posts/Subscriptions filter bars, SubscriptionsTab table,
SeriesReader, PostCard) still need a mobile pass — follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 09:06:33 -04:00
bvandeusenandClaude Opus 4.8 0497394710 feat(ui): double showcase cadence + filter bar matches TopNav frost
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 17s
CI / integration (push) Successful in 2m56s
- Showcase reveal cadence 80ms -> 160ms (slower, more deliberate one-at-a-time
  cascade per operator). Bump showcase.spec timer advances to cover 60x160ms.
- Gallery filter bar now uses the EXACT gradiated-obsidian frost + blur as
  TopNav (was a flat rgba(...,0.55) block), so the two read as one continuous
  piece of chrome with images visibly scrolling under both; the nav's
  transparent bottom edge against the bar's opaque top leaves a faint seam that
  separates them at the very top of scroll.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 09:02:21 -04:00
bvandeusenandClaude Opus 4.8 21a73cd1dc feat(gallery): visual 'more like this' UI (Phase 3 frontend)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / integration (push) Successful in 2m57s
Modal 'Related' strip (RelatedStrip.vue) — top-12 similar thumbs, fetched on
its own DEFERRED, single-flighted path (200ms after the modal is up) so it
never blocks or slows the modal; collapses silently on empty/slow/error and is
hidden when the image has no embedding (has_embedding flag). 'See all similar'
closes the modal and navigates the gallery to ?similar_to=<id>.

Gallery store: similar_to filter field + loadSimilar() (ranked, hasMore=false,
no timeline); applyFilterFromQuery routes similar-mode to /similar with the
scope filters composed; cloneFilter/filterToQuery carry similar_to. Filter bar:
clearable 'Similar to #id' chip, sort hidden in similar-mode; timeline sidebar
hidden too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 08:52:42 -04:00
bvandeusenandClaude Opus 4.8 79cd1234e2 feat(gallery): visual 'more like this' search (Phase 3 backend)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 28s
CI / integration (push) Successful in 2m57s
GalleryService.similar() ranks images by pgvector cosine distance to a source
image's precomputed SigLIP embedding — no query-time ML inference. Composes
with the Phase-1/2 scope filters (AND) but replaces the date sort (always
nearest-first, bounded top-N, no cursor). Returns None for a missing source
(→404), [] for a source with no embedding (video / pending ML); excludes self
and NULL-embedding rows. New GET /api/gallery/similar?similar_to=<id>&limit=N.
Image-detail payload gains has_embedding so the UI can hide the surface.

Alembic 0036 adds an HNSW vector_cosine_ops index on siglip_embedding (1152<2000
dims) so the search is sub-50ms ANN instead of a full scan; one-time ~30-60s
build over existing embeddings on deploy. Shared _gallery_images/_image_json
helpers de-dup the scroll/similar builders.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 08:47:43 -04:00
bvandeusenandClaude Opus 4.8 3f6ea601f8 perf(ci): collapse the 3 integration shards into one job
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 21s
CI / integration (push) Successful in 2m55s
With fsync-off the whole integration suite runs in ~45s (was ~13min across
shards), so the 3-way split only triplicated the ~2min fixed overhead
(container + install + migrate) and consumed 3 of 6 runner slots for no
wall-clock gain. Merge intapi/intimp/intcore into one `integration` job:
spin up once, install once, migrate once, run `pytest -m integration` over
the whole suite. Frees 2 runner slots (6 jobs -> 4) and drops ~140 lines of
near-duplicate YAML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 08:17:50 -04:00
bvandeusenandClaude Opus 4.8 6a25db4b8b perf(ci): relax Postgres durability in integration shards (fsync off)
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Successful in 2m15s
CI / intcore (push) Successful in 2m28s
CI / intapi (push) Successful in 2m34s
Option 1 (pooling the teardown connection) left teardowns at ~1.5-2s/test, so
the cost is the per-test TRUNCATE's commit forcing an fsync, not the connect
handshake. Each shard now ALTER SYSTEM SETs fsync/synchronous_commit/
full_page_writes off + pg_reload_conf() right after deps install, before
alembic — sighup/user-context GUCs apply with no restart. The DB is ephemeral
(rebuilt per run) so fsync-off is safe; the step is non-fatal so a perms
surprise can't red a shard. Speeds up every test's commit (setup inserts +
the teardown TRUNCATE), stacking on the pooled engine from the prior commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:45:21 -04:00
bvandeusenandClaude Opus 4.8 c802b26406 perf(ci): reuse a session-scoped engine for the per-test DB reset
CI / intimp (push) Successful in 3m35s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m33s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 20s
The autouse integration teardown created a fresh SQLAlchemy engine + Postgres
connection for EVERY test, then disposed it — --durations showed the 15
slowest ops in both long shards were all ~1.5-2s teardowns (the connect+SCRAM
handshake, not test logic). Hoist the truncate engine to a session-scoped,
pool_pre_ping'd fixture so the pooled connection is reused across teardowns;
the TRUNCATE+restore still runs per test, so isolation is unchanged. Lazy
create_engine means the no-DB unit job instantiates but never connects.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:35:28 -04:00
bvandeusenandClaude Opus 4.8 3a4270e6be fix(showcase): reveal each tile only once its image is fully decoded
CI / backend-lint-and-test (push) Successful in 13s
CI / intapi (push) Successful in 7m41s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / intimp (push) Successful in 3m33s
CI / intcore (push) Successful in 8m23s
The buffered cascade revealed tiles on an 80ms timer regardless of image
load, so the flip-in animation played on a gray placeholder and the thumbnail
popped in afterward. Worse, MasonryGrid ALSO applied a per-index
animation-delay (index×70ms) that compounded on top of the insert cadence,
so the cascade visibly dragged and desynced as it grew.

Now the producer preloads each queued thumbnail (decode pipelined ahead) and
the consumer awaits that decode before pushing the item — every tile animates
in fully loaded, strictly one at a time. Drop the compounding CSS stagger;
the store's one-item-at-a-time push is the sole pacer, so each tile animates
the instant it mounts. New utils/preloadImage.js (load+decode+timeout gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:19:44 -04:00
bvandeusenandClaude Opus 4.8 ae569c0f9a fix(gallery): ruff C408 (dict literal) + panel auto-open on deep-link
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m29s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 9m2s
Rewrite facets() common/plat_scope as dict literals (C408). Open the refine
panel via a watch on hasRefineFilters rather than reading filter state at
bar-setup time — the parent applies the URL query in its onMounted, after the
bar child has set up, so the initial read was always the default (empty) state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:14:15 -04:00
bvandeusenandClaude Opus 4.8 1adc47f59c feat(gallery): faceted refine panel UI (Phase 2 frontend)
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m30s
Add a 'Refine ▾' toggle to the gallery filter bar that expands a full-width
GalleryFacetPanel below it, inside the same sticky hazey chrome. The panel
offers platform chips (with live counts + a 'No platform' unsourced bucket),
two count-badged curation-flag toggles (Untagged / No artist), and a from/to
date range bounded by the facet min/max.

Store gains the platform/untagged/no_artist/date_from/date_to filter params
(URL-mirrored, AND-composed) and a panel-gated, single-flighted loadFacets()
that fetches /api/gallery/facets scoped to the active filter. Shared
cloneFilter/filterToQuery helpers keep the bar and panel writing one URL
format. The panel auto-opens on deep-link when refine filters are present and
refetches counts (debounced) on every filter change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:06:51 -04:00
bvandeusenandClaude Opus 4.8 9fe534139a feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 3m34s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m50s
Extend the composable gallery filter with platform / untagged / no_artist /
date_from / date_to, AND-composed with the existing tag/artist/media/sort
params and threaded through scroll, timeline, and jump_cursor.

Add GalleryService.facets() + GET /api/gallery/facets returning live counts
scoped to the current filter with per-group minus-self semantics: platform
counts (COUNT(DISTINCT image) incl. a null unsourced bucket), curation-flag
counts (untagged / no_artist), and effective_date min/max bounds. The
UNSOURCED_PLATFORM sentinel makes filesystem-imported content reachable via
the platform facet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 07:01:07 -04:00
bvandeusenandClaude Opus 4.8 16e0268da7 fix(modal): next arrow clears metadata panel + arrows work in empty tag input
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 25s
CI / intimp (push) Successful in 3m35s
CI / intapi (push) Successful in 7m41s
CI / intcore (push) Successful in 8m23s
Image viewer (#609):
- The next (▶) arrow was offset from the viewport edge (right:16px) so it
  floated over the 320px metadata side panel. Offset it off a shared
  --fc-side-w var so it sits at the image's right edge instead; full-width
  again below 900px when the panel stacks under the image.
- Arrow nav was fully disabled whenever a text field was focused. Now it
  yields to the caret ONLY when the field has text; an empty tag-entry field
  still navigates ←/→. Extracted to utils/textEntry.js (arrowNavAllowed).
  ESC behaviour unchanged (already closes the modal, overlay-aware).

Test: arrowNavAllowed — empty/non-text → navigate, text present → don't.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 06:37:16 -04:00
bvandeusenandClaude Opus 4.8 1f4ce8513b style: ruff I001 — keep _sync_engine as-import on its own line (combine-as-imports=false)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Successful in 19s
CI / intapi (push) Successful in 8m17s
CI / intcore (push) Successful in 8m38s
CI / intimp (push) Successful in 4m6s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:59:26 -04:00
bvandeusenandClaude Opus 4.8 5a116ca9d0 style: ruff I001 — aliased _sync_session_factory sorts before get_sync_engine
CI / lint (push) Failing after 4s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m49s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 9m37s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:55:18 -04:00
bvandeusenandClaude Opus 4.8 ef3ee5aceb feat(maintenance): DB maintenance UI card + fix ruff I001
CI / lint (push) Failing after 2s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 18s
CI / intimp (push) Successful in 4m10s
CI / intapi (push) Successful in 8m26s
CI / intcore (push) Successful in 9m54s
- Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM
  ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat
  readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats.
- dbMaintenance store (loadStats / runVacuum) + test.
- Fix ruff I001: combine the two _sync_engine imports onto one line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:52:36 -04:00
bvandeusenandClaude Opus 4.8 914033db29 feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Successful in 23s
CI / intimp (push) Successful in 3m35s
CI / intapi (push) Successful in 7m53s
CI / intcore (push) Successful in 9m15s
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.

- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
  (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
  Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
  autocommit connection.
- GET  /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
  last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
  black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.

Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:49:39 -04:00
bvandeusenandClaude Opus 4.8 d495605c12 style(gallery): hazey filter bar attached to the TopNav
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m43s
CI / intapi (push) Successful in 7m47s
CI / intcore (push) Successful in 8m49s
- Filter bar gets the same obsidian translucent + backdrop-blur as the
  TopNav so the two read as one piece of chrome.
- margin-top:-8px cancels the v-container's pt-2 so the bar sits flush at
  64px even at scroll 0 — fixes the gap/separation when scrolled to top.
- Inputs/toggles get a more-opaque backing so they stay legible on the haze.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:40:32 -04:00
bvandeusenandClaude Opus 4.8 76d8ad42a8 fix(showcase): buffered producer/consumer for a steady cascade
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m35s
CI / intcore (push) Successful in 9m8s
The cascade "burped" — chunks appeared unevenly — because the old pipeline
coupled display to fetch timing: it trickled each batch right after its
fetch and assumed the next round-trip would land inside the ~240ms trickle
window. When a fetch ran long (TABLESAMPLE hits random, sometimes-cold
blocks; RTT jitter) the animation starved, then a clump burst in.

Decouple the two:
- Producer (_fill) races ahead fetching batches into a buffer up to a
  target depth, refilling when it dips below BUFFER_MIN.
- Consumer (_drain) reveals one item every CADENCE_MS regardless of when
  fetches land; it only waits if the buffer genuinely starves.

A small PRIME buffer precedes the drain so it doesn't starve at the front;
the buffer (BUFFER_MIN×CADENCE runway) absorbs per-fetch jitter so images
appear at an even pace. Public store API (loadInitial/shuffle/fetchPage/
images/loading/hasMore/isEmpty) unchanged — ShowcaseView/MasonryGrid need
no change.

Test (fake timers): fire-order + dedup, one-item-per-cadence rate limit,
empty-library flag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:35:25 -04:00
bvandeusenandClaude Opus 4.8 6d630d13d6 feat(gallery): pinned filter bar (Phase 1)
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 29s
CI / intimp (push) Successful in 3m29s
CI / intapi (push) Successful in 7m24s
CI / intcore (push) Successful in 8m4s
Gallery now has in-view filtering, styled like the app's sticky v-tabs
chrome (pinned at top:64px under TopNav).

- GalleryFilterBar: combined tag+artist autocomplete (searches
  /api/tags + /api/artists), closable filter chips (multi-tag AND),
  media toggle (All/Images/Videos), Newest/Oldest sort, Clear. Writes all
  state to the URL via router.push.
- gallery store: filter is now { tag_ids, artist_id, media_type, sort,
  post_id }; applyFilterFromQuery makes the URL the single source of truth
  (deep-linkable, back-button works); chip labels resolved by id or
  pre-noted on pick. Replaces the standalone tag chip + setTag/PostFilter.
- GalleryView: renders the bar (hidden in post-detail), syncs route.query
  → store on mount + every query change.

Also untracks the transient .claude/scheduled_tasks.lock committed in
3f30327 and gitignores it.

Tests: store parses query → composable scroll params, post_id exclusivity,
newest-sort omitted, label pre-seed, single initial fetch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:57:11 -04:00
bvandeusenandClaude Opus 4.8 3f30327fa5 feat(gallery): composable scroll filter (multi-tag AND, media, sort)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 17s
CI / frontend-build (push) Successful in 24s
CI / intimp (push) Successful in 3m50s
CI / intcore (push) Successful in 8m33s
CI / intapi (push) Successful in 7m42s
Phase 1 backend for the gallery filter bar. Extends scroll/timeline/jump
from a single mutually-exclusive filter to a composable one:

- tag_ids: image must carry ALL of them (one correlated EXISTS per tag —
  AND, no row multiplication), replacing the single-tag JOIN.
- artist_id composes with tags; media_type ('image'|'video') narrows by
  mime; post_id stays the exclusive post-detail path.
- sort ('newest'|'oldest') flips the effective_date/id cursor comparison
  and ordering; the cursor value is unchanged (direction comes from the
  request). jump_cursor honors sort too.
- Shared _apply_scope helper applied across scroll/timeline/jump so the
  timeline sidebar reflects the filtered set. API _parse_filters parses
  tag_id (comma list), artist_id, media, sort.

Tests: multi-tag AND, media filter, sort reversal (service + API);
post_id-excludes-others; single tag_id back-compat.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:50:19 -04:00
bvandeusenandClaude Opus 4.8 4f9464d215 feat(gallery,tags): clear active filters
CI / backend-lint-and-test (push) Successful in 13s
CI / intcore (push) Successful in 8m36s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 38s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m5s
Two gaps where a filter couldn't be removed:

- Gallery: a tag_id filter (from clicking a tag) had no indicator or clear
  control — only post_id did (PostInfoHeader). Add an "Tag: <name> ✕" chip
  that clears the filter by dropping tag_id from the URL. New lightweight
  GET /api/tags/<id> resolves the name; the store fetches it on filter set.
- Tags view: the kind chip-group used mandatory="false" — a STRING ("false"
  is truthy in JS), which made the group mandatory so the active kind chip
  couldn't be deselected. Fixed to :mandatory="false" so the filter clears.

Tests: GET /tags/<id> shape + 404; gallery store resolves filterTagName.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:26:04 -04:00
bvandeusenandClaude Opus 4.8 e678d1dfdf feat(tags): fandom-edit UI in tags directory + image modal
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 13s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m47s
CI / intapi (push) Successful in 7m52s
CI / intcore (push) Successful in 8m58s
Adds the missing UI to change a character tag's fandom, in both places:

- FandomSetDialog (shared): pick an existing fandom, create a new one, or
  clear it; on a name collision in the target fandom it surfaces a merge
  confirmation and resolves via setFandom(merge:true). Reuses the tags
  store's fandom cache.
- TagCard kebab gains "Set fandom…" for character tags (→ TagsView opens
  the dialog, reloads on success).
- TagPanel chip kebab gains "Set fandom…" for character tags (→ reloads the
  modal's tag list on success).
- tags store: setFandom(tagId, fandomId, {merge}) action + test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:21:25 -04:00
bvandeusenandClaude Opus 4.8 d9ab6e15c6 feat(tags): edit a character tag's fandom (backend)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 27s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m46s
CI / intcore (push) Successful in 8m29s
No way existed to change which fandom a character tag belongs to after
creation — PATCH /tags/<id> only renamed.

- TagService.set_fandom(tag_id, fandom_id, merge=False): set / change /
  clear (fandom_id=None) a character's fandom, with the same validation as
  find_or_create. On a name collision in the target fandom it raises
  TagMergeConflict (→ 409, same shape as rename); merge=True resolves it by
  merging this tag INTO the existing character.
- Extract _do_merge(source, target) from merge() so set_fandom can perform
  the deliberate CROSS-fandom merge the public merge() validation forbids.
- PATCH /tags/<id> now accepts optional fandom_id (+ merge flag) alongside
  name, and returns fandom_id.

Tests: set/change/clear, non-character + bad-ref rejection, collision
raises, merge resolves; API set/clear + collision→merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:13:24 -04:00
bvandeusenandClaude Opus 4.8 e05e0b9f37 perf(gallery): materialize indexed effective_date sort key
CI / lint (push) Successful in 3s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m47s
CI / intcore (push) Successful in 8m19s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 24s
The gallery cursored on COALESCE(post.post_date, image_record.created_at)
across the Post outer join — an expression spanning two tables that no
index can serve, so every /scroll sorted a large slice of the library
(and the old frontend fired ten serially). Materialize it:

- image_record.effective_date column + ix_image_record_effective_date
  (effective_date DESC, id DESC); alembic 0035 backfills
  COALESCE(primary post's post_date, created_at) for existing rows.
- gallery_service._effective_date_col() now returns the column, so scroll
  / timeline / jump / neighbors all order off the index instead of
  re-deriving the COALESCE. _neighbors reads record.effective_date
  directly (drops an extra Post lookup).
- importer._apply_sidecar maintains it: when a primary post with a date is
  linked, effective_date = post.post_date; plain inserts keep the
  created_at-equivalent server default.

Tests: sidecar import asserts effective_date == post.post_date; gallery
ordering/timeline/jump test seeds set effective_date alongside created_at.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:58:46 -04:00
bvandeusenandClaude Opus 4.8 56cc253009 feat(gallery): reveal tiles on image load + single initial fetch
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 29s
CI / intcore (push) Successful in 8m22s
CI / intimp (push) Successful in 3m43s
CI / intapi (push) Successful in 7m43s
The 5×10 metadata batching only staggered the cheap layer (JSON);
thumbnails load as independent <img> requests and clustered, so tiles
"popped in together" after a wait. Two changes:

- GalleryItem reveals each tile when ITS OWN thumbnail fires @load (with
  an onMounted complete-check for cached thumbs), playing a showcase-style
  flip-up entrance. Tiles now cascade in natural load order instead of all
  at once. Honors prefers-reduced-motion.
- gallery store does ONE initial fetch (limit=50) instead of 10 serial
  /scroll round-trips. Fewer RTTs, faster first paint; the reveal-on-load
  is what makes appearance progressive now. Infinite scroll pulls 25/trigger.

Tests: GalleryItem gains is-loaded only after @load; loadInitial issues
exactly one scroll request at the initial limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:50:03 -04:00
bvandeusenandClaude Opus 4.8 576e16d14d fix(download): release DB connections across the gallery-dl subprocess
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 27s
CI / intimp (push) Successful in 3m30s
CI / intapi (push) Successful in 7m19s
CI / intcore (push) Successful in 8m6s
Backfill events were STILL stranding empty after the timeout-ladder fix.
Worker logs showed the salvage path working ("Download timeout for
anduo/patreon after 1170.0s (18 files written)") but then:
  Retry in 3s: DBAPIError(ConnectionDoesNotExistError: connection was
  closed in the middle of operation)
  ...succeeded in 0.149s   <- in-flight guard no-op

Root cause: DownloadService held the async + sync DB connections checked
out across the entire (≤19.5-min backfill) gallery-dl subprocess. The
server reaps the idle connection, so phase 3's first query hits a dead
socket. That DBAPIError trips download_source's autoretry_for, the retry
re-enters _phase1_setup, sees the event still 'running', returns
in_flight and no-ops — leaving the event to be stranded empty by the
recovery sweep. pool_pre_ping was already on both engines but can't help
a *held* connection (it only validates on pool checkout).

Fix:
- DownloadService.download_source closes the async + sync sessions after
  phase 1, before the subprocess, so phase 3 re-acquires a live
  connection (matches the class's "Phase 2 — no DB connection" docstring).
- The per-task async engine switches to NullPool so phase 3 always opens
  a fresh connection rather than a pooled one the server may have reaped.

Tests: assert connections are released before gdl.download runs and the
event still finalizes; assert the task engine uses NullPool. Also fixes a
stale 1800s->1170s comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 21:49:52 -04:00
bvandeusenandClaude Opus 4.8 9cb24c9e1b style(test): fix ruff I001 import order in download task test
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m16s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 17:04:29 -04:00
bvandeusenandClaude Opus 4.8 6590dcdb39 fix(download): salvage soft-time-limit kills + fix timeout ladder
CI / frontend-build (push) Successful in 21s
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 21s
CI / intimp (push) Successful in 3m32s
CI / intapi (push) Successful in 7m22s
CI / intcore (push) Successful in 8m4s
Backfill downloads stranded with empty logs + a generic "stranded by
recovery sweep" error. Root cause: the backfill gallery-dl subprocess
timeout (1170s) exceeded download_source's Celery soft_time_limit (900s),
so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The
TimeoutExpired path (which captures partial stdout/stderr and finalizes
the event) never ran, the event was left 'running', and phase 3 never
decremented backfill_runs_remaining — so the source re-ran and
re-stranded every tick (Anduo #39912).

Two layers:
1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both
   subprocess budgets (870 tick / 1170 backfill) sit below the soft
   limit with phase-3 persist headroom. Promote to module constants and
   guard the invariant with a test.
2. Catch SoftTimeLimitExceeded in download_source and finalize the
   in-flight event with a real reason, mirror phase-3 source-health, and
   decrement backfill so a chronically-slow source self-heals to tick
   mode. The existing celery_signals handler only covered TaskRun, not
   DownloadEvent — that was the gap.

Updates stale 900/1200 references in gallery_dl.py + maintenance.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 16:56:13 -04:00
bvandeusen 3162cff96b fix(artist): ruff UP017 + test_directory_card_shape pin
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 27s
CI / intimp (push) Successful in 3m29s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 8m4s
Two CI bounces on b65e956:
1. ruff UP017 — Python 3.14's preferred form is `datetime.UTC`, not
   `timezone.utc`. Switch the test's two TZ literals.
2. test_directory_card_shape pinned the card key set to the pre-feature
   shape; `unseen_count` was added to the API payload but the pin
   wasn't updated. Same shape as the recurring 'plan-grep-pinned-tests'
   trap — should have grepped tests/ for card.keys() before pushing.
2026-06-03 15:45:59 -04:00
bvandeusen b65e956ad2 feat(artist): "new since last visit" badge + banner
CI / lint (push) Failing after 2s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 22s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Failing after 7m41s
CI / intcore (push) Successful in 8m42s
Per-artist "+N" accent pill on the artists directory and a "N new since
last visit" banner inside ArtistView. Counts new IMAGES (not posts) so
multi-image posts increment correctly.

- alembic 0034: artist_visit (artist_id PK, last_viewed_at NOT NULL).
  Seeds every existing artist with last_viewed_at=NOW() so the badge
  starts at 0 across the board — no noisy "5000 unseen images" on
  first deploy.
- ArtistService.find_or_create autoseeds a visit row alongside new
  artists, so freshly imported content doesn't read as unseen.
- ArtistService.overview reads pre-visit last_viewed_at, counts images
  created since, then atomically UPSERTs last_viewed_at=NOW() via
  postgres ON CONFLICT DO UPDATE (no SELECT-then-INSERT race per
  reference_scalar_one_or_none_duplicates). Returns the pre-update
  count as `unseen_count_at_visit` so the banner has data.
- ArtistDirectoryService.list_artists adds an `unseen_count` aggregate
  to each card via LEFT JOIN artist_visit + conditional COUNT. NULL
  last_viewed_at (artist created before this code shipped) defensively
  counts as "never visited" → all images unseen.
- Frontend: ArtistCard renders an accent pill in the preview-strip
  corner when unseen_count > 0 (capped at 99+); ArtistView shows a
  closable v-alert banner on initial load when
  unseen_count_at_visit > 0, re-arms on slug change.

Single-row-per-artist (no user_id) — rule #47 multi-user ACL is
aspirational; widens to (user_id, artist_id) PK when User lands, per
rule #22.

Scribe plan #597.
2026-06-03 15:27:11 -04:00
120 changed files with 9247 additions and 1294 deletions
+30 -149
View File
@@ -92,28 +92,25 @@ jobs:
- run: npm run test:unit
- run: npm run build
# Integration suite split into THREE parallel shards (2026-05-25, runner
# capacity bumped 2→6). Each shard gets its own Postgres + Redis service
# set and runs alembic + a disjoint subset of integration tests. Shards
# share no DB state, so the autouse TRUNCATE fixture in tests/conftest.py
# stays single-threaded per shard but multiple shards run in parallel
# wall-clock. Approximate split — rebalance once --durations=15 output
# reveals which shard is the long pole.
# Single integration job — collapsed from a 3-way shard split on 2026-06-04.
# The shards existed to parallelize ~8.5min of integration tests; once the
# throwaway Postgres runs with fsync OFF (the durability step below) the whole
# suite runs in ~45s, so the split only triplicated the ~2min fixed overhead
# (container + `uv pip install` + `alembic upgrade head`) and burned 3 of 6
# runner slots for no wall-clock gain. One job now: spin up once, install
# once, migrate once, run every integration test.
#
# Each shard's docker-ps filter uses its own unique job name to scope
# service-container resolution. act_runner appears to strip underscores
# from job names when building container labels — `int_api` yielded
# zero matches on 2026-05-25 — so shards use no-separator names
# (`intapi`, `intimp`, `intcore`) instead. Each step prints
# `docker ps -a` first so a future naming-convention shift surfaces in
# the log without another guess-and-push cycle.
# The docker-ps filter scopes to THIS job's own Postgres/Redis service
# containers by job name. act_runner strips underscores from job names when
# labelling containers (`int_api` matched nothing on 2026-05-25), so the name
# stays separator-free (`integration`). The step prints `docker ps -a` first
# so a future naming-convention shift surfaces in the log without a
# guess-and-push cycle.
#
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT
# done — per ci-requirements.md, FC is the only Python consumer of that
# image and the CI-Runner project's "add deps to image when used by >1
# project" rule keeps the install per-job.
intapi:
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT done —
# per ci-requirements.md, FC is the only Python consumer of that image and the
# CI-Runner "add deps to image when used by >1 project" rule keeps it per-job.
integration:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
@@ -144,14 +141,14 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: API integration shard (resolve service IPs, migrate, test)
- name: Integration suite (resolve service IPs, migrate, test)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intapi" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intapi" --filter "ancestor=redis:7-alpine" -q | head -n1)
PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -168,130 +165,14 @@ jobs:
else
pip install -r requirements.txt pytest pytest-asyncio
fi
# Relax durability on the throwaway CI Postgres so the per-test
# TRUNCATE's commit-fsync — the integration teardown's dominant cost
# (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
# skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
# is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
# NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
# surprise can't red the job; fabledcurator is the postgres image's
# bootstrap superuser.
python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
alembic upgrade head
pytest tests/test_api_*.py -v -m integration --durations=15
intimp:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Importer integration shard (resolve service IPs, migrate, test)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intimp" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intimp" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/test_importer*.py tests/test_import_*.py tests/test_migration_*.py tests/test_phash_*.py tests/test_sidecar_*.py tests/test_scan_*.py tests/test_archive_extractor.py tests/test_backfill_phash.py -v -m integration --durations=15
intcore:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intcore" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intcore" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration --durations=15 \
--ignore-glob='tests/test_api_*.py' \
--ignore-glob='tests/test_importer*.py' \
--ignore-glob='tests/test_import_*.py' \
--ignore-glob='tests/test_migration_*.py' \
--ignore-glob='tests/test_phash_*.py' \
--ignore-glob='tests/test_sidecar_*.py' \
--ignore-glob='tests/test_scan_*.py' \
--ignore-glob='tests/test_archive_extractor.py' \
--ignore-glob='tests/test_backfill_phash.py'
pytest tests/ -v -m integration --durations=15
+3
View File
@@ -61,6 +61,9 @@ Thumbs.db
# Claude Code per-user local overrides (shared .claude/settings.json is OK to commit)
.claude/settings.local.json
# Transient scheduler lock/state (committed by accident in 3f30327)
.claude/scheduled_tasks.lock
.claude/scheduled_tasks*.json
# Alembic / DB scratch
alembic/versions/__pycache__/
+53
View File
@@ -0,0 +1,53 @@
"""artist_visit: per-artist last-viewed timestamp for the "+N new" badge
Revision ID: 0034
Revises: 0033
Create Date: 2026-06-03
Powers the artists-directory "+N new since last visit" badge + ArtistView
banner. Single row per artist (no user_id yet — rule #47 multi-user ACL
is aspirational; widens to (user_id, artist_id) PK when User lands).
Seed every existing artist with `last_viewed_at = NOW()` so the badge
starts at 0 across the board — no noisy "you have 5000 unseen images"
on first deploy. New artists auto-get a row via
`ArtistService.find_or_create`.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0034"
down_revision: Union[str, None] = "0033"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"artist_visit",
sa.Column(
"artist_id",
sa.Integer,
sa.ForeignKey("artist.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column(
"last_viewed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
)
# Seed: every existing artist starts "fully caught up". Without this,
# every operator with N artists would see N badges (worth of every
# image ever imported) on first deploy.
op.execute(
"INSERT INTO artist_visit (artist_id, last_viewed_at) "
"SELECT id, NOW() FROM artist"
)
def downgrade() -> None:
op.drop_table("artist_visit")
@@ -0,0 +1,70 @@
"""image_record.effective_date: materialized gallery sort key + index
Revision ID: 0035
Revises: 0034
Create Date: 2026-06-04
The gallery ordered/cursored on COALESCE(post.post_date,
image_record.created_at) across the Post outer join. That expression spans
two tables, so no index can serve it — every /scroll sorted a large slice
of the library, and the frontend fired ten of them serially per initial
load. Materialize the value into image_record.effective_date and index
(effective_date DESC, id DESC) so the cursor scroll is an index range scan.
Backfill = COALESCE(primary post's post_date, created_at) so existing rows
keep their exact ordering. New rows get the created_at-equivalent server
default; services/importer.py overrides it with the post's date when a
primary post with a date is linked.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0035"
down_revision: Union[str, None] = "0034"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Add nullable first so the backfill can populate before NOT NULL.
op.add_column(
"image_record",
sa.Column("effective_date", sa.DateTime(timezone=True), nullable=True),
)
# Pure set-based UPDATEs (no per-row params) — immune to the 65535
# bind-parameter ceiling regardless of library size.
op.execute(
"""
UPDATE image_record AS ir
SET effective_date = COALESCE(p.post_date, ir.created_at)
FROM post AS p
WHERE ir.primary_post_id = p.id
"""
)
op.execute(
"""
UPDATE image_record
SET effective_date = created_at
WHERE effective_date IS NULL
"""
)
op.alter_column(
"image_record",
"effective_date",
nullable=False,
server_default=sa.text("now()"),
)
# DESC/DESC matches the gallery's ORDER BY effective_date DESC, id DESC
# so the scroll is a forward index scan; raw SQL because alembic's
# column list doesn't express per-column DESC cleanly.
op.execute(
"CREATE INDEX ix_image_record_effective_date "
"ON image_record (effective_date DESC, id DESC)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_effective_date", table_name="image_record")
op.drop_column("image_record", "effective_date")
@@ -0,0 +1,41 @@
"""image_record.siglip_embedding: HNSW cosine index for "more like this"
Revision ID: 0036
Revises: 0035
Create Date: 2026-06-04
Gallery Phase 3 (visual similarity search) ranks images by
`siglip_embedding.cosine_distance(source_embedding)`. Without an index that's
a sequential scan computing a 1152-dim distance for every row — fine at small
scale, but it grows linearly with the library. Add an HNSW index with
`vector_cosine_ops` so the top-N nearest search is sub-50ms ANN.
1152 dims is under pgvector's 2000-dim HNSW limit, so HNSW (no training,
better recall than IVFFlat) is the right choice. ONE-TIME COST: building the
index over the existing embeddings (~57k vectors on the operator's library)
locks image_record for ~30-60s during this migration on deploy — acceptable
for a single-operator homelab. NULL embeddings (videos / not-yet-embedded
rows) are simply not indexed.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0036"
down_revision: Union[str, None] = "0035"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Raw SQL: alembic's create_index doesn't express the `USING hnsw (...
# vector_cosine_ops)` access-method + opclass cleanly. Must match the
# query's cosine_distance operator class to be usable by the planner.
op.execute(
"CREATE INDEX ix_image_record_siglip_hnsw "
"ON image_record USING hnsw (siglip_embedding vector_cosine_ops)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_siglip_hnsw", table_name="image_record")
@@ -0,0 +1,53 @@
"""patreon_seen_media: per-source ledger of already-ingested Patreon media
Revision ID: 0037
Revises: 0036
Create Date: 2026-06-05
Native Patreon ingester (build step 2a). Replaces gallery-dl's
archive.sqlite3 with our own queryable table. The downloader upserts one
row per (source, media) so routine walks skip media we've already
processed; a future "recovery" mode bypasses the ledger to re-walk.
`filehash` is a 32-hex Patreon CDN MD5, OR a video sentinel of the form
``video:<post_id>:<media_id>`` — hence String(128). The unique
constraint on (source_id, filehash) is the dedup upsert key.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0037"
down_revision: Union[str, None] = "0036"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"patreon_seen_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("post_id", sa.String(64), nullable=True),
sa.Column(
"seen_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_patreon_seen_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("patreon_seen_media")
+68 -1
View File
@@ -19,7 +19,7 @@ from __future__ import annotations
import hashlib
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from sqlalchemy import select, text
from ..extensions import get_session
from ..models import Artist
@@ -222,3 +222,70 @@ async def tags_purge_legacy():
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
)
return jsonify(result)
@admin_bp.route("/tags/reset-content", methods=["POST"])
async def tags_reset_content():
"""Tier-A: delete ALL general + character tags (the Camie-suggestable
content vocabulary) so the operator can re-tag from scratch via
auto-suggest. fandom + series tags + series_page ordering are preserved,
and image tagger_predictions are untouched so suggestions repopulate.
dry-run preview returns per-kind counts + applications + a sample so the
UI shows exactly what'll go before the operator confirms (dry_run=false).
Irreversible except via DB backup restore."""
from ..services.cleanup_service import reset_content_tagging
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: reset_content_tagging(sync_sess, dry_run=dry_run)
)
return jsonify(result)
@admin_bp.route("/maintenance/db-stats", methods=["GET"])
async def db_stats():
"""Per-table bloat readout (pg_stat_user_tables) for the high-churn tables
so the operator can see when a VACUUM is worth running."""
from ..tasks.maintenance import VACUUM_TABLES
wanted = set(VACUUM_TABLES)
async with get_session() as session:
rows = (await session.execute(text(
"SELECT relname, n_live_tup, n_dead_tup, last_vacuum, "
"last_autovacuum, last_analyze FROM pg_stat_user_tables"
))).all()
def _iso(v):
return v.isoformat() if v is not None else None
out = []
for r in rows:
if r.relname not in wanted:
continue
live = r.n_live_tup or 0
dead = r.n_dead_tup or 0
total = live + dead
out.append({
"table": r.relname,
"live": live,
"dead": dead,
"dead_pct": round(100 * dead / total, 1) if total else 0.0,
"last_vacuum": _iso(r.last_vacuum),
"last_autovacuum": _iso(r.last_autovacuum),
"last_analyze": _iso(r.last_analyze),
})
out.sort(key=lambda t: t["dead"], reverse=True)
return jsonify({"tables": out})
@admin_bp.route("/maintenance/vacuum", methods=["POST"])
async def trigger_vacuum():
"""Operator-triggered VACUUM (ANALYZE) over the high-churn tables — the
same maintenance-queue task the weekly Beat schedule runs."""
from ..tasks.maintenance import vacuum_analyze
vacuum_analyze.delay()
return jsonify({"status": "queued"}), 202
+8 -5
View File
@@ -154,12 +154,15 @@ async def audit_history():
limit = min(int(request.args.get("limit", "20")), 100)
except ValueError:
return _bad("invalid_limit")
# Optional rule filter so a card can reconnect to ITS latest run on mount
# (?rule=transparency&limit=1) — the audit survives navigation; the UI
# rehydrates from this rather than losing the in-flight scan.
rule = request.args.get("rule") or None
async with get_session() as session:
rows = (await session.execute(
select(LibraryAuditRun)
.order_by(LibraryAuditRun.id.desc())
.limit(limit)
)).scalars().all()
stmt = select(LibraryAuditRun).order_by(LibraryAuditRun.id.desc())
if rule is not None:
stmt = stmt.where(LibraryAuditRun.rule == rule)
rows = (await session.execute(stmt.limit(limit))).scalars().all()
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
+12 -10
View File
@@ -121,13 +121,15 @@ async def delete_credential(platform: str):
@credentials_bp.route("/<platform>/verify", methods=["POST"])
async def verify_credential(platform: str):
"""Test the stored credential by running gallery-dl --simulate
against one of the platform's enabled sources. On success stamps
last_verified. Returns {valid: bool|null, reason, last_verified?}.
valid=null means "couldn't test" (no credential, or no enabled
source to point at)."""
"""Test the stored credential against one of the platform's enabled sources,
WITHOUT downloading. Routes through the platform's backend
(download_backends.verify_credential) — native ingester for Patreon, an
authenticated API page; gallery-dl --simulate for the rest. On success
stamps last_verified. Returns {valid: bool|null, reason, last_verified?};
valid=null means "couldn't test" (no credential, no enabled source, or an
inconclusive network/drift result)."""
from ..models import Artist, Source
from ..services.gallery_dl import GalleryDLService, SourceConfig
from ..services.download_backends import verify_source_credential
async with get_session() as session:
if not await _ext_key_ok(session):
@@ -154,14 +156,14 @@ async def verify_credential(platform: str):
cookies_path = await svc.get_cookies_path(platform)
auth_token = await svc.get_token(platform)
gdl = GalleryDLService(images_root=Path("/images"))
ok, message = await gdl.verify(
ok, message = await verify_source_credential(
platform=platform,
url=source.url,
artist_slug=artist.slug,
platform=platform,
source_config=SourceConfig.from_dict(source.config_overrides or {}),
config_overrides=source.config_overrides or {},
cookies_path=str(cookies_path) if cookies_path else None,
auth_token=auth_token,
images_root=Path("/images"),
)
last_verified = None
+131 -43
View File
@@ -1,4 +1,6 @@
"""Gallery API: cursor scroll, timeline, jump, image detail."""
"""Gallery API: cursor scroll, timeline, jump, image detail, facets."""
from datetime import UTC, datetime, timedelta
from quart import Blueprint, jsonify, request
@@ -8,46 +10,88 @@ from ..services.gallery_service import GalleryService
gallery_bp = Blueprint("gallery", __name__, url_prefix="/api/gallery")
def _image_json(i):
"""Serialize a GalleryImage for the scroll/similar list responses."""
return {
"id": i.id,
"sha256": i.sha256,
"mime": i.mime,
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
def _parse_date(raw):
"""Parse a YYYY-MM-DD query value to a UTC midnight datetime, or None.
Raises ValueError (→ 400) on a malformed value."""
if not raw:
return None
return datetime.strptime(raw, "%Y-%m-%d").replace(tzinfo=UTC)
def _parse_filters():
"""Parse the composable gallery filters from query args, returning
``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates.
`tag_id` accepts a single id or a comma-separated list (AND); `media` is
image|video; `sort` is newest|oldest; `platform` selects one platform
(or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are boolean
flags; `date_from`/`date_to` are inclusive calendar-day bounds (date_to is
widened by a day so the whole day is covered by the service's half-open
`< date_to`)."""
tag_raw = request.args.get("tag_id")
tag_ids = (
[int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None
) or None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
media = request.args.get("media")
media_type = media if media in ("image", "video") else None
sort = request.args.get("sort")
sort = sort if sort in ("newest", "oldest") else "newest"
platform = request.args.get("platform") or None
untagged = request.args.get("untagged") in ("1", "true", "yes")
no_artist = request.args.get("no_artist") in ("1", "true", "yes")
date_from = _parse_date(request.args.get("date_from"))
date_to = _parse_date(request.args.get("date_to"))
if date_to is not None:
date_to += timedelta(days=1) # inclusive of the date_to calendar day
filters = {
"tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id,
"media_type": media_type, "platform": platform,
"untagged": untagged, "no_artist": no_artist,
"date_from": date_from, "date_to": date_to,
}
return filters, sort
@gallery_bp.route("/scroll", methods=["GET"])
async def scroll():
cursor = request.args.get("cursor") or None
try:
limit = int(request.args.get("limit", "50"))
filters, sort = _parse_filters()
except ValueError:
return jsonify({"error": "limit must be an integer"}), 400
tag_id_raw = request.args.get("tag_id")
tag_id = int(tag_id_raw) if tag_id_raw else None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
return jsonify({"error": "invalid filter or limit parameter"}), 400
async with get_session() as session:
svc = GalleryService(session)
try:
page = await svc.scroll(
cursor=cursor, limit=limit, tag_id=tag_id,
post_id=post_id, artist_id=artist_id,
cursor=cursor, limit=limit, sort=sort, **filters,
)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(
{
"images": [
{
"id": i.id,
"sha256": i.sha256,
"mime": i.mime,
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
for i in page.images
],
"images": [_image_json(i) for i in page.images],
"next_cursor": page.next_cursor,
"date_groups": [
{"year": y, "month": m, "image_ids": ids} for y, m, ids in page.date_groups
@@ -56,20 +100,46 @@ async def scroll():
)
@gallery_bp.route("/timeline", methods=["GET"])
async def timeline():
tag_id_raw = request.args.get("tag_id")
tag_id = int(tag_id_raw) if tag_id_raw else None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
@gallery_bp.route("/similar", methods=["GET"])
async def similar():
"""Visual "more like this": images ranked by cosine distance to the
`similar_to` image's embedding. Composes with the scope filters (AND) but
ignores post_id and sort. Bounded top-N, no cursor."""
try:
similar_to = int(request.args["similar_to"])
limit = int(request.args.get("limit", "100"))
filters, _sort = _parse_filters()
except (KeyError, ValueError):
return jsonify({"error": "similar_to query param required"}), 400
# post_id is the exclusive post-detail view — not a similarity scope.
scope = {k: v for k, v in filters.items() if k != "post_id"}
async with get_session() as session:
svc = GalleryService(session)
try:
buckets = await svc.timeline(
tag_id=tag_id, post_id=post_id, artist_id=artist_id
)
images = await svc.similar(image_id=similar_to, limit=limit, **scope)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
if images is None:
return jsonify({"error": "not found"}), 404
return jsonify(
{
"images": [_image_json(i) for i in images],
"next_cursor": None,
"date_groups": [],
}
)
@gallery_bp.route("/timeline", methods=["GET"])
async def timeline():
try:
filters, _sort = _parse_filters()
except ValueError:
return jsonify({"error": "invalid filter parameter"}), 400
async with get_session() as session:
svc = GalleryService(session)
try:
buckets = await svc.timeline(**filters)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(
@@ -77,25 +147,43 @@ async def timeline():
)
@gallery_bp.route("/facets", methods=["GET"])
async def facets():
try:
filters, _sort = _parse_filters()
except ValueError:
return jsonify({"error": "invalid filter parameter"}), 400
async with get_session() as session:
svc = GalleryService(session)
try:
f = await svc.facets(**filters)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(
{
"total": f.total,
"platforms": f.platforms,
"untagged": f.untagged,
"no_artist": f.no_artist,
"date_min": f.date_min.isoformat() if f.date_min else None,
"date_max": f.date_max.isoformat() if f.date_max else None,
}
)
@gallery_bp.route("/jump", methods=["GET"])
async def jump():
try:
year = int(request.args["year"])
month = int(request.args["month"])
filters, sort = _parse_filters()
except (KeyError, ValueError):
return jsonify({"error": "year and month query params required"}), 400
tag_id_raw = request.args.get("tag_id")
tag_id = int(tag_id_raw) if tag_id_raw else None
post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id")
artist_id = int(artist_id_raw) if artist_id_raw else None
async with get_session() as session:
svc = GalleryService(session)
try:
cursor = await svc.jump_cursor(
year=year, month=month, tag_id=tag_id,
post_id=post_id, artist_id=artist_id,
year=year, month=month, sort=sort, **filters,
)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
+21 -15
View File
@@ -122,26 +122,32 @@ async def delete_source(source_id: int):
@sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
async def set_backfill(source_id: int):
"""Plan #544: arm a source for backfill mode for the next N download
runs. Body: `{"runs": int}` (1..10, default 3). Returns the updated
source dict. While backfill_runs_remaining > 0, downloads use
gallery-dl's full-walk config (skip: True + 30-min timeout) instead
of the catch-up default (skip: "exit:20" + 14.5-min timeout)."""
"""Plan #693/#697: start/stop a run-until-done backfill, or start a recovery.
Body: `{"action": "start" | "stop" | "recover"}` (default "start"). 'start'
walks the full post history in time-boxed chunks until it reaches the bottom
(then the source shows 'complete'); 'recover' is the same walk but bypasses
the Patreon seen-ledger to re-fetch dropped-and-deleted near-dups under the
current pHash threshold; 'stop' cancels either back to tick mode. Returns the
updated source dict (incl. backfill_state / backfill_chunks /
backfill_bypass_seen)."""
payload = await request.get_json(silent=True) or {}
runs = payload.get("runs", 3)
try:
runs = int(runs)
except (TypeError, ValueError):
return _bad("invalid_runs", detail="runs must be an integer")
action = payload.get("action", "start")
if action not in ("start", "stop", "recover"):
return _bad(
"invalid_action",
detail="action must be 'start', 'stop', or 'recover'",
)
async with get_session() as session:
try:
record = await SourceService(session).set_backfill_runs(
source_id, runs,
)
svc = SourceService(session)
if action == "start":
record = await svc.start_backfill(source_id)
elif action == "recover":
record = await svc.start_recovery(source_id)
else:
record = await svc.stop_backfill(source_id)
except LookupError:
return _bad("not_found", status=404)
except ValueError as exc:
return _bad("invalid_runs", detail=str(exc))
return jsonify(record.to_dict())
+42 -6
View File
@@ -194,15 +194,46 @@ async def remove_tag_from_image(image_id: int, tag_id: int):
return "", 204
@tags_bp.route("/tags/<int:tag_id>", methods=["GET"])
async def get_tag(tag_id: int):
"""Resolve a single tag (used by the gallery to label its active
tag-filter chip)."""
async with get_session() as session:
tag = await session.get(Tag, tag_id)
if tag is None:
return jsonify({"error": "tag not found"}), 404
return jsonify(
{
"id": tag.id,
"name": tag.name,
"kind": tag.kind.value,
"fandom_id": tag.fandom_id,
}
)
@tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"])
async def rename_tag(tag_id: int):
body = await request.get_json()
if not body or "name" not in body:
return jsonify({"error": "name required"}), 400
async def update_tag(tag_id: int):
"""Rename and/or re-fandom a tag. Body may carry `name` and/or
`fandom_id` (a fandom tag id, or null to clear — character tags only).
`merge: true` resolves a collision by merging into the existing tag.
"""
body = await request.get_json() or {}
has_name = "name" in body
has_fandom = "fandom_id" in body
if not has_name and not has_fandom:
return jsonify({"error": "name or fandom_id required"}), 400
do_merge = bool(body.get("merge"))
async with get_session() as session:
svc = TagService(session)
try:
tag = await svc.rename(tag_id, body["name"])
tag = None
if has_name:
tag = await svc.rename(tag_id, body["name"])
if has_fandom:
tag = await svc.set_fandom(
tag_id, body["fandom_id"], merge=do_merge
)
except TagMergeConflict as exc:
return jsonify(
{
@@ -219,7 +250,12 @@ async def rename_tag(tag_id: int):
return jsonify({"error": str(exc)}), 400
await session.commit()
return jsonify(
{"id": tag.id, "name": tag.name, "kind": tag.kind.value}
{
"id": tag.id,
"name": tag.name,
"kind": tag.kind.value,
"fandom_id": tag.fandom_id,
}
)
+4
View File
@@ -97,6 +97,10 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.prune_task_runs",
"schedule": 86400.0, # daily
},
"vacuum-analyze": {
"task": "backend.app.tasks.maintenance.vacuum_analyze",
"schedule": 604800.0, # weekly — reclaim dead-tuple bloat + refresh stats
},
"fc3h-backup-db-nightly": {
"task": "backend.app.tasks.backup.backup_db_nightly",
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
+4
View File
@@ -2,6 +2,7 @@
from .app_setting import AppSetting
from .artist import Artist
from .artist_visit import ArtistVisit
from .backup_run import BackupRun
from .base import Base
from .credential import Credential
@@ -13,6 +14,7 @@ from .import_settings import ImportSettings
from .import_task import ImportTask
from .library_audit_run import LibraryAuditRun
from .ml_settings import MLSettings
from .patreon_seen_media import PatreonSeenMedia
from .post import Post
from .post_attachment import PostAttachment
from .series_page import SeriesPage
@@ -28,9 +30,11 @@ __all__ = [
"Base",
"AppSetting",
"Artist",
"ArtistVisit",
"BackupRun",
"Source",
"Credential",
"PatreonSeenMedia",
"Post",
"PostAttachment",
"SeriesPage",
+36
View File
@@ -0,0 +1,36 @@
"""ArtistVisit — per-artist 'last viewed' timestamp.
Powers the "+N new since last visit" badge on the artists directory and
the matching banner on `ArtistView`. One row per artist, single global
operator. When the multi-user model lands, the PK widens to
`(user_id, artist_id)` — currently aspirational only (no User model,
no services/access.py); operator approved skipping `user_id` for now
under rule #22 (breaking changes welcome).
Seed at migration time: every existing artist gets `last_viewed_at = NOW()`
so the badge starts at 0 across the board (no noisy "5000 unseen" on
first deploy). New artists also auto-get a row via
`ArtistService.find_or_create`.
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class ArtistVisit(Base):
__tablename__ = "artist_visit"
artist_id: Mapped[int] = mapped_column(
Integer,
ForeignKey("artist.id", ondelete="CASCADE"),
primary_key=True,
)
last_viewed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
+11
View File
@@ -74,6 +74,17 @@ class ImageRecord(Base):
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
# Denormalized gallery sort key = COALESCE(primary post's post_date,
# created_at) (alembic 0035). The gallery used to compute this as a
# COALESCE across the Post outer join on every /scroll, which can't use
# an index and re-sorted a large slice of the library per page (×10 with
# the old serial batching). Materializing it lets the cursor scroll read
# ix_image_record_effective_date directly. Maintained by the importer
# (services/importer.py _apply_sidecar) when a primary post with a date
# is linked; plain inserts keep the created_at-equivalent server default.
effective_date: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
+38
View File
@@ -0,0 +1,38 @@
"""PatreonSeenMedia — per-source ledger of Patreon media already
downloaded+processed.
Replaces gallery-dl's archive.sqlite3 with our own queryable table so
routine walks can skip media we've already ingested (and a future
"recovery" mode can deliberately bypass the ledger to re-walk).
`filehash` is normally a Patreon CDN MD5 (32 hex chars), but videos —
which have no stable content hash at discovery time — use a sentinel of
the form ``video:<post_id>:<media_id>``, hence String(128) rather than 32.
"""
from datetime import datetime
from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from sqlalchemy.types import DateTime
from .base import Base
class PatreonSeenMedia(Base):
__tablename__ = "patreon_seen_media"
__table_args__ = (
# Dedup key the downloader upserts against: one ledger row per
# (source, media). A second sighting of the same media is a no-op.
UniqueConstraint("source_id", "filehash", name="uq_patreon_seen_media_source_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
)
filehash: Mapped[str] = mapped_column(String(128), nullable=False)
post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
seen_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
@@ -13,10 +13,10 @@ from __future__ import annotations
import base64
from dataclasses import dataclass
from sqlalchemy import and_, exists, func, or_, select
from sqlalchemy import and_, case, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ImageRecord, Source
from ..models import Artist, ArtistVisit, ImageRecord, Source
from .gallery_service import thumbnail_url
_SEP = "|"
@@ -58,9 +58,27 @@ class ArtistDirectoryService:
raise ValueError("limit must be between 1 and 200")
count_col = func.count(ImageRecord.id).label("image_count")
# Unseen = images imported since the artist's last_viewed_at.
# NULL last_viewed_at (artist created before alembic 0034 seed
# or before find_or_create autoseed) defensively counts as
# "never visited" → all images unseen. Single grouped query, no
# N+1.
unseen_col = func.count(
case(
(
or_(
ArtistVisit.last_viewed_at.is_(None),
ImageRecord.created_at > ArtistVisit.last_viewed_at,
),
ImageRecord.id,
),
else_=None,
)
).label("unseen_count")
stmt = (
select(Artist, count_col)
select(Artist, count_col, unseen_col)
.outerjoin(ImageRecord, ImageRecord.artist_id == Artist.id)
.outerjoin(ArtistVisit, ArtistVisit.artist_id == Artist.id)
.group_by(Artist.id)
)
if q:
@@ -94,7 +112,7 @@ class ArtistDirectoryService:
next_cursor = _encode(last_artist.name, last_artist.id)
rows = rows[:limit]
artist_ids = [a.id for a, _ in rows]
artist_ids = [a.id for a, _, _ in rows]
previews = await self._previews(artist_ids)
cards = [
@@ -104,9 +122,10 @@ class ArtistDirectoryService:
"slug": artist.slug,
"is_subscription": bool(artist.is_subscription),
"image_count": int(image_count),
"unseen_count": int(unseen_count),
"preview_thumbnails": previews.get(artist.id, []),
}
for artist, image_count in rows
for artist, image_count, unseen_count in rows
]
return DirectoryPage(cards=cards, next_cursor=next_cursor)
+49
View File
@@ -9,11 +9,13 @@ Dates come from Post.post_date via ImageProvenance.post_id.
from dataclasses import dataclass
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import (
Artist,
ArtistVisit,
ImageProvenance,
ImageRecord,
Post,
@@ -122,6 +124,12 @@ class ArtistService:
)
).scalar_one()
# Mark this artist as "visited now"; the returned count is what
# the operator should see in the banner ("N new since last
# visit"). Done LAST so the read aggregates above all see the
# pre-visit state (cosmetic — none depend on visit data).
unseen_at_visit = await self._mark_visited_returning_unseen(aid)
return {
"id": artist.id,
"name": artist.name,
@@ -129,6 +137,7 @@ class ArtistService:
"is_subscription": bool(artist.is_subscription),
"image_count": int(image_count),
"post_count": int(post_count),
"unseen_count_at_visit": unseen_at_visit,
"date_range": {
"min": dmin.isoformat() if dmin else None,
"max": dmax.isoformat() if dmax else None,
@@ -157,6 +166,39 @@ class ArtistService:
],
}
async def _mark_visited_returning_unseen(self, artist_id: int) -> int:
"""Read pre-visit `last_viewed_at`, count images added since,
then upsert `last_viewed_at = NOW()`. Returns the count BEFORE
the upsert so the banner has data to render.
Postgres UPSERT (`ON CONFLICT DO UPDATE`) keeps the write
atomic — no SELECT-then-INSERT race per
`reference_scalar_one_or_none_duplicates`.
"""
prev = (
await self.session.execute(
select(ArtistVisit.last_viewed_at).where(
ArtistVisit.artist_id == artist_id
)
)
).scalar_one_or_none()
count_stmt = select(func.count(ImageRecord.id)).where(
ImageRecord.artist_id == artist_id
)
if prev is not None:
count_stmt = count_stmt.where(ImageRecord.created_at > prev)
unseen = (await self.session.execute(count_stmt)).scalar_one()
upsert = pg_insert(ArtistVisit.__table__).values(artist_id=artist_id)
upsert = upsert.on_conflict_do_update(
index_elements=["artist_id"],
set_={"last_viewed_at": func.now()},
)
await self.session.execute(upsert)
await self.session.commit()
return int(unseen)
async def images(
self, slug: str, cursor: str | None, limit: int = 60
) -> ArtistImagesPage | None:
@@ -230,6 +272,13 @@ class ArtistService:
artist = Artist(name=cleaned, slug=slug)
self.session.add(artist)
await self.session.flush()
# New artist starts "caught up" — seed ArtistVisit so the
# directory's `+N new` badge stays at 0 until real new
# content arrives. Without this, the unseen-count query
# treats NULL last_viewed_at as "never visited" and would
# count every image imported in the same session.
self.session.add(ArtistVisit(artist_id=artist.id))
await self.session.flush()
await sp.commit()
except IntegrityError:
await sp.rollback()
+56
View File
@@ -455,6 +455,62 @@ def purge_legacy_tags(session: Session, *, dry_run: bool = False) -> dict:
return result
# The Camie-suggestable CONTENT vocabulary. "Reset content tagging" wipes
# these so the operator can re-tag from scratch via auto-suggest. fandom +
# series (and series_page ordering) are deliberately NOT here — they're kept.
RESETTABLE_TAG_KINDS = ("general", "character")
def reset_content_tagging(session: Session, *, dry_run: bool = False) -> dict:
"""Count (dry_run) or DELETE every general + character tag so the operator
can re-tag from scratch via the Camie auto-suggest.
PRESERVED: fandom + series tags and their series_page ordering, plus every
image's image_record.tagger_predictions (untouched) so suggestions
repopulate immediately. CASCADE on image_tag / tag_alias / tag_allowlist /
tag_reference_embedding / tag_suggestion_rejection clears each deleted
tag's applications + metadata. Tag.fandom_id is SET NULL, so deleting
character tags never touches the fandom rows. Irreversible except via DB
backup restore.
Returns:
{"by_kind": {"general": N, "character": M},
"count": total tags,
"applications": image_tag rows that will be / were removed,
"sample_names": [first 50],
and on live runs "deleted": total}
"""
predicate = Tag.kind.in_(RESETTABLE_TAG_KINDS)
rows = session.execute(
select(Tag.id, Tag.name, Tag.kind).where(predicate)
).all()
by_kind: dict[str, int] = {}
for _id, _name, kind in rows:
key = kind.value if hasattr(kind, "value") else str(kind)
by_kind[key] = by_kind.get(key, 0) + 1
# Headline impact: applications (image_tag rows) that vanish via cascade.
applications = session.execute(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id.in_(select(Tag.id).where(predicate)))
).scalar_one()
sample = [name for _id, name, _kind in rows[:50]]
total = len(rows)
result = {
"by_kind": by_kind,
"count": total,
"applications": applications,
"sample_names": sample,
}
if dry_run:
return result
if total:
session.execute(Tag.__table__.delete().where(predicate))
session.commit()
result["deleted"] = total
return result
# ---------------------------------------------------------------------------
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
# ---------------------------------------------------------------------------
+71
View File
@@ -0,0 +1,71 @@
"""Platform → download-backend dispatch (one place that knows which platforms
are served by the native FC ingester vs. the gallery-dl subprocess).
gallery-dl wasn't built to be driven by an automated scheduler — no native
checkpoint/resume, no structured logs, per-file HEADs that dominate wall-clock.
The native ingester (services/patreon_ingester.py, plan #697) replaces it for
Patreon and is the path we grow as more platforms migrate. To keep that
migration DRY, every caller that has to behave differently per backend —
download routing, the credential-verify probe, cursor handling — asks THIS
module instead of testing ``platform == "patreon"`` inline. When a platform gets
a native ingester, it moves into ``NATIVE_INGESTER_PLATFORMS`` here and both the
download path and verify switch over together.
The backend surfaces share a UNIFORM signature so a caller invokes the same
function regardless of platform:
- verify_credential(...) → (ok: bool|None, message: str)
- (download stays in download_service for now; uses_native_ingester() is the
shared predicate it routes on, so the decision lives here too.)
"""
from __future__ import annotations
from pathlib import Path
# Platforms whose download + verify go through the native ingester rather than
# gallery-dl. gallery-dl still serves every other platform (subscribestar,
# hentaifoundry, discord, pixiv, deviantart) unchanged.
NATIVE_INGESTER_PLATFORMS = frozenset({"patreon"})
def uses_native_ingester(platform: str) -> bool:
"""True when `platform` is served by the native ingester (not gallery-dl).
The single predicate the download path and verify both route on."""
return platform in NATIVE_INGESTER_PLATFORMS
async def verify_source_credential(
*,
platform: str,
url: str,
artist_slug: str,
config_overrides: dict | None,
cookies_path: str | None,
auth_token: str | None,
images_root: Path,
) -> tuple[bool | None, str]:
"""Uniform credential probe across backends. Returns `(ok, message)`:
True = authenticated, False = rejected, None = inconclusive (drift /
network / nothing to test). Callers don't branch on platform — they call
this and render the result.
"""
if uses_native_ingester(platform):
# Native ingester platforms verify via their own lightweight auth probe
# (resolve campaign id + one authenticated API page). Patreon today.
from .patreon_ingester import verify_patreon_credential
return await verify_patreon_credential(url, cookies_path, config_overrides)
# gallery-dl platforms: --simulate one item; the extractor errors before it
# can list if auth is bad.
from .gallery_dl import GalleryDLService, SourceConfig
gdl = GalleryDLService(images_root=images_root)
return await gdl.verify(
url=url,
artist_slug=artist_slug,
platform=platform,
source_config=SourceConfig.from_dict(config_overrides or {}),
cookies_path=cookies_path,
auth_token=auth_token,
)
+247 -98
View File
@@ -13,7 +13,6 @@ from __future__ import annotations
import asyncio
import logging
import re
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -25,44 +24,26 @@ from sqlalchemy.orm import joinedload
from ..models import Artist, DownloadEvent, Source
from .credential_service import CredentialService
from .download_backends import uses_native_ingester
from .gallery_dl import (
BACKFILL_CHUNK_SECONDS,
BACKFILL_SKIP_VALUE,
BACKFILL_TIMEOUT_SECONDS,
TICK_SKIP_VALUE,
DownloadResult,
ErrorType,
GalleryDLService,
SourceConfig,
parse_last_cursor,
)
from .importer import Importer
from .patreon_resolver import resolve_campaign_id
from .patreon_ingester import PatreonIngester
from .patreon_resolver import resolve_campaign_id_for_source
from .platforms import auth_type_for
from .scheduler_service import set_platform_cooldown
log = logging.getLogger(__name__)
_PATREON_VANITY_RE = re.compile(
r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
re.IGNORECASE,
)
_CAMPAIGN_ID_FAILURE_PATTERN = "failed to extract campaign id"
def _extract_patreon_vanity(url: str) -> str | None:
m = _PATREON_VANITY_RE.match(url)
return m.group(1) if m else None
def _looks_like_campaign_id_failure(stdout: str, stderr: str) -> bool:
return _CAMPAIGN_ID_FAILURE_PATTERN in f"{stdout}\n{stderr}".lower()
def _effective_url(platform: str, source_url: str, overrides: dict) -> str:
if platform == "patreon" and overrides.get("patreon_campaign_id"):
return f"https://www.patreon.com/id:{overrides['patreon_campaign_id']}"
return source_url
class DownloadService:
"""Async orchestrator. The Celery task runs `asyncio.run(svc.download_source(N))`.
@@ -77,12 +58,18 @@ class DownloadService:
gdl: GalleryDLService,
importer: Importer,
cred_service: CredentialService,
sync_session_factory=None,
):
self.async_session = async_session
self.sync_session = sync_session
self.gdl = gdl
self.importer = importer
self.cred_service = cred_service
# Sync sessionmaker the native Patreon ingester opens SHORT-LIVED
# sessions from for its seen-ledger reads/writes (never one held across
# the multi-minute walk — see PatreonIngester). Only the patreon branch
# of phase 2 uses it; gallery-dl sources leave it None.
self.sync_session_factory = sync_session_factory
async def download_source(self, source_id: int) -> int:
"""Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
@@ -91,64 +78,171 @@ class DownloadService:
return setup["event_id"]
ctx = setup
# Release the phase-1 DB connections before the (up to ~19.5-min in
# backfill) gallery-dl subprocess. Held checked-out across that idle
# window, the asyncpg/psycopg connections get reaped by the server,
# and phase 3's first query then hits a dead socket
# (asyncpg ConnectionDoesNotExistError) → download_source autoretry →
# _phase1_setup's in-flight guard no-ops the retry → the event
# strands empty for the recovery sweep (Anduo #40014, 2026-06-04).
# pool_pre_ping can't help a *held* connection — it only validates on
# pool checkout. Closing returns them to the pool so phase 3 re-
# acquires a live one (the async task engine uses NullPool, the sync
# engine pre_ping + pool_recycle=300). This is what makes the
# "Phase 2 — no DB connection" contract in the class docstring true.
await self.async_session.close()
self.sync_session.close()
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
# alembic 0031 / plan #544: derive skip_value + timeout from the
# source's backfill_runs_remaining counter. When > 0, walk the full
# post history (skip: True + 1800s); when 0, exit gallery-dl after
# 20 contiguous archived items (skip: "exit:20" + the default
# 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
# Backfill mode (plan #693): the source's `_backfill_state == "running"`
# selects a time-boxed deep-walk chunk — skip: True (walk full history)
# + the BACKFILL_CHUNK_SECONDS budget, resuming from the cursor
# checkpoint (plan #689). State is the single source of truth; it stays
# "running" across ticks until the walk reaches the bottom (phase 3
# flips it to "complete"). Otherwise tick mode: exit gallery-dl after
# 20 contiguous archived items (skip: "exit:20" + the default 870s).
# Operator drives this via POST /api/sources/{id}/backfill {action}.
overrides = ctx["config_overrides"] or {}
in_backfill = overrides.get("_backfill_state") == "running"
# Recovery (plan #697) reuses the entire #693 backfill state machine —
# cursor checkpoint, time-boxed chunks, complete/stall lifecycle — and
# differs only in bypassing the tier-1 seen-ledger so dropped-and-deleted
# near-dups get re-fetched and re-evaluated under the CURRENT pHash
# threshold (tier-2 disk still spares files we kept). The
# `_backfill_bypass_seen` flag rides alongside the running backfill state;
# download mode is "recovery" when both are set.
bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
if in_backfill:
skip_value: bool | str = BACKFILL_SKIP_VALUE
source_config.timeout = BACKFILL_TIMEOUT_SECONDS
source_config.timeout = BACKFILL_CHUNK_SECONDS
pending_cursor = overrides.get("_backfill_cursor")
if uses_native_ingester(ctx["platform"]) and pending_cursor:
source_config.resume_cursor = pending_cursor
else:
skip_value = TICK_SKIP_VALUE
effective_url = _effective_url(
ctx["platform"], ctx["url"], ctx["config_overrides"] or {}
)
dl_result = await self.gdl.download(
url=effective_url,
artist_slug=ctx["artist_slug"],
platform=ctx["platform"],
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
resolved_campaign_id: str | None = None
if (
ctx["platform"] == "patreon"
and not dl_result.success
and "patreon_campaign_id" not in (ctx["config_overrides"] or {})
and _looks_like_campaign_id_failure(dl_result.stdout, dl_result.stderr)
):
vanity = _extract_patreon_vanity(ctx["url"])
if vanity:
log.info(
"Attempting campaign-ID resolution for %s (%s)",
ctx["artist_slug"], vanity,
if uses_native_ingester(ctx["platform"]):
# Native ingester (plan #697) fully replaces gallery-dl for Patreon
# in phase 2 — zero per-file HEADs, native cursor/resume, loud drift
# detection. Returns a DownloadResult-shaped object so phase 3 is
# untouched. The gallery-dl Patreon path + its campaign-id retry are
# removed at the step-5 cutover.
if in_backfill and bypass_seen:
mode = "recovery"
elif in_backfill:
mode = "backfill"
else:
mode = "tick"
dl_result, resolved_campaign_id = await self._run_patreon_ingester(
ctx, source_config, mode,
)
else:
dl_result = await self.gdl.download(
url=ctx["url"],
artist_slug=ctx["artist_slug"],
platform=ctx["platform"],
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
# A backfill chunk that hit its time-box but made forward progress is
# NORMAL, not a failure — reclassify TIMEOUT → PARTIAL so it reads as
# "ok/progress" (PARTIAL maps to status "ok"), not a red error, and the
# next chunk just resumes from the new cursor (plan #693). A chunk that
# timed out with NO progress stays TIMEOUT and feeds phase 3's
# stall-guard. Leave RATE_LIMITED alone so the platform-cooldown fires.
if in_backfill and dl_result.error_type == ErrorType.TIMEOUT:
new_cursor = parse_last_cursor(dl_result.stdout, dl_result.stderr)
advanced = bool(
(new_cursor and new_cursor != overrides.get("_backfill_cursor"))
or dl_result.files_downloaded > 0
)
if advanced:
dl_result.error_type = ErrorType.PARTIAL
dl_result.error_message = (
f"Backfill chunk: {dl_result.files_downloaded} file(s) — continuing"
)
resolved_campaign_id = await resolve_campaign_id(
vanity, ctx["cookies_path"]
)
if resolved_campaign_id:
dl_result = await self.gdl.download(
url=f"https://www.patreon.com/id:{resolved_campaign_id}",
artist_slug=ctx["artist_slug"],
platform=ctx["platform"],
source_config=source_config,
cookies_path=ctx["cookies_path"],
auth_token=ctx["auth_token"],
skip_value=skip_value,
)
return await self._phase3_persist(
ctx["event_id"], ctx, dl_result, resolved_campaign_id,
)
async def _run_patreon_ingester(
self, ctx: dict, source_config, mode: str,
) -> tuple[DownloadResult, str | None]:
"""Phase-2 Patreon branch: resolve the campaign id, then run the native
ingester in a worker thread (it is sync requests/subprocess).
Returns (DownloadResult, resolved_campaign_id). `resolved_campaign_id` is
non-None only when we had to look it up from the vanity URL this run, so
phase 3 caches it on the source the same way the old gallery-dl retry did.
A campaign id we cannot resolve is a loud failure (NOT_FOUND) — never a
silent empty success.
"""
overrides = ctx["config_overrides"] or {}
# Shared resolution path (override / id: URL / vanity lookup) — the same
# helper the credential-verify probe uses. resolved_campaign_id is
# non-None only when a vanity lookup ran, so phase 3 caches it.
campaign_id, resolved_campaign_id = await resolve_campaign_id_for_source(
ctx["url"], ctx["cookies_path"], overrides
)
if not campaign_id:
return (
DownloadResult(
success=False,
url=ctx["url"],
artist_slug=ctx["artist_slug"],
platform="patreon",
error_type=ErrorType.NOT_FOUND,
error_message=(
"Could not resolve Patreon campaign id from the source "
"URL (vanity lookup failed — cookies expired or creator "
"moved?)"
),
),
None,
)
# Honor the operator's existing rate-limit knobs on the native path
# (plan #703): the global download_rate_limit_seconds (gallery-dl's
# `rate_limit`, here on self.gdl) paces media downloads. Page fetches use
# the per-source sleep_request override, else `max(0.5, rate_limit/4)` —
# the SAME API-pacing default gallery-dl applied as its `sleep-request`
# (gallery_dl._get_default_config), so the rate-limited /api/posts
# endpoint stays politely paced without over-throttling.
rate_limit = self.gdl._rate_limit
request_sleep = (
source_config.sleep_request
if source_config.sleep_request is not None
else max(0.5, rate_limit / 4)
)
ingester = PatreonIngester(
images_root=self.gdl.images_root,
cookies_path=ctx["cookies_path"],
session_factory=self.sync_session_factory,
validate=self.gdl._validate_files,
rate_limit=rate_limit,
request_sleep=request_sleep,
)
loop = asyncio.get_running_loop()
dl_result = await loop.run_in_executor(
None,
lambda: ingester.run(
source_id=ctx["source_id"],
campaign_id=campaign_id,
artist_slug=ctx["artist_slug"],
url=ctx["url"],
mode=mode,
resume_cursor=source_config.resume_cursor,
time_budget_seconds=source_config.timeout,
),
)
return dl_result, resolved_campaign_id
async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
source = (await self.async_session.execute(
select(Source).options(joinedload(Source.artist)).where(Source.id == source_id)
@@ -380,36 +474,91 @@ class DownloadService:
source_id=ctx["source_id"], status=status, error_message=ev.error,
error_type=dl_result.error_type.value if dl_result.error_type else None,
)
# Plan #544: backfill lifecycle — auto-complete when a clean
# backfill run drained the queue (gallery-dl exited 0 + zero files
# downloaded means there was nothing to fetch); otherwise decrement
# the counter. Next tick falls back to tick mode once it hits 0.
#
# Audit 2026-06-02 gating: VALIDATION_FAILED also exits the
# subprocess with return_code=0 and files_downloaded=0 (every
# file was quarantined), which used to match the auto-complete
# predicate exactly — zeroing the operator's armed budget on
# the FIRST quarantine run instead of decrementing. Require
# dl_result.success + no error_type so only genuinely-empty
# successful runs drain the counter.
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
if backfill_remaining > 0:
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
queue_drained = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
and dl_result.files_downloaded == 0
)
if queue_drained:
src.backfill_runs_remaining = 0
else:
src.backfill_runs_remaining = max(0, backfill_remaining - 1)
await self._apply_backfill_lifecycle(ctx, dl_result)
await self.async_session.commit()
return event_id
async def _apply_backfill_lifecycle(self, ctx: dict, dl_result) -> None:
"""Backfill state machine (plan #693, building on the cursor of #689).
A backfill runs in time-boxed chunks while
`config_overrides["_backfill_state"] == "running"`. Each chunk:
- COMPLETES the walk → clean rc=0 (gallery-dl with skip:True exits 0
only after exhausting the newest→oldest walk; a chunk cut short by
its time-box returns success=False / rc<0 via TimeoutExpired). On
completion: state="complete", clear the cursor, return to tick mode.
- made PROGRESS (cursor advanced and/or files written) → stay
"running", checkpoint the new cursor, bump the chunk counter, and
spend one of the safety-cap chunks (backfill_runs_remaining). If the
cap is exhausted without finishing → state="stalled".
- made NO progress → increment the stall counter; two strikes →
state="stalled", clear the cursor (a wedged walk can't loop).
State is the single source of truth; the cursor lives only inside a
running backfill. config_overrides is reassigned (not mutated in place)
for SQLAlchemy JSON change detection.
"""
overrides = ctx["config_overrides"] or {}
if overrides.get("_backfill_state") != "running":
return # not backfilling — tick mode, nothing to do
old_cursor = overrides.get("_backfill_cursor")
cap_remaining = ctx.get("backfill_runs_remaining", 0) or 0
src = (await self.async_session.execute(
select(Source).where(Source.id == ctx["source_id"])
)).scalar_one()
new_overrides = dict(src.config_overrides or {})
chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
new_overrides["_backfill_chunks"] = chunks
completed = (
dl_result.success
and dl_result.error_type is None
and dl_result.return_code == 0
)
if completed:
new_overrides["_backfill_state"] = "complete"
new_overrides.pop("_backfill_cursor", None)
new_overrides.pop("_backfill_cursor_stalls", None)
# plan #697: a recovery walk shares this lifecycle; clear its bypass
# flag on completion so the next routine tick honors the seen-ledger.
new_overrides.pop("_backfill_bypass_seen", None)
src.config_overrides = new_overrides
src.backfill_runs_remaining = 0
return
# Did not finish. Patreon checkpoints + resumes via cursor; other
# platforms have no resumable cursor (every chunk re-walks from the
# top), so they advance only by the download archive growing.
new_cursor = (
parse_last_cursor(dl_result.stdout, dl_result.stderr)
if uses_native_ingester(ctx["platform"]) else None
)
advanced = bool(
(new_cursor and new_cursor != old_cursor)
or dl_result.files_downloaded > 0
)
if advanced:
if new_cursor:
new_overrides["_backfill_cursor"] = new_cursor
new_overrides.pop("_backfill_cursor_stalls", None)
cap_remaining = max(0, cap_remaining - 1)
src.backfill_runs_remaining = cap_remaining
if cap_remaining == 0:
# Safety cap hit before reaching the bottom — pause, don't loop.
new_overrides["_backfill_state"] = "stalled"
else:
stalls = int(new_overrides.get("_backfill_cursor_stalls", 0)) + 1
if stalls >= 2:
new_overrides["_backfill_state"] = "stalled"
new_overrides.pop("_backfill_cursor", None)
new_overrides.pop("_backfill_cursor_stalls", None)
else:
new_overrides["_backfill_cursor_stalls"] = stalls
src.config_overrides = new_overrides
async def _update_source_health(
self, *, source_id: int, status: str, error_message: str | None,
error_type: str | None = None,
+9 -8
View File
@@ -16,7 +16,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
from ..utils.slug import slugify
from .source_service import NEW_SOURCE_BACKFILL_RUNS
from .source_service import BACKFILL_MAX_CHUNKS
class UnknownPlatformError(Exception):
@@ -205,16 +205,17 @@ class ExtensionService:
return existing, False
sp = await self.session.begin_nested()
try:
# New subscription sources arm a few backfill runs so the
# first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built).
# Mirrors SourceService.create — without it, Firefox quick-
# add on a creator with >20 unsynced posts would surface
# as "check failed" with no diagnosis. Audit 2026-06-02.
# New subscription sources arm run-until-done backfill (plan #693)
# so the first ticks walk the full history (otherwise gallery-dl's
# exit:20 short-circuits before the archive is built). Mirrors
# SourceService.create — without it, Firefox quick-add on a creator
# with >20 unsynced posts would surface as "check failed" with no
# diagnosis. Audit 2026-06-02.
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
backfill_runs_remaining=NEW_SOURCE_BACKFILL_RUNS,
config_overrides={"_backfill_state": "running"},
backfill_runs_remaining=BACKFILL_MAX_CHUNKS,
)
self.session.add(src)
await self.session.flush()
+82 -62
View File
@@ -13,6 +13,7 @@ from __future__ import annotations
import asyncio
import json
import logging
import re
import subprocess
import sys
import tempfile
@@ -39,6 +40,12 @@ class ErrorType(StrEnum):
HTTP_ERROR = "http_error"
UNSUPPORTED_URL = "unsupported_url"
VALIDATION_FAILED = "validation_failed"
# Native Patreon ingester only (plan #697): a response parsed as JSON but
# didn't match the JSON:API shape the ingester depends on (missing `data`,
# media lacking `file_name`/`url`, etc.). Distinct from AUTH_ERROR — the fix
# is updating the ingester's field-set/parser, not rotating credentials. The
# contract test guards against silently shipping this.
API_DRIFT = "api_drift"
# Run made real progress (downloaded ≥1 file) but did not finish in the
# subprocess budget. Distinct from UNKNOWN_ERROR — the downstream status
# mapping classifies this as "ok" because the next tick continues.
@@ -55,33 +62,36 @@ class ErrorType(StrEnum):
# 20 contiguous HEADs is still negligible.
TICK_SKIP_VALUE = "exit:20"
# Backfill mode (operator-triggered deep scan): walk the full history.
# Source.backfill_runs_remaining > 0 selects this mode; the longer
# timeout below absorbs creators with thousands of posts.
# Backfill mode (operator-triggered deep scan): walk the full history,
# one TIME-BOXED CHUNK per run (plan #693). config_overrides["_backfill_state"]
# == "running" selects this mode; the cursor checkpoint (plan #689) lets each
# chunk resume where the last stopped, so the walk advances across chunks
# until gallery-dl exits cleanly (= reached the bottom).
#
# 30 seconds shy of Celery's hard `time_limit=1200` on download_source
# (tasks/download.py:33). subprocess.run MUST raise TimeoutExpired
# before Celery SIGKILLs the worker — same rationale as the tick
# default at line 74. The audit (2026-06-02) caught this at 1800,
# guaranteeing SIGKILL on any backfill that ran to its subprocess
# budget: stdout/stderr lost, backfill_runs_remaining never
# decrements, recovery sweep stamps generic "stranded" 30 min later.
# Recreates the exact Knuxy #38275 failure mode the tick 870s default
# was added to prevent. backfill_runs_remaining=3 still gives ~58
# minutes of cumulative walk across three runs for prolific creators.
# The chunk budget is deliberately FAR below download_source's Celery
# soft_time_limit (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py), not
# "just under" it. Hitting this budget is the NORMAL chunk boundary, not a
# failure: subprocess.run raises TimeoutExpired (which captures partial
# stdout/stderr + the last emitted cursor), and download_service reclassifies
# a chunk that made progress as PARTIAL (status "ok"), not an error. The huge
# headroom means a stuck file or a slow chunk can never let Celery's
# SoftTimeLimitExceeded preempt TimeoutExpired (the failure mode behind Knuxy
# #38275 / Anduo #39912/#40411). Earlier we ran one ~1170s run-to-the-wall
# per arming; that died as a timeout error every time on large catalogs.
BACKFILL_SKIP_VALUE = True
BACKFILL_TIMEOUT_SECONDS = 1170
BACKFILL_CHUNK_SECONDS = 600
# 30 seconds shy of download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). subprocess.run MUST raise TimeoutExpired before
# Celery raises SoftTimeLimitExceeded — otherwise Celery wins the race,
# SIGKILLs the worker, in-memory stdout/stderr is lost, and the
# DownloadEvent ends up empty-logged with "stranded by recovery sweep"
# 18 minutes later (operator-flagged 2026-05-31, Knuxy event #38275).
# The 30s buffer absorbs scheduler jitter / GC pauses without making
# legitimately-long-running syncs timeout-friendlier. Per-source bumps
# still live in source.config_overrides for legitimately long syncs.
# Sits well below download_source's Celery soft_time_limit
# (DOWNLOAD_SOFT_TIME_LIMIT=1350, tasks/download.py). subprocess.run MUST
# raise TimeoutExpired before Celery raises SoftTimeLimitExceeded —
# otherwise Celery wins the race, SIGKILLs the worker, in-memory
# stdout/stderr is lost, and the DownloadEvent ends up empty-logged with
# "stranded by recovery sweep" (operator-flagged 2026-05-31, Knuxy event
# #38275; recurred in backfill mode as Anduo #39912). Per-source bumps
# still live in source.config_overrides for legitimately long syncs —
# keep any override below the soft limit, or the soft-limit salvage path
# in tasks/download.py (_finalize_soft_limited) is the only safety net.
_DEFAULT_GDL_TIMEOUT_SECONDS = 870
@@ -94,6 +104,17 @@ class SourceConfig:
Source.backfill_runs_remaining column at the download_service layer
and is passed to _build_config_for_source as `skip_value`. Same for
the per-run subprocess timeout.
`resume_cursor` is RUNTIME state, not persisted operator config: the
cursor-paged backfill checkpoint (see parse_last_cursor + the
download_service backfill lifecycle). It is consumed only by the native
Patreon ingester (the sole cursor-paged platform; gallery-dl's Patreon path
was removed at the #697 cutover): on a backfill/recovery run the ingester
resumes its newest→oldest walk from that pagination cursor instead of
restarting from the top. download_service threads it from
config_overrides["_backfill_cursor"]; SourceConfig deliberately does NOT read
it in from_dict (config_overrides also holds operator config, and
resume_cursor must only apply in backfill/recovery mode).
"""
content_types: list[str] = field(default_factory=lambda: ["all"])
sleep: float | None = None
@@ -102,6 +123,7 @@ class SourceConfig:
filename_pattern: str | None = None
save_metadata: bool = True
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
resume_cursor: str | None = None
@classmethod
def from_dict(cls, data: dict) -> SourceConfig:
@@ -150,6 +172,22 @@ def _summarize_validation_failures(failures: list[dict]) -> str:
return f"{n} files quarantined ({top_count}× {top_reason}, mixed)"
# The native Patreon ingester emits its pagination cursor as `Cursor: <token>`
# — one line per fetched page — into the DownloadResult stdout (mirroring the
# convention gallery-dl used before the #697 cutover, so the backfill lifecycle
# stayed unchanged). The LAST such line is the furthest-progressed page, i.e.
# the resume point. We scan both streams (be defensive) and take the final
# match; download_service checkpoints it as the next chunk's resume_cursor.
# Survives a time-boxed chunk too: the ingester emits the cursor for a page when
# it STARTS it, so an interrupted walk still yields its last cursor.
_CURSOR_RE = re.compile(r"Cursor:\s*(\S+)")
def parse_last_cursor(stdout: str, stderr: str) -> str | None:
matches = _CURSOR_RE.findall(f"{stdout or ''}\n{stderr or ''}")
return matches[-1] if matches else None
class GalleryDLService:
"""Service for executing gallery-dl downloads."""
@@ -183,16 +221,10 @@ class GalleryDLService:
"permission denied", "tier required", "pledge required",
]
# Per-platform defaults. Lifted from GS — same six platforms FC supports.
# Per-platform defaults for the gallery-dl-backed platforms. Patreon was
# removed at the plan-#697 cutover — it now uses the native ingester
# (services/patreon_ingester.py), not gallery-dl.
PLATFORM_DEFAULTS = {
"patreon": {
"content_types": ["images", "image_large", "attachments", "postfile", "content"],
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
"filename": "{num:>02}_{filename}.{extension}",
"videos": True,
"embeds": True,
"cursor": True,
},
"subscribestar": {
"content_types": ["all"],
"directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
@@ -251,7 +283,12 @@ class GalleryDLService:
"skip": True,
"sleep": self._rate_limit,
"sleep-request": max(0.5, self._rate_limit / 4),
"retries": 3,
# 2 (not 3) retries — a stuck CDN host shouldn't burn a whole
# backfill chunk on one file. Anduo #40838 spent ~600s on a
# single image (4 extractor HEAD retries × 30s + 4 downloader
# GET retries × 120s); halving retries + the downloader
# timeout below caps a wedged file at ~1-2 min instead.
"retries": 2,
"timeout": 30.0,
"verify": True,
"postprocessors": [
@@ -266,27 +303,17 @@ class GalleryDLService:
"downloader": {
"part": True,
"part-directory": str(self._config_dir / "temp"),
"retries": 3,
"timeout": 120.0,
# Forward Patreon as Referer/Origin to yt-dlp when it
# fetches video manifests. Operator-flagged 2026-06-01
# (DaferQ patreon): video posts hosted on Mux carry a JWT
# playback restriction that checks Referer/Origin on every
# request — not just the token signature. gallery-dl's
# HEAD probe to stream.mux.com returns 200 (the token is
# valid), but yt-dlp's actual GET-with-Range to fetch the
# m3u8 manifest 403s because yt-dlp sends its own default
# Referer, which Mux's policy rejects. Forcing the right
# headers fixes the headers-only case; Mux IP-range
# restrictions are unfixable from here.
"ytdl": {
"raw-options": {
"http_headers": {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
},
},
},
# See the extractor retries note above (Anduo #40838): 2
# retries + a 60s read-timeout (was 120) so a stalled
# connection fails fast. 60s is a per-read timeout, not a
# transfer cap — large GIFs that keep streaming are unaffected.
"retries": 2,
"timeout": 60.0,
# NOTE: the Patreon/Mux yt-dlp Referer/Origin forwarding lived
# here until the plan-#697 cutover. It was Patreon-specific (and
# would have wrongly tagged the other platforms' yt-dlp fetches);
# Patreon video is now handled by the native ingester's
# downloader (patreon_downloader._VIDEO_HEADERS), so it's gone.
},
"output": {"progress": True},
}
@@ -338,14 +365,7 @@ class GalleryDLService:
platform_section = config["extractor"].setdefault(platform, {})
if platform == "patreon":
if "all" in source_config.content_types:
platform_section["files"] = [
"images", "image_large", "attachments", "postfile", "content",
]
else:
platform_section["files"] = source_config.content_types
elif platform == "hentaifoundry":
if platform == "hentaifoundry":
if "pictures" in source_config.content_types or "all" in source_config.content_types:
platform_section["include"] = "all"
+332 -81
View File
@@ -18,15 +18,22 @@ import base64
from dataclasses import dataclass
from datetime import datetime
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy import Select, and_, distinct, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import aliased
from ..models import Artist, ImageProvenance, ImageRecord, Post, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
# Reserved `platform` filter value selecting images with NO platformed
# provenance (filesystem imports). Returned by facets() as a null-valued
# bucket; the frontend maps that null back to this sentinel in the URL so the
# bucket is selectable. Underscore-wrapped so it can't collide with a real
# gallery-dl platform name (patreon/pixiv/...).
UNSOURCED_PLATFORM = "__unsourced__"
def encode_cursor(effective_date: datetime, image_id: int) -> str:
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
@@ -43,16 +50,17 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
def _effective_date_col():
"""SQL expression: COALESCE(post.post_date, image_record.created_at).
"""The materialized gallery sort key: image_record.effective_date
(alembic 0035) = COALESCE(primary post's post_date, created_at),
maintained at write time by the importer.
Used as the canonical sort/group/filter key across the gallery so
images backfilled with primary_post_id (e.g. via tag_apply phase 4)
surface at their original publish date, not their FC import date.
Images without a Post (or with Post.post_date NULL) fall back to
image_record.created_at and still order coherently against
post-attached ones.
Canonical sort/group/filter key across the gallery so images attached
to a post surface at their original publish date, not their FC import
date — and, now that it's a single indexed column rather than a
COALESCE across the Post outer join, the cursor scroll is an index
range scan instead of a full re-sort per page.
"""
return func.coalesce(Post.post_date, ImageRecord.created_at)
return ImageRecord.effective_date
def _outer_join_primary_post(stmt: Select) -> Select:
@@ -91,6 +99,16 @@ class TimelineBucket:
count: int
@dataclass(frozen=True)
class GalleryFacets:
total: int # images matching the FULL active filter
platforms: list[dict] # [{"value": str|None, "count": int}], null = unsourced
untagged: int # how many the Untagged flag would isolate
no_artist: int # how many the No-artist flag would isolate
date_min: datetime | None
date_max: datetime | None
def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str:
"""Return the URL to fetch a thumbnail.
@@ -117,13 +135,85 @@ def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str
return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
def _require_single_filter(tag_id, post_id, artist_id) -> None:
if sum(x is not None for x in (tag_id, post_id, artist_id)) > 1:
def _require_single_filter(tag_ids, post_id, artist_id) -> None:
"""post_id is the post-detail view — it can't combine with the
composable filters. tag_ids + artist_id (+ media_type) compose freely
(AND)."""
if post_id is not None and (tag_ids or artist_id is not None):
raise ValueError(
"tag_id, post_id, artist_id are mutually exclusive"
"post_id cannot be combined with tag or artist filters"
)
def _apply_scope(
stmt, *, tag_ids, post_id, artist_id, media_type,
platform=None, untagged=False, no_artist=False,
date_from=None, date_to=None,
):
"""Apply the composable gallery filters to a statement.
All clauses are correlated EXISTS / scalar predicates on ImageRecord, so
they AND together without row-multiplication and don't require any join to
be present on `stmt` (the artist/platform paths alias Post/Source inside
their own EXISTS).
- tag_ids: image must carry ALL of them — one correlated EXISTS per tag.
- post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded
by _require_single_filter).
- media_type: 'image' | 'video' narrows by mime prefix.
- platform: EXISTS a provenance→source with that platform; the
UNSOURCED_PLATFORM sentinel inverts it (NO platformed provenance).
- untagged: NOT EXISTS any image_tag row.
- no_artist: ImageRecord.artist_id IS NULL.
- date_from / date_to: half-open [from, to) bounds on effective_date.
"""
for tid in tag_ids or []:
stmt = stmt.where(
exists().where(
image_tag.c.image_record_id == ImageRecord.id,
image_tag.c.tag_id == tid,
)
)
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
if media_type == "image":
stmt = stmt.where(ImageRecord.mime.like("image/%"))
elif media_type == "video":
stmt = stmt.where(ImageRecord.mime.like("video/%"))
if platform is not None:
stmt = stmt.where(_platform_clause(platform))
if untagged:
stmt = stmt.where(
~exists().where(image_tag.c.image_record_id == ImageRecord.id)
)
if no_artist:
stmt = stmt.where(ImageRecord.artist_id.is_(None))
eff = _effective_date_col()
if date_from is not None:
stmt = stmt.where(eff >= date_from)
if date_to is not None:
stmt = stmt.where(eff < date_to)
return stmt
def _platform_clause(platform):
"""Correlated EXISTS on a provenance row whose Source carries `platform`.
The UNSOURCED_PLATFORM sentinel inverts to NOT EXISTS(any sourced
provenance) — i.e. filesystem-imported content with no platform."""
src = aliased(Source)
if platform == UNSOURCED_PLATFORM:
return ~exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == src.id,
)
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == src.id,
src.platform == platform,
)
def _provenance_clause(post_id, artist_id):
"""Correlated EXISTS clause (NOT a join) so an image with multiple
matching provenance rows is returned exactly once and the
@@ -155,6 +245,27 @@ def _provenance_clause(post_id, artist_id):
return None
def _gallery_images(rows, artists: dict[int, dict]) -> list[GalleryImage]:
"""Build GalleryImage list from (record, posted_at, eff_date) rows + the
artist hydration map. Shared by scroll() and similar()."""
return [
GalleryImage(
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
artist=artists.get(record.id),
)
for record, posted_at, eff_date in rows
]
async def _artists_for(session, image_ids: list[int]) -> dict[int, dict]:
"""Map image_id -> {"name","slug"} via the canonical
image_record.artist_id (FC-2d-vii-c). Bounded by page size."""
@@ -179,35 +290,50 @@ class GalleryService:
self,
cursor: str | None,
limit: int = 50,
tag_id: int | None = None,
tag_ids: list[int] | None = None,
post_id: int | None = None,
artist_id: int | None = None,
media_type: str | None = None,
sort: str = "newest",
platform: str | None = None,
untagged: bool = False,
no_artist: bool = False,
date_from: datetime | None = None,
date_to: datetime | None = None,
) -> GalleryPage:
if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200")
_require_single_filter(tag_id, post_id, artist_id)
_require_single_filter(tag_ids, post_id, artist_id)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
)
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
descending = sort != "oldest"
if cursor:
cur_ts, cur_id = decode_cursor(cursor)
stmt = stmt.where(
or_(
eff < cur_ts,
and_(eff == cur_ts, ImageRecord.id < cur_id),
# The cursor is just (last eff, last id); the request's sort
# decides which side of it the next page lies on.
if descending:
stmt = stmt.where(
or_(eff < cur_ts, and_(eff == cur_ts, ImageRecord.id < cur_id))
)
else:
stmt = stmt.where(
or_(eff > cur_ts, and_(eff == cur_ts, ImageRecord.id > cur_id))
)
)
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
if descending:
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc())
else:
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
stmt = stmt.limit(limit + 1)
rows = (await self.session.execute(stmt)).all()
next_cursor = None
@@ -219,22 +345,7 @@ class GalleryService:
artists = await _artists_for(
self.session, [r[0].id for r in rows]
)
images = [
GalleryImage(
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
artist=artists.get(record.id),
)
for record, posted_at, eff_date in rows
]
images = _gallery_images(rows, artists)
return GalleryPage(
images=images,
next_cursor=next_cursor,
@@ -243,9 +354,15 @@ class GalleryService:
async def timeline(
self,
tag_id: int | None = None,
tag_ids: list[int] | None = None,
post_id: int | None = None,
artist_id: int | None = None,
media_type: str | None = None,
platform: str | None = None,
untagged: bool = False,
no_artist: bool = False,
date_from: datetime | None = None,
date_to: datetime | None = None,
) -> list[TimelineBucket]:
eff = _effective_date_col()
year_col = func.date_part("year", eff).label("yr")
@@ -254,25 +371,28 @@ class GalleryService:
year_col, month_col, func.count(ImageRecord.id).label("cnt")
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
)
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
_require_single_filter(tag_ids, post_id, artist_id)
stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
stmt = stmt.group_by(year_col, month_col).order_by(year_col.desc(), month_col.desc())
rows = (await self.session.execute(stmt)).all()
return [TimelineBucket(year=int(r.yr), month=int(r.mo), count=int(r.cnt)) for r in rows]
async def jump_cursor(
self, year: int, month: int, tag_id: int | None = None,
self, year: int, month: int, tag_ids: list[int] | None = None,
post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, sort: str = "newest",
platform: str | None = None, untagged: bool = False,
no_artist: bool = False, date_from: datetime | None = None,
date_to: datetime | None = None,
) -> str | None:
"""Returns a cursor that, when passed to scroll(), positions at the
first image of the given year-month (by effective_date, not
created_at). None if the bucket is empty.
"""Returns a cursor that, when passed to scroll() with the same sort,
positions at the first image of the given year-month. None if the
bucket is empty.
"""
from sqlalchemy import extract
@@ -282,22 +402,157 @@ class GalleryService:
extract("month", eff) == month,
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
)
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).first()
_require_single_filter(tag_ids, post_id, artist_id)
stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
descending = sort != "oldest"
if descending:
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc())
else:
stmt = stmt.order_by(eff.asc(), ImageRecord.id.asc())
first = (await self.session.execute(stmt.limit(1))).first()
if first is None:
return None
record, eff_date = first
# Cursor is exclusive; we encode a cursor with id+1 so the row itself
# is the first result in the next scroll().
return encode_cursor(eff_date, record.id + 1)
# Cursor is exclusive; nudge the id one past the boundary row (in the
# scan direction) so the row itself is the first result of scroll().
boundary = record.id + 1 if descending else record.id - 1
return encode_cursor(eff_date, boundary)
async def facets(
self, *, tag_ids: list[int] | None = None,
post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, platform: str | None = None,
untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None,
) -> GalleryFacets:
"""Live facet counts scoped to the current filter. Each facet GROUP is
computed with all OTHER active filters applied but its OWN selection
ignored ("minus-self"), so sibling options stay visible/switchable.
No outer join is needed — every clause is a correlated EXISTS or a
column predicate on ImageRecord.
"""
_require_single_filter(tag_ids, post_id, artist_id)
common = {
"tag_ids": tag_ids, "post_id": post_id,
"artist_id": artist_id, "media_type": media_type,
}
# total — the full active filter (the headline result count).
total = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
)).scalar_one()
# platforms — scope minus the platform selection. Inner-join
# provenance→source and COUNT(DISTINCT image) per platform (a
# cross-posted image counts under each of its platforms).
plat_scope = {
**common, "untagged": untagged, "no_artist": no_artist,
"date_from": date_from, "date_to": date_to,
}
src = aliased(Source)
plat_stmt = (
select(src.platform, func.count(distinct(ImageRecord.id)))
.select_from(ImageRecord)
.join(ImageProvenance, ImageProvenance.image_record_id == ImageRecord.id)
.join(src, src.id == ImageProvenance.source_id)
)
plat_stmt = _apply_scope(plat_stmt, **plat_scope).group_by(src.platform)
platforms = [
{"value": p, "count": c}
for p, c in (await self.session.execute(plat_stmt)).all()
]
# Unsourced (filesystem) bucket — same minus-platform scope.
unsourced = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **plat_scope,
platform=UNSOURCED_PLATFORM,
)
)).scalar_one()
if unsourced:
platforms.append({"value": None, "count": unsourced})
# curation flags — each minus its OWN flag.
untagged_count = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, no_artist=no_artist,
date_from=date_from, date_to=date_to, untagged=True,
)
)).scalar_one()
no_artist_count = (await self.session.execute(
_apply_scope(
select(func.count(ImageRecord.id)), **common,
platform=platform, untagged=untagged,
date_from=date_from, date_to=date_to, no_artist=True,
)
)).scalar_one()
# date bounds — scope minus the date params (those drive the picker).
eff = _effective_date_col()
dmin, dmax = (await self.session.execute(
_apply_scope(
select(func.min(eff), func.max(eff)), **common,
platform=platform, untagged=untagged, no_artist=no_artist,
)
)).one()
return GalleryFacets(
total=total, platforms=platforms,
untagged=untagged_count, no_artist=no_artist_count,
date_min=dmin, date_max=dmax,
)
async def similar(
self, image_id: int, limit: int = 100, *,
tag_ids: list[int] | None = None, artist_id: int | None = None,
media_type: str | None = None, platform: str | None = None,
untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None,
) -> list[GalleryImage] | None:
"""Visual "more like this": images ranked by cosine distance to
`image_id`'s SigLIP embedding (pgvector, HNSW-indexed — alembic 0036).
No ML inference here; the embedding was computed at import.
Returns None if the source image doesn't exist (→ 404), [] if it has
no embedding (a video / not-yet-embedded). Composes with the Phase-1/2
scope filters (AND) but REPLACES the date sort — always nearest-first,
bounded to `limit` (no cursor; distance-ranking has no date cursor).
"""
if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200")
src = await self.session.get(ImageRecord, image_id)
if src is None:
return None
if src.siglip_embedding is None:
return []
distance = ImageRecord.siglip_embedding.cosine_distance(src.siglip_embedding)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
stmt = stmt.where(
ImageRecord.siglip_embedding.is_not(None),
ImageRecord.id != image_id,
)
stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=None,
artist_id=artist_id, media_type=media_type,
platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to,
)
stmt = stmt.order_by(distance.asc()).limit(limit)
rows = (await self.session.execute(stmt)).all()
artists = await _artists_for(self.session, [r[0].id for r in rows])
return _gallery_images(rows, artists)
async def get_image_with_tags(self, image_id: int) -> dict | None:
record = await self.session.get(ImageRecord, image_id)
@@ -336,6 +591,9 @@ class GalleryService:
"height": record.height,
"size_bytes": record.size_bytes,
"integrity_status": record.integrity_status,
# Phase 3: lets the modal hide the "Related"/find-similar surface
# for images that have no embedding yet (videos / pending ML).
"has_embedding": record.siglip_embedding is not None,
"created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
@@ -357,17 +615,10 @@ class GalleryService:
}
async def _neighbors(self, record: ImageRecord) -> dict:
# Compute the boundary image's effective_date in Python (one query
# below + the SELECT we already have on `record`) and use it for
# the neighbor comparison. Cheaper than re-deriving in SQL via
# correlated subquery.
boundary_eff = record.created_at
if record.primary_post_id is not None:
post_date = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
if post_date is not None:
boundary_eff = post_date
# The boundary image's sort key is materialized on the row now
# (alembic 0035) — read it directly instead of re-deriving COALESCE
# via an extra Post lookup.
boundary_eff = record.effective_date
eff = _effective_date_col()
prev_stmt = _outer_join_primary_post(
+8
View File
@@ -960,6 +960,14 @@ class Importer:
sp.rollback()
if record.primary_post_id is None:
record.primary_post_id = post.id
# Keep the denormalized gallery sort key (alembic 0035) aligned with
# the primary post's publish date so /scroll orders off
# ix_image_record_effective_date instead of COALESCE-ing across the
# post join. Only override when THIS post is the primary AND carries
# a date; otherwise the column keeps its created_at-equivalent server
# default (matches the old COALESCE(post_date, created_at) fallback).
if record.primary_post_id == post.id and post.post_date is not None:
record.effective_date = post.post_date
self.session.flush()
def _copy_to_library(
+588
View File
@@ -0,0 +1,588 @@
"""Native Patreon JSON:API client (build step 1 of the native ingester).
Clean-room reimplementation of the Patreon `/api/posts` read path. This is a
plain, synchronous client over `requests` — the orchestrator already wraps
sync importer calls, so nothing here needs to be async (mirrors
patreon_resolver.py, which wraps its sync lookup in run_in_executor at the
call site).
Scope (build step 1): fetch + page + parse only. This module is NOT wired into
download_service yet — that is a later step. The public surface here exists so
the later step can drive it:
- PatreonClient(cookies_path).iter_posts(campaign_id)
→ (post, included_index, page_cursor)
- extract_media(post, included_index) → list[MediaItem]
- parse_cursor_from_url(url) → cursor
Drift detection is loud on purpose: Patreon ships JSON:API and the shapes we
depend on (top-level `data`, media resources carrying `file_name`/`url`) are
the contract. If a response comes back as an HTML login page or a media
resource is missing the fields we resolve against, we raise PatreonDriftError
rather than silently yielding empty media — so the later import step surfaces
"Patreon changed something" instead of "creator has no posts".
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import http.cookiejar
import logging
import os
import re
import time
from collections.abc import Iterator
from dataclasses import dataclass
from html import unescape
from pathlib import Path
from urllib.parse import parse_qs, urlsplit
import requests
log = logging.getLogger(__name__)
_POSTS_URL = "https://www.patreon.com/api/posts"
_USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
)
_TIMEOUT_SECONDS = 30.0
# 429 backoff (plan #703): ride out a transient API rate-limit instead of
# failing the whole walk (which would stamp RATE_LIMITED → platform-wide
# cooldown → every Patreon source dark). Honor the server's `Retry-After`;
# otherwise exponential base·2^(n-1), capped. Only after the retries are
# exhausted does the 429 propagate as terminal RATE_LIMITED.
_MAX_429_RETRIES = 3
_BACKOFF_BASE_SECONDS = 2.0
_BACKOFF_CAP_SECONDS = 30.0
def _retry_after_seconds(
resp: requests.Response,
attempt: int,
*,
base: float = _BACKOFF_BASE_SECONDS,
cap: float = _BACKOFF_CAP_SECONDS,
) -> float:
"""Backoff delay for a 429: the `Retry-After` seconds header if present and
numeric, else exponential `base·2^(attempt-1)`, both capped. (HTTP-date form
of Retry-After is rare here and falls through to exponential.)"""
header = resp.headers.get("Retry-After")
if header:
try:
return min(float(header), cap)
except (TypeError, ValueError):
pass
return min(base * (2 ** max(0, attempt - 1)), cap)
# JSON:API request contract (observed from real traffic — see module plan).
_INCLUDE = (
"campaign,access_rules,attachments,attachments_media,audio,images,media,"
"native_video_insights,user,user_defined_tags,ti_checks"
)
_FIELDS_POST = (
"content,post_file,image,post_type,published_at,title,url,patreon_url,"
"current_user_can_view"
)
_FIELDS_MEDIA = "id,image_urls,download_url,metadata,file_name"
_FIELDS_CAMPAIGN = "name,url"
# A CDN download URL embeds a 32-char hex (MD5) path segment; that segment is
# Patreon's stable per-file identity and is what we dedup + ledger against.
# Same role gallery-dl's _filehash plays. Match the FIRST 32-hex run anywhere
# in the URL (path or query); real Patreon CDN URLs carry exactly one.
_FILEHASH_RE = re.compile(r"([0-9a-fA-F]{32})")
# Inline post `content` is HTML; images are emitted as <img ... src="...">.
# Pull every src; downstream dedup collapses any that duplicate a gallery item
# by filehash. Tolerant of attribute ordering and single/double quotes.
_CONTENT_IMG_RE = re.compile(r"<img\b[^>]*?\bsrc=[\"']([^\"']+)[\"']", re.IGNORECASE)
# A bounded, sane extension parse (see importer._safe_ext for the FC gotcha:
# URL-encoded basenames make Path.suffix return base64-ish junk). We only ever
# use this for a fallback filename, never a network call.
_MAX_EXT_LEN = 16
class PatreonAPIError(Exception):
"""Base for native Patreon client failures.
`status_code` carries the HTTP status when the failure was an HTTP response
(None for transport-level / parse failures), so the ingester can map it to a
DownloadResult.error_type (429 → rate_limited, 404 → not_found, …).
"""
def __init__(self, message: str, *, status_code: int | None = None):
super().__init__(message)
self.status_code = status_code
class PatreonAuthError(PatreonAPIError):
"""Authentication / authorization failure — missing or expired session
cookies, an insufficient pledge tier, or an HTML login/challenge page served
where JSON was expected. DISTINCT from drift: the fix is rotating the
credential, not updating the ingester. Maps to error_type 'auth_error'.
"""
class PatreonDriftError(PatreonAPIError):
"""A JSON response did not match the JSON:API shape we depend on.
Raised for: a missing top-level `data` list, `data` not a list, or a media
resource lacking the `file_name`/`url` fields we resolve against. Fail loud
so the import step flags API drift (ingester needs update) instead of
silently importing nothing. An HTML-login / non-JSON body is auth, not
drift — that raises PatreonAuthError.
"""
@dataclass
class MediaItem:
"""One resolved downloadable item belonging to a post.
Fields:
url — the CDN/download URL to fetch.
filename — media `file_name` when present; otherwise the URL basename
(NEVER a network call). Bounded/sane extension.
kind — one of: "images", "image_large", "attachments", "postfile",
"content". Mirrors gallery-dl's `files` content-type names so
the later step can honor the same per-source content_types.
filehash — the 32-char hex (MD5) segment from the CDN URL, or None if
the URL carries no such segment. Used for in-post dedup and
the cross-run seen-ledger.
post_id — the owning post's id (so a flattened media list stays
traceable to its post).
"""
url: str
filename: str
kind: str
filehash: str | None
post_id: str
def _load_session(cookies_path: str | Path | None) -> requests.Session:
session = requests.Session()
session.headers.update(
{
"User-Agent": _USER_AGENT,
"Accept": "application/vnd.api+json",
}
)
if cookies_path and os.path.isfile(str(cookies_path)):
try:
jar = http.cookiejar.MozillaCookieJar(str(cookies_path))
jar.load(ignore_discard=True, ignore_expires=True)
session.cookies = jar # type: ignore[assignment]
except (OSError, http.cookiejar.LoadError) as exc:
log.warning("Could not load Patreon cookies from %s: %s", cookies_path, exc)
return session
def _filehash(url: str) -> str | None:
if not url:
return None
match = _FILEHASH_RE.search(url)
return match.group(1).lower() if match else None
def _safe_ext(name: str) -> str:
suffix = Path(name).suffix.lower()
if not suffix or len(suffix) > _MAX_EXT_LEN:
return ""
if not all(c.isalnum() for c in suffix[1:]):
return ""
return suffix
def _basename_from_url(url: str) -> str:
"""Derive a sane filename from a URL when the media has no file_name.
Strips query/fragment, takes the path basename, and drops a junk
extension (the importer._safe_ext gotcha) so we never write base64 noise
as a name. Falls back to the filehash, then to "file".
"""
path = urlsplit(url).path
base = os.path.basename(path)
if base:
ext = _safe_ext(base)
stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
# Keep the stem bounded; URL-encoded stems can be enormous.
stem = stem[:120] or "file"
return f"{stem}{ext}"
fh = _filehash(url)
return fh or "file"
def parse_cursor_from_url(url: str | None) -> str | None:
"""Extract the `page[cursor]` query param from a links.next URL."""
if not url:
return None
query = urlsplit(url).query
values = parse_qs(query).get("page[cursor]")
if values and values[0]:
return values[0]
return None
class PatreonClient:
"""Synchronous Patreon JSON:API read client.
Construct with a path to a Netscape cookies.txt (the same file
CredentialService.get_cookies_path materializes). Cookies are loaded into a
requests.Session; no secure-context APIs are used.
"""
def __init__(
self,
cookies_path: str | Path | None,
*,
request_sleep: float = 0.0,
max_retries: int = _MAX_429_RETRIES,
):
self.cookies_path = str(cookies_path) if cookies_path else None
self._session = _load_session(cookies_path)
# Politeness: seconds to sleep before each /api/posts page fetch (paces
# the rate-limited API endpoint). 0 = no pacing. plan #703.
self._request_sleep = request_sleep or 0.0
self._max_retries = max_retries
# -- request -----------------------------------------------------------
def _params(self, campaign_id: str, cursor: str | None) -> dict[str, str]:
params = {
"include": _INCLUDE,
"fields[post]": _FIELDS_POST,
"fields[media]": _FIELDS_MEDIA,
"fields[campaign]": _FIELDS_CAMPAIGN,
"filter[campaign_id]": campaign_id,
"filter[contains_exclusive_posts]": "true",
"filter[is_draft]": "false",
"sort": "-published_at",
"json-api-version": "1.0",
}
if cursor:
params["page[cursor]"] = cursor
return params
def _fetch(self, campaign_id: str, cursor: str | None) -> dict:
if self._request_sleep > 0:
time.sleep(self._request_sleep) # pace the API endpoint
attempt = 0
while True:
try:
resp = self._session.get(
_POSTS_URL,
params=self._params(campaign_id, cursor),
timeout=_TIMEOUT_SECONDS,
)
except requests.RequestException as exc:
raise PatreonAPIError(
f"Patreon posts request failed (campaign_id={campaign_id}): {exc}"
) from exc
# Transient rate-limit: back off and retry rather than failing the
# whole walk. Only a PERSISTENT 429 (retries exhausted) falls
# through to the terminal RATE_LIMITED raise below.
if resp.status_code == 429 and attempt < self._max_retries:
attempt += 1
delay = _retry_after_seconds(resp, attempt)
log.warning(
"Patreon 429 (campaign_id=%s) — backing off %.1fs (retry %d/%d)",
campaign_id, delay, attempt, self._max_retries,
)
time.sleep(delay)
continue
break
if resp.status_code in (401, 403):
# Auth rejected — expired/missing cookies or an insufficient tier.
# Actionable as "rotate credentials", so it's auth, not drift/http.
raise PatreonAuthError(
f"Patreon posts API returned HTTP {resp.status_code} — auth "
f"rejected (cookies expired or tier insufficient; "
f"campaign_id={campaign_id})",
status_code=resp.status_code,
)
if resp.status_code != 200:
raise PatreonAPIError(
f"Patreon posts API returned HTTP {resp.status_code} "
f"(campaign_id={campaign_id})",
status_code=resp.status_code,
)
try:
payload = resp.json()
except ValueError as exc:
# A non-JSON body here is almost always the HTML login/challenge
# page served when cookies are missing/expired — that is an AUTH
# failure (rotate cookies), not API drift (update the ingester) and
# not a transient network error.
raise PatreonAuthError(
"Patreon posts API returned a non-JSON response (likely an "
f"HTML login/challenge page — session expired; "
f"campaign_id={campaign_id}): {exc}"
) from exc
return payload
# -- parsing -----------------------------------------------------------
@staticmethod
def _transform(response: dict) -> dict:
"""Flatten the JSON:API `included` array for relationship resolution.
Returns a dict keyed by `(type, id)` → that resource's `attributes`
(so a post's relationships can be resolved in O(1)). Missing/oddly
shaped `included` entries are skipped rather than fatal — drift
detection for the top-level shape lives in _validate_response.
"""
index: dict[tuple[str, str], dict] = {}
for inc in response.get("included") or []:
if not isinstance(inc, dict):
continue
rtype = inc.get("type")
rid = inc.get("id")
if rtype is None or rid is None:
continue
index[(str(rtype), str(rid))] = inc.get("attributes") or {}
return index
@staticmethod
def _validate_response(response: dict) -> None:
if not isinstance(response, dict):
raise PatreonDriftError("Patreon response was not a JSON object")
if "data" not in response:
raise PatreonDriftError("Patreon response missing top-level 'data' key")
data = response.get("data")
if not isinstance(data, list):
raise PatreonDriftError("Patreon response 'data' was not a list")
def _related_ids(self, post: dict, rel_name: str) -> list[str]:
rels = post.get("relationships") or {}
rel = rels.get(rel_name) or {}
data = rel.get("data")
if isinstance(data, dict): # to-one relationship
data = [data]
if not isinstance(data, list):
return []
ids: list[str] = []
for ref in data:
if isinstance(ref, dict) and ref.get("id") is not None:
ids.append(str(ref["id"]))
return ids
@staticmethod
def _media_url(attrs: dict) -> str | None:
"""Pick the best fetchable URL for a media resource.
Prefer the full-size `download_url`; fall back to the largest
`image_urls` size. gallery-dl prefers download_url too, only dipping
into image_urls when a smaller configured size is requested — FC
always wants the original, so download_url first.
"""
download_url = attrs.get("download_url")
if isinstance(download_url, str) and download_url:
return download_url
image_urls = attrs.get("image_urls")
if isinstance(image_urls, dict):
for key in ("original", "full", "large", "default"):
candidate = image_urls.get(key)
if isinstance(candidate, str) and candidate:
return candidate
# Otherwise take any non-empty string value.
for candidate in image_urls.values():
if isinstance(candidate, str) and candidate:
return candidate
return None
def _media_item(
self, attrs: dict, kind: str, post_id: str, *, require_file_name: bool
) -> MediaItem:
url = self._media_url(attrs)
if not url:
raise PatreonDriftError(
f"Patreon media (post {post_id}, kind={kind}) had no resolvable URL "
f"(no download_url / image_urls)"
)
file_name = attrs.get("file_name")
if require_file_name and not (isinstance(file_name, str) and file_name):
raise PatreonDriftError(
f"Patreon media resource (post {post_id}, kind={kind}) missing "
f"'file_name'"
)
filename = file_name if isinstance(file_name, str) and file_name else _basename_from_url(url)
return MediaItem(
url=url,
filename=filename,
kind=kind,
filehash=_filehash(url),
post_id=post_id,
)
def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
"""Resolve all downloadable media for one post.
Walks the kinds in the same order gallery-dl does — images,
image_large (post cover), attachments, postfile, content (inline
<img>) — and dedups within the post by filehash (first wins). The
image_large cover commonly duplicates a gallery image; deduping by
filehash collapses them to the gallery item (encountered first).
"""
post_id = str(post.get("id") or "")
attrs = post.get("attributes") or {}
items: list[MediaItem] = []
def _resolve_rel(rel_name: str, kind: str) -> None:
for mid in self._related_ids(post, rel_name):
media_attrs = included_index.get(("media", mid))
if media_attrs is None:
# Referenced but not in `included`: a media id with no
# resource is drift (we asked for include=media).
raise PatreonDriftError(
f"Patreon post {post_id} references media {mid} "
f"({rel_name}) not present in 'included'"
)
items.append(
self._media_item(
media_attrs, kind, post_id, require_file_name=True
)
)
# 1. gallery images
_resolve_rel("images", "images")
# 2. image_large — the post-level cover (`image.large_url`). Not a
# media relationship; it lives on the post attributes.
image = attrs.get("image")
if isinstance(image, dict):
large_url = image.get("large_url") or image.get("url")
if isinstance(large_url, str) and large_url:
items.append(
MediaItem(
url=large_url,
filename=_basename_from_url(large_url),
kind="image_large",
filehash=_filehash(large_url),
post_id=post_id,
)
)
# 3. attachments
_resolve_rel("attachments_media", "attachments")
# 4. postfile — the post's primary attached file (`post_file`).
post_file = attrs.get("post_file")
if isinstance(post_file, dict):
pf_url = post_file.get("url") or post_file.get("download_url")
if isinstance(pf_url, str) and pf_url:
pf_name = post_file.get("name")
filename = (
pf_name
if isinstance(pf_name, str) and pf_name
else _basename_from_url(pf_url)
)
items.append(
MediaItem(
url=pf_url,
filename=filename,
kind="postfile",
filehash=_filehash(pf_url),
post_id=post_id,
)
)
# 5. content — inline <img> in the post HTML body.
content = attrs.get("content")
if isinstance(content, str) and content:
for raw_src in _CONTENT_IMG_RE.findall(content):
src = unescape(raw_src)
if not src:
continue
items.append(
MediaItem(
url=src,
filename=_basename_from_url(src),
kind="content",
filehash=_filehash(src),
post_id=post_id,
)
)
return _dedup_by_filehash(items)
# -- iteration ---------------------------------------------------------
def iter_posts(
self, campaign_id: str, cursor: str | None = None
) -> Iterator[tuple[dict, dict, str | None]]:
"""Yield (post, included_index, page_cursor) for every post in the feed.
Pages newest→oldest via `links.next`, validating each response for
drift before yielding. The triple gives a caller everything it needs to
resolve and checkpoint without re-fetching:
- post — the raw post resource.
- included_index — the page's flattened `included` (the same object
for every post on a page), to pass straight to
extract_media(post, included_index).
- page_cursor — the cursor that FETCHED this post's page (None for
the first page). The caller checkpoints THIS value,
matching the existing backfill cursor logic where
the saved cursor re-fetches the page being
processed (so a chunk cut mid-page resumes the page,
not the one after it).
"""
current_cursor = cursor
while True:
response = self._fetch(campaign_id, current_cursor)
self._validate_response(response)
page_cursor = current_cursor
included_index = self._transform(response)
for post in response.get("data") or []:
if isinstance(post, dict):
yield post, included_index, page_cursor
next_url = (response.get("links") or {}).get("next")
next_cursor = parse_cursor_from_url(next_url)
if not next_cursor:
return
current_cursor = next_cursor
# -- verify ------------------------------------------------------------
def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:
"""Cheap auth probe: fetch the first `/api/posts` page and report whether
the credential authenticated, WITHOUT downloading anything.
Returns `(ok, message)` matching the credential-verify contract:
- True — authenticated (the feed returned a valid JSON:API page).
- False — the credential was rejected (PatreonAuthError: 401/403, or an
HTML login page → cookies expired / tier insufficient).
- None — inconclusive: API drift (our parser is stale, not a cred
problem) or a transient network/HTTP error.
"""
try:
response = self._fetch(campaign_id, None)
self._validate_response(response)
except PatreonAuthError as exc:
return False, f"Patreon rejected the credential — {exc}"
except PatreonDriftError as exc:
return None, f"Couldn't verify — Patreon's API shape changed: {exc}"
except PatreonAPIError as exc:
return None, f"Couldn't verify (network/HTTP issue): {exc}"
return True, "Credentials valid — the Patreon feed authenticated."
def _dedup_by_filehash(items: list[MediaItem]) -> list[MediaItem]:
"""Drop later items sharing a filehash with an earlier one (first wins).
Items with no filehash (None) are never deduped against each other — we
can't prove they're the same file, so keep them all.
"""
seen: set[str] = set()
out: list[MediaItem] = []
for item in items:
if item.filehash is not None:
if item.filehash in seen:
continue
seen.add(item.filehash)
out.append(item)
return out
+416
View File
@@ -0,0 +1,416 @@
"""Native Patreon media downloader (build step 2b of the native ingester).
Given a Patreon post and its already-resolved `MediaItem`s (from
patreon_client.extract_media), download the media to the EXACT on-disk layout
gallery-dl produces, write a sidecar JSON the existing importer consumes, and
report per-media outcomes.
This module is PURE: no DB. The cross-run seen-ledger and the iter_posts
orchestration are a LATER step. The tier-1 (seen) skip is an INJECTED predicate
(`is_seen`), so this module needs no DB and is unit-testable without network.
On-disk layout (matches gallery-dl):
<images_root>/<artist_slug>/patreon/<DIR>/<NN>_<filename>
where <DIR> = "<YYYY-MM-DD>_<post_id>_<title40>" (date prefix omitted when the
post's published_at is unparseable), <NN> is the 1-based index of the item in
the post zero-padded to 2, and <filename> is MediaItem.filename. The sidecar
is written next to the media as <NN>_<stem>.json (media_path.with_suffix).
Video: a MediaItem whose URL is a Mux/HLS stream (host stream.mux.com or path
endswith .m3u8) is fetched with yt-dlp (subprocess), passing the same
Referer/Origin headers gallery-dl forwards for Mux playback (see gallery_dl.py
_get_default_config). yt-dlp may remux to a container of its own choosing, so we
accept the actual output extension and record the real path.
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import contextlib
import json
import logging
import os
import shutil
import subprocess
import time
from collections.abc import Callable
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from urllib.parse import urlsplit
import requests
from .file_validator import is_validatable, validate_file
from .patreon_client import _load_session, _retry_after_seconds
log = logging.getLogger(__name__)
_TITLE_MAX = 40
_TIMEOUT_SECONDS = 120.0
_CHUNK = 1 << 16
# A CDN media GET rarely 429s, but if it does, a couple of backoff retries keep
# one throttled file from becoming a per-item error (plan #703).
_MAX_MEDIA_429_RETRIES = 2
# Referer/Origin yt-dlp must send for Mux-hosted Patreon video. Mux's JWT
# playback policy checks Referer/Origin on every request, so yt-dlp must send
# Patreon's, not its own default. (gallery-dl forwarded the same headers before
# the #697 cutover removed its Patreon path; this is now the only place they
# live.)
_VIDEO_HEADERS = {
"Referer": "https://www.patreon.com/",
"Origin": "https://www.patreon.com",
}
# Characters Windows/gallery-dl path-restrict forbids, plus path separators.
_FORBIDDEN = set('<>:"/\\|?*')
def _sanitize(name: str) -> str:
"""Make `name` safe for a single filesystem path segment.
Replaces path separators, the Windows-forbidden set <>:"/\\|?* and control
characters with `_`, then strips trailing dots/spaces (gallery-dl
path-restrict behavior). Never returns empty (falls back to "_").
"""
out = []
for ch in name:
if ch in _FORBIDDEN or ord(ch) < 32:
out.append("_")
else:
out.append(ch)
cleaned = "".join(out).rstrip(". ")
return cleaned or "_"
def _is_video_url(url: str) -> bool:
parts = urlsplit(url)
if parts.hostname and parts.hostname.lower() == "stream.mux.com":
return True
return parts.path.lower().endswith(".m3u8")
def _post_dir_name(post: dict) -> str:
"""Build the post directory name matching gallery-dl's layout."""
post_id = str(post.get("id") or "")
attrs = post.get("attributes") or {}
title = attrs.get("title")
title = title if isinstance(title, str) else ""
title40 = title[:_TITLE_MAX]
published = attrs.get("published_at")
date_prefix = None
if isinstance(published, str) and published:
s = published.strip()
if s.endswith("Z"):
s = s[:-1] + "+00:00"
try:
dt = datetime.fromisoformat(s)
except ValueError:
dt = None
if dt is not None:
date_prefix = f"{dt:%Y-%m-%d}"
if date_prefix:
raw = f"{date_prefix}_{post_id}_{title40}"
else:
raw = f"{post_id}_{title40}"
return _sanitize(raw)
@dataclass
class MediaOutcome:
"""Per-media result of a download_post pass.
status is one of: "downloaded", "skipped_seen", "skipped_disk", "error".
`path` is the final on-disk path for "downloaded" (the actual yt-dlp output
for video), or the path that already existed for "skipped_disk"; None for
"skipped_seen" and (usually) "error". `error` carries the failure reason for
"error", else None.
"""
media: object # MediaItem (avoid importing the name for a bare annotation)
status: str
path: Path | None
error: str | None
class PatreonDownloader:
"""Download resolved Patreon media to gallery-dl's on-disk layout.
PURE: no DB. The HTTP session and the yt-dlp invocation are injectable seams
so tests run without network or a real subprocess:
- pass `session=` to stub `session.get`, or monkeypatch `_fetch_to_file`.
- monkeypatch `_run_ytdlp` to avoid spawning yt-dlp.
"""
def __init__(
self,
images_root: Path,
cookies_path: str | None = None,
*,
validate: bool = True,
rate_limit: float = 0.0,
session: requests.Session | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
self._validate = validate
# Politeness: seconds to sleep before each actual media download (paces
# the CDN; honors ImportSettings.download_rate_limit_seconds, the same
# value gallery-dl used as its between-downloads `sleep`). 0 = no pacing.
# Applied only to real downloads, not to seen/disk skips. plan #703.
self._rate_limit = rate_limit or 0.0
# Build a cookie-loaded session the same way patreon_client does, so the
# CDN GETs carry the creator's auth.
self.session = session if session is not None else _load_session(cookies_path)
# -- public ------------------------------------------------------------
def download_post(
self,
post: dict,
media_items: list,
artist_slug: str,
*,
is_seen: Callable[[object], bool] = lambda m: False,
) -> list[MediaOutcome]:
"""Download every media item of one post; return per-item outcomes.
Builds the post directory, iterates media (1-based NN), applies the
two-tier skip (injected is_seen, then disk), downloads (plain GET or
yt-dlp for video), writes the sidecar for each freshly-downloaded item,
validates, and returns outcomes. Resilient: one media's failure yields
an "error" outcome for that item; the rest proceed.
"""
post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
outcomes: list[MediaOutcome] = []
for i, media in enumerate(media_items, start=1):
try:
outcomes.append(
self._download_one(post, media, post_dir, artist_slug, i, is_seen)
)
except Exception as exc: # resilient: isolate one item's failure
log.warning(
"Patreon media failed (post %s, item %d): %s",
post.get("id"), i, exc,
)
outcomes.append(
MediaOutcome(media=media, status="error", path=None, error=str(exc))
)
return outcomes
# -- per-item ----------------------------------------------------------
def _download_one(
self,
post: dict,
media,
post_dir: Path,
artist_slug: str,
index: int,
is_seen: Callable[[object], bool],
) -> MediaOutcome:
# tier-1: seen ledger (injected; no DB here).
if is_seen(media):
return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
nn = f"{index:02d}"
final_name = _sanitize(f"{nn}_{media.filename}")
media_path = post_dir / final_name
# tier-2: already on disk.
if media_path.exists():
return MediaOutcome(
media=media, status="skipped_disk", path=media_path, error=None
)
# Video may land at a different extension; honor a pre-existing remux.
if _is_video_url(media.url):
existing = self._existing_video_output(media_path)
if existing is not None:
return MediaOutcome(
media=media, status="skipped_disk", path=existing, error=None
)
post_dir.mkdir(parents=True, exist_ok=True)
# Pace real downloads only (the skips above already returned). plan #703.
if self._rate_limit > 0:
time.sleep(self._rate_limit)
if _is_video_url(media.url):
out_path = self._run_ytdlp(media.url, media_path, _VIDEO_HEADERS)
if out_path is None or not Path(out_path).exists():
return MediaOutcome(
media=media,
status="error",
path=None,
error="yt-dlp produced no output",
)
out_path = Path(out_path)
else:
out_path = self._fetch_get(media.url, media_path)
invalid = self._validate_path(out_path, artist_slug)
if invalid is not None:
return MediaOutcome(
media=media, status="error", path=None, error=invalid
)
self._write_sidecar(post, out_path)
return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
# -- download seams ----------------------------------------------------
def _fetch_get(self, url: str, dest: Path) -> Path:
"""Stream `url` to a .part file then atomic-rename to `dest`.
Thin wrapper over `_fetch_to_file` so tests can stub either the whole
GET path (`_fetch_to_file`) or just `session.get`.
"""
part = dest.with_name(dest.name + ".part")
try:
self._fetch_to_file(url, part)
except Exception:
with contextlib.suppress(OSError):
part.unlink()
raise
os.replace(part, dest)
return dest
def _fetch_to_file(self, url: str, dest: Path) -> None:
"""Stream a non-video URL to `dest` via the (stubbable) session.
Retries a transient 429 with backoff (plan #703); any other non-2xx
falls through to raise_for_status → HTTPError, which download_post turns
into a resilient per-item error outcome.
"""
attempt = 0
while True:
resp = self.session.get(url, stream=True, timeout=_TIMEOUT_SECONDS)
if resp.status_code == 429 and attempt < _MAX_MEDIA_429_RETRIES:
attempt += 1
delay = _retry_after_seconds(resp, attempt)
log.warning(
"Patreon media 429 (%s) — backing off %.1fs (retry %d/%d)",
url, delay, attempt, _MAX_MEDIA_429_RETRIES,
)
time.sleep(delay)
continue
break
resp.raise_for_status()
with open(dest, "wb") as fh:
for chunk in resp.iter_content(chunk_size=_CHUNK):
if chunk:
fh.write(chunk)
def _run_ytdlp(self, url: str, dest: Path, headers: dict) -> Path | None:
"""Invoke yt-dlp to fetch a Mux/HLS stream to (around) `dest`.
Returns the actual output path (yt-dlp may remux to a different
container, so we resolve the real file afterward). Overridden/
monkeypatched in tests to avoid spawning a real subprocess.
The output template uses `dest` without its extension; yt-dlp appends
the chosen container extension. We pass Referer/Origin (Mux JWT policy)
and the cookies file.
"""
dest = Path(dest)
out_template = str(dest.with_suffix("")) + ".%(ext)s"
cmd = ["yt-dlp", "--no-progress", "-o", out_template]
for key, value in headers.items():
cmd += ["--add-header", f"{key}:{value}"]
if self.cookies_path and os.path.isfile(self.cookies_path):
cmd += ["--cookies", self.cookies_path]
cmd.append(url)
try:
subprocess.run(
cmd,
check=True,
capture_output=True,
text=True,
timeout=_TIMEOUT_SECONDS,
)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("yt-dlp failed for %s: %s", url, exc)
return None
return self._existing_video_output(dest)
def _existing_video_output(self, dest: Path) -> Path | None:
"""Find a yt-dlp output for `dest` regardless of chosen extension.
Returns `dest` itself if present, else any sibling sharing the same
stem (the remuxed container). None if nothing matched.
"""
dest = Path(dest)
if dest.exists():
return dest
stem = dest.with_suffix("").name
parent = dest.parent
if not parent.is_dir():
return None
for cand in sorted(parent.iterdir()):
if cand.is_file() and cand.stem == stem and cand.name != dest.name:
return cand
return None
# -- validation --------------------------------------------------------
def _validate_path(self, path: Path, artist_slug: str) -> str | None:
"""Validate a freshly-written file; quarantine + return reason if bad.
Mirrors gallery_dl._validate_and_quarantine: fail-open for unknown
formats, move the corrupt file to _quarantine/<slug>/patreon, and return
the failure reason string (None when ok / not validatable / disabled).
"""
if not self._validate or not is_validatable(path):
return None
try:
result = validate_file(path)
except Exception as exc:
log.warning("Validator raised on %s: %s", path, exc)
return None
if result.ok:
return None
quarantine_root = self.images_root / "_quarantine" / artist_slug / "patreon"
try:
quarantine_root.mkdir(parents=True, exist_ok=True)
dest = quarantine_root / path.name
counter = 1
while dest.exists():
dest = quarantine_root / f"{path.stem}.{counter}{path.suffix}"
counter += 1
shutil.move(str(path), str(dest))
except OSError as exc:
log.error("Failed to quarantine %s: %s. File left in place.", path, exc)
return result.reason or "validation failed"
# -- sidecar -----------------------------------------------------------
def _write_sidecar(self, post: dict, media_path: Path) -> Path:
"""Write the importer-consumed sidecar next to `media_path`.
Patreon uses base-default sidecar keys (parse_sidecar maps
category->platform, id->external_post_id, title->post_title,
content->description, published_at->post_date, url->post_url). Patreon
registers no derive_post_url, so `url` is trusted as the permalink — we
pass the post's attributes.url.
"""
attrs = post.get("attributes") or {}
title = attrs.get("title")
content = attrs.get("content")
published = attrs.get("published_at")
url = attrs.get("url")
data = {
"category": "patreon",
"id": str(post.get("id") or ""),
"title": title if isinstance(title, str) else "",
"content": content if isinstance(content, str) else "",
"published_at": published if isinstance(published, str) else None,
"url": url if isinstance(url, str) else None,
}
sidecar_path = media_path.with_suffix(".json")
sidecar_path.write_text(json.dumps(data, indent=2))
return sidecar_path
+414
View File
@@ -0,0 +1,414 @@
"""Native Patreon ingester — phase-2 orchestrator (build step 3).
Ties build steps 1 (`patreon_client`) and 2 (`patreon_downloader` +
`patreon_seen_media`) together into a single sync walk that REPLACES the
gallery-dl subprocess for Patreon. `download_service.download_source` calls
`PatreonIngester.run(...)` from phase 2 (in a thread, via run_in_executor, since
everything here is sync `requests`/`subprocess`) and gets back a
`DownloadResult` — the exact shape gallery-dl returns — so phase 1 (DB setup)
and phase 3 (import → pHash dedup → thumbnails → ML) are untouched and cannot
tell the difference.
Three modes (selected by `download_service` from `config_overrides` state):
- tick — newest→oldest, skip seen (tier-1 ledger + tier-2 disk), early-out
after N contiguous already-have-it items (the cheap native
equivalent of gallery-dl's `exit:20`, now free of per-file HEADs).
- backfill — full-history walk in a time-boxed chunk, resuming from the
pagination cursor checkpoint; reaches the bottom → "complete".
- recovery — like backfill but BYPASSES the tier-1 seen-ledger, so
deliberately-dropped-and-deleted near-dups get re-fetched and
re-evaluated under the current pHash threshold (tier-2 disk skip
still spares files we kept). Triggered by the same #693 backfill
state machine plus the `_backfill_bypass_seen` flag, so the whole
cursor/chunk/complete/stall lifecycle is reused verbatim.
Cursor contract: the ingester emits gallery-dl-style ``Cursor: <token>`` lines
into the returned `stdout`, one per page it walks. That is exactly what
`download_service`'s existing backfill lifecycle reads via
`parse_last_cursor(stdout, stderr)` — so checkpointing, the TIMEOUT→PARTIAL
reclassification, and completion detection all keep working unchanged.
The seen-ledger lives in Postgres (`patreon_seen_media`). The ingester opens a
SHORT-LIVED sync session per page batch (via an injected sessionmaker) — never
held across a network fetch — so the multi-minute walk can't strand a checked-out
connection for the server to reap ([[db-connection-held-across-subprocess]]).
FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
"""
from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import Callable
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..models import PatreonSeenMedia
from .gallery_dl import DownloadResult, ErrorType
from .patreon_client import (
MediaItem,
PatreonAPIError,
PatreonAuthError,
PatreonClient,
PatreonDriftError,
)
from .patreon_downloader import PatreonDownloader
from .patreon_resolver import resolve_campaign_id_for_source
log = logging.getLogger(__name__)
# gallery-dl's `exit:20` default ported over: stop a tick after this many
# CONTIGUOUS already-have-it media (seen-ledger or on-disk). Native walks have
# zero per-file HEADs, so the only cost of a higher number is a few extra cheap
# ledger lookups — 20 is operator-set headroom against paywalled/undownloadable
# items interleaving with archived ones.
_TICK_SEEN_THRESHOLD = 20
# Ledger keys are stored in patreon_seen_media.filehash VARCHAR(128); bound any
# synthesized key so a pathologically long file_name can't overflow the column.
_LEDGER_KEY_MAX = 128
def _ledger_key(media: MediaItem) -> str:
"""Stable per-media identity for the cross-run seen-ledger.
A Patreon CDN URL carries a 32-char MD5 (`media.filehash`) — that is the
natural key. Some media have none: Mux/HLS video (`stream.mux.com`, no
content hash at discovery) and the odd inline-content `<img>` pointing at a
hashless URL. The plan calls the video case the ``video:<post_id>:<media_id>``
sentinel; `MediaItem` carries no media_id, so the post-scoped filename is the
stable proxy. Bounded to the column width.
"""
if media.filehash:
return media.filehash
return f"{media.post_id}:{media.filename}"[:_LEDGER_KEY_MAX]
class PatreonIngester:
"""Walk a Patreon campaign's posts, download unseen media, return a
`DownloadResult`.
Construct with the per-source `cookies_path` and a sync sessionmaker for the
seen-ledger. `client` / `downloader` are injectable seams so unit tests run
without network, subprocess, or a real CDN.
"""
def __init__(
self,
images_root: Path,
cookies_path: str | None,
session_factory: Callable[[], object],
*,
validate: bool = True,
rate_limit: float = 0.0,
request_sleep: float = 0.0,
client: PatreonClient | None = None,
downloader: PatreonDownloader | None = None,
):
self.images_root = Path(images_root)
self.cookies_path = str(cookies_path) if cookies_path else None
self.session_factory = session_factory
# Pacing (plan #703): request_sleep paces the API page fetches,
# rate_limit paces the media downloads. Injected client/downloader (in
# tests) already carry their own pacing, so these only apply to the
# default-constructed ones.
self.client = (
client
if client is not None
else PatreonClient(cookies_path, request_sleep=request_sleep)
)
self.downloader = (
downloader
if downloader is not None
else PatreonDownloader(
self.images_root, cookies_path, validate=validate, rate_limit=rate_limit
)
)
# -- public ------------------------------------------------------------
def run(
self,
*,
source_id: int,
campaign_id: str,
artist_slug: str,
url: str,
mode: str,
resume_cursor: str | None = None,
time_budget_seconds: float = 870.0,
seen_threshold: int = _TICK_SEEN_THRESHOLD,
) -> DownloadResult:
"""Walk + download for one source, returning a gallery-dl-shaped result.
`mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
seen-ledger (tier-2 disk still skips kept files). The walk stops on:
- budget exhaustion (time_budget_seconds) → TIMEOUT / PARTIAL
- tick early-out (seen_threshold contiguous seen) → success
- reaching the bottom of the feed → success (rc 0)
A client-level failure (drift / auth / network) fails the whole run loud.
"""
bypass_seen = mode == "recovery"
start = time.monotonic()
log_lines: list[str] = []
written: list[str] = []
downloaded = 0
errors = 0
consecutive_seen = 0
emitted_cursor: str | None = None
reached_bottom = False
budget_hit = False
early_out = False
def _result(
*, success: bool, return_code: int,
error_type: ErrorType | None, error_message: str | None,
) -> DownloadResult:
return DownloadResult(
success=success,
url=url,
artist_slug=artist_slug,
platform="patreon",
files_downloaded=downloaded,
files_quarantined=0,
quarantined_paths=[],
written_paths=written,
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
error_type=error_type,
error_message=error_message,
duration_seconds=time.monotonic() - start,
)
try:
for post, included, page_cursor in self.client.iter_posts(
campaign_id, cursor=resume_cursor
):
# Checkpoint: emit the cursor that FETCHED this page once, the
# moment we START it — so a chunk cut mid-page resumes the page,
# not the one after it (matches the gallery-dl cursor semantics
# download_service's lifecycle already depends on).
if page_cursor and page_cursor != emitted_cursor:
log_lines.append(f"Cursor: {page_cursor}")
emitted_cursor = page_cursor
# Time-box check at the post boundary (coarse, like a gallery-dl
# chunk). Backfill/recovery resume from emitted_cursor next chunk.
if time.monotonic() - start >= time_budget_seconds:
budget_hit = True
break
media = self.client.extract_media(post, included)
if not media:
continue
keys = [_ledger_key(m) for m in media]
seen = (
set()
if bypass_seen
else self._seen_keys(source_id, keys)
)
def _is_seen(m: MediaItem, _seen=seen) -> bool:
return _ledger_key(m) in _seen
outcomes = self.downloader.download_post(
post, media, artist_slug, is_seen=_is_seen
)
to_mark: list[tuple[str, str]] = []
for media_item, outcome in zip(media, outcomes, strict=False):
key = _ledger_key(media_item)
if outcome.status == "downloaded":
downloaded += 1
if outcome.path is not None:
written.append(str(outcome.path))
to_mark.append((key, media_item.post_id))
consecutive_seen = 0
elif outcome.status == "skipped_disk":
# Already on disk (a prior run). Reconcile the ledger so a
# later tick skips it at tier-1 without a disk stat, but
# do NOT re-feed it to phase 3 — attach_in_place would see
# the duplicate sha256 and unlink the on-disk copy.
to_mark.append((key, media_item.post_id))
consecutive_seen += 1
elif outcome.status == "skipped_seen":
consecutive_seen += 1
elif outcome.status == "error":
errors += 1
# An error neither advances nor resets the run-of-seen.
if mode == "tick" and consecutive_seen >= seen_threshold:
early_out = True
break
# Mark seen AFTER the network fetch, on its own short session.
if to_mark:
self._mark_seen(source_id, to_mark)
if early_out:
break
else:
reached_bottom = True
except PatreonAPIError as exc:
# Base of PatreonAuthError + PatreonDriftError — catches every
# client-level failure; _failure_result maps it to a typed error.
return self._failure_result(exc, _result)
if errors:
log_lines.append(f"{errors} media item(s) failed")
log_lines.append(
f"Patreon ingest ({mode}): {downloaded} downloaded, "
f"{errors} error(s)"
+ (", reached end" if reached_bottom else "")
+ (", time-boxed" if budget_hit else "")
)
if budget_hit:
# A chunk that hit its time-box but made forward progress is a
# NORMAL chunk boundary, not a failure (PARTIAL → status "ok"); the
# next chunk resumes from the emitted cursor. No progress → TIMEOUT,
# which feeds download_service's backfill stall-guard. rc<0 mirrors
# subprocess TimeoutExpired so completion detection stays false.
made_progress = downloaded > 0 or emitted_cursor != resume_cursor
if made_progress:
return _result(
success=False, return_code=-1,
error_type=ErrorType.PARTIAL,
error_message=(
f"Patreon backfill chunk: {downloaded} file(s) — continuing"
),
)
return _result(
success=False, return_code=-1,
error_type=ErrorType.TIMEOUT,
error_message="Patreon chunk timed out with no progress",
)
# Normal success: reached the bottom, or a tick that early-outed. rc 0 +
# error_type None is REQUIRED for a backfill/recovery walk that reached
# the bottom to be marked COMPLETE by
# download_service._apply_backfill_lifecycle — so we return None even
# when downloaded == 0 (a re-confirming walk that found nothing new still
# completed). NO_NEW_CONTENT would read as "not finished" there and trip
# the stall guard. success=True maps to status "ok" regardless. A tick
# that early-outed also returns here; ticks never set backfill state so
# the lifecycle is a no-op for them.
return _result(
success=True, return_code=0,
error_type=None, error_message=None,
)
# -- failure mapping ---------------------------------------------------
def _failure_result(self, exc: Exception, _result) -> DownloadResult:
"""Map a client-level exception to a loud, typed failed DownloadResult.
We NEVER return a silent zero-download "success" — the whole point of the
native ingester is to fail RED when Patreon's API shape or our auth
changes. The typed mapping lets FailingSourcesCard render the right chip
and tells the operator what to do:
- PatreonAuthError → AUTH_ERROR (rotate cookies)
- PatreonDriftError → API_DRIFT (ingester field-set/parser needs update)
- HTTP 429 / 404 → RATE_LIMITED / NOT_FOUND
- other HTTP status → HTTP_ERROR; transport failure → NETWORK_ERROR
PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
they must be matched before the generic HTTP/transport fallthrough.
"""
message = str(exc)
if isinstance(exc, PatreonAuthError):
error_type = ErrorType.AUTH_ERROR
elif isinstance(exc, PatreonDriftError):
error_type = ErrorType.API_DRIFT
message = f"Patreon API changed — ingester needs update: {message}"
else: # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
status = getattr(exc, "status_code", None)
if status == 429:
error_type = ErrorType.RATE_LIMITED
elif status == 404:
error_type = ErrorType.NOT_FOUND
elif status is not None:
error_type = ErrorType.HTTP_ERROR
else:
error_type = ErrorType.NETWORK_ERROR
log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
return _result(
success=False, return_code=1,
error_type=error_type, error_message=message,
)
# -- seen-ledger (short-lived sessions) --------------------------------
def _seen_keys(self, source_id: int, keys: list[str]) -> set[str]:
"""Which of `keys` are already in the ledger for this source.
One short SELECT on its own session — opened and closed without any
network in between (the GETs happen after, in download_post).
"""
if not keys:
return set()
with self.session_factory() as session:
rows = session.execute(
select(PatreonSeenMedia.filehash).where(
PatreonSeenMedia.source_id == source_id,
PatreonSeenMedia.filehash.in_(keys),
)
).scalars().all()
return set(rows)
def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None:
"""Idempotent upsert of (filehash, post_id) ledger rows for a page.
ON CONFLICT DO NOTHING against the (source_id, filehash) UNIQUE so a
re-sighting — or a concurrent walk — is a harmless no-op
([[scalar_one_or_none-duplicates]]: never check-then-insert without the
DB constraint backing it). De-dup the batch locally first so a single
page can't present the same key twice to one INSERT.
"""
seen_local: set[str] = set()
values = []
for key, post_id in items:
if key in seen_local:
continue
seen_local.add(key)
values.append(
{"source_id": source_id, "filehash": key, "post_id": post_id}
)
if not values:
return
with self.session_factory() as session:
stmt = pg_insert(PatreonSeenMedia).values(values)
stmt = stmt.on_conflict_do_nothing(
constraint="uq_patreon_seen_media_source_id"
)
session.execute(stmt)
session.commit()
async def verify_patreon_credential(
url: str,
cookies_path: str | None,
overrides: dict | None,
) -> tuple[bool | None, str]:
"""Native Patreon credential probe — the verify counterpart to the ingester's
download path, sharing its campaign-id resolution. Resolves the campaign id
(override / id: URL / vanity) then does ONE authenticated `/api/posts` fetch
via PatreonClient.verify_auth. Returns the uniform `(ok, message)` contract
(True / False / None) so download_backends.verify_credential can treat it
interchangeably with the gallery-dl probe. No download, no DB.
"""
campaign_id, _ = await resolve_campaign_id_for_source(url, cookies_path, overrides)
if not campaign_id:
return None, (
"Couldn't resolve the Patreon campaign id from the source URL — "
"can't verify (cookies expired, or the creator moved/renamed?)."
)
client = PatreonClient(cookies_path)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, client.verify_auth, campaign_id)
+45
View File
@@ -19,12 +19,22 @@ import asyncio
import http.cookiejar
import logging
import os
import re
import requests
log = logging.getLogger(__name__)
_CAMPAIGNS_URL = "https://www.patreon.com/api/campaigns"
# A source URL of the form `.../id:<digits>` already carries the campaign id
# (no lookup needed). The vanity regex deliberately EXCLUDES the id: form so the
# two paths don't overlap.
_ID_URL_RE = re.compile(r"/id:(\d+)")
_VANITY_RE = re.compile(
r"^https?://(?:www\.)?patreon\.com/(?:c/)?(?!id:)([^/?#]+)",
re.IGNORECASE,
)
_USER_AGENT = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
@@ -100,3 +110,38 @@ async def resolve_campaign_id(
Never raises."""
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, _sync_lookup, vanity, cookies_path)
def extract_vanity(url: str) -> str | None:
"""The vanity slug from a Patreon creator URL, or None for an `id:` URL."""
m = _VANITY_RE.match(url or "")
return m.group(1) if m else None
async def resolve_campaign_id_for_source(
url: str,
cookies_path: str | None,
overrides: dict | None,
) -> tuple[str | None, str | None]:
"""Resolve a Patreon source to its campaign id — the single resolution path
shared by the download ingester and the credential-verify probe.
Order: cached `patreon_campaign_id` override → an `id:<digits>` URL → a
vanity lookup against the campaigns API. Returns
`(campaign_id, newly_resolved_id)`: `newly_resolved_id` is non-None ONLY when
a vanity lookup actually ran, so the caller knows to cache it on the source
(the override/id: paths needed no lookup). `(None, None)` when unresolvable.
Never raises.
"""
overrides = overrides or {}
cached = overrides.get("patreon_campaign_id")
if cached:
return cached, None
id_match = _ID_URL_RE.search(url or "")
if id_match:
return id_match.group(1), None
vanity = extract_vanity(url)
if vanity:
resolved = await resolve_campaign_id(vanity, cookies_path)
return resolved, resolved
return None, None
+89 -25
View File
@@ -64,6 +64,12 @@ class SourceRecord:
consecutive_failures: int
next_check_at: str | None
backfill_runs_remaining: int
# plan #693: derived from config_overrides for the UI badge.
backfill_state: str | None # "running" | "complete" | "stalled" | None (idle)
backfill_chunks: int
# plan #697: a running deep-walk that bypasses the Patreon seen-ledger
# (recovery) vs. a normal backfill. Lets the badge label it "Recovering".
backfill_bypass_seen: bool
def to_dict(self) -> dict:
return {
@@ -82,6 +88,9 @@ class SourceRecord:
"consecutive_failures": self.consecutive_failures,
"next_check_at": self.next_check_at,
"backfill_runs_remaining": self.backfill_runs_remaining,
"backfill_state": self.backfill_state,
"backfill_chunks": self.backfill_chunks,
"backfill_bypass_seen": self.backfill_bypass_seen,
}
@@ -89,10 +98,13 @@ class SourceRecord:
_EDITABLE = {"enabled", "url", "config_overrides", "check_interval_override", "platform"}
# Plan #544 follow-up: newly created enabled sources pre-arm backfill so
# their first N polls walk gallery-dl's full post history with the longer
# timeout (matches the manual "Deep scan" button's default).
NEW_SOURCE_BACKFILL_RUNS = 3
# Plan #693: backfill safety cap. "Start backfill" (and a newly created
# enabled source) arms a run-until-done walk; this caps how many time-boxed
# chunks it may spend before pausing as "stalled", so a pathological walk that
# never reaches the bottom can't run forever. Generous on purpose — at
# BACKFILL_CHUNK_SECONDS (600s) per chunk this is ~33h of cumulative walk, far
# beyond any real catalog; the cursor stall-guard is the real terminator.
BACKFILL_MAX_CHUNKS = 200
class SourceService:
@@ -135,6 +147,7 @@ class SourceService:
self, source: Source, artist: Artist, settings: ImportSettings,
) -> SourceRecord:
nxt = compute_next_check_at(source, artist, settings)
co = source.config_overrides or {}
return SourceRecord(
id=source.id,
artist_id=source.artist_id,
@@ -151,6 +164,9 @@ class SourceService:
consecutive_failures=source.consecutive_failures or 0,
next_check_at=nxt.isoformat() if nxt else None,
backfill_runs_remaining=source.backfill_runs_remaining or 0,
backfill_state=co.get("_backfill_state"),
backfill_chunks=int(co.get("_backfill_chunks", 0)),
backfill_bypass_seen=bool(co.get("_backfill_bypass_seen")),
)
async def _row_to_record(self, source: Source) -> SourceRecord:
@@ -212,16 +228,17 @@ class SourceService:
select(func.count(Source.id)).where(Source.artist_id == artist_id)
)).scalar_one()
# Plan #544 follow-up: a freshly added subscription has no archive
# yet, so the first few polls would walk the full post history in
# tick mode and trip exit:20 after ~20 contiguous archive hits —
# except there are none yet, so tick mode would walk forever and
# blow the wall-clock cap. Pre-arm backfill so the initial syncs
# use the longer timeout + skip:True walk. Tick mode resumes once
# the budget is spent or the queue drains.
# Disabled sources (incl. sidecar synthetics, url='sidecar:...')
# are never polled, so leave their counter at 0.
backfill_runs = NEW_SOURCE_BACKFILL_RUNS if enabled else 0
# Plan #693: a freshly added subscription has no archive yet, so it
# should walk its full post history once. Arm run-until-done backfill
# (state="running" + the chunk cap); the time-boxed chunks march to the
# bottom across ticks, then flip to "complete" and tick mode takes over.
# Disabled sources (incl. sidecar synthetics, url='sidecar:...') are
# never polled, so leave them idle.
if enabled:
config_overrides = {**(config_overrides or {}), "_backfill_state": "running"}
backfill_runs = BACKFILL_MAX_CHUNKS
else:
backfill_runs = 0
source = Source(
artist_id=artist_id, platform=platform, url=url,
enabled=enabled, config_overrides=config_overrides,
@@ -286,22 +303,69 @@ class SourceService:
await self.session.commit()
return await self._row_to_record(source)
async def set_backfill_runs(
self, source_id: int, runs: int,
) -> SourceRecord:
"""Plan #544: arm a source for backfill mode. The next `runs`
download runs will use gallery-dl's full-walk config (skip: True
+ 30-min timeout) instead of the catch-up default. Runs must be
1..10 — bigger is rejected to keep the operator from accidentally
setting a runaway budget."""
if not isinstance(runs, int) or runs < 1 or runs > 10:
raise ValueError("runs must be an integer in [1, 10]")
async def start_backfill(self, source_id: int) -> SourceRecord:
"""Plan #693: arm a run-until-done backfill. Sets state="running" and
the chunk cap; download runs then walk the full post history in
time-boxed chunks (skip:True + BACKFILL_CHUNK_SECONDS), resuming from
the cursor each chunk, until gallery-dl reaches the bottom (→ state
"complete") or the cap/stall-guard pauses it (→ "stalled"). Clears any
prior cursor/chunk/stall state so a re-start walks fresh from the top."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
source.backfill_runs_remaining = runs
co = dict(source.config_overrides or {})
co["_backfill_state"] = "running"
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def start_recovery(self, source_id: int) -> SourceRecord:
"""Plan #697: arm a RECOVERY walk — a backfill that bypasses the Patreon
seen-ledger so deliberately-dropped-and-deleted near-dups get re-fetched
and re-evaluated under the CURRENT pHash threshold (tier-2 disk still
spares files we kept). Reuses the entire #693 backfill state machine
(time-boxed chunks, cursor checkpoint, complete/stall lifecycle) plus the
`_backfill_bypass_seen` flag that flips download mode to recovery. Clears
any prior cursor/chunk/stall state so it walks fresh from the top. The
flag is cleared on completion (download_service) and on stop.
Recovery is Patreon-only (the seen-ledger is Patreon's); for other
platforms the flag is inert (download_service ignores it) and the walk
runs as a plain backfill. The UI gates the action to Patreon sources."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
co["_backfill_state"] = "running"
co["_backfill_bypass_seen"] = True
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
await self.session.commit()
return await self._row_to_record(source)
async def stop_backfill(self, source_id: int) -> SourceRecord:
"""Plan #693: cancel an in-progress backfill — back to idle/tick mode.
Clears the running state + cursor/chunk/stall bookkeeping."""
source = (await self.session.execute(
select(Source).where(Source.id == source_id)
)).scalar_one_or_none()
if source is None:
raise LookupError(f"source id={source_id} not found")
co = dict(source.config_overrides or {})
for k in ("_backfill_state", "_backfill_cursor", "_backfill_cursor_stalls",
"_backfill_chunks", "_backfill_bypass_seen"):
co.pop(k, None)
source.config_overrides = co
source.backfill_runs_remaining = 0
await self.session.commit()
return await self._row_to_record(source)
+70
View File
@@ -280,6 +280,69 @@ class TagService:
await self.session.flush()
return tag
async def set_fandom(
self, tag_id: int, fandom_id: int | None, *, merge: bool = False
) -> Tag:
"""Set / change / clear a character tag's fandom.
Raises TagValidationError unless the tag is a character and fandom_id
(when given) references a fandom tag. If the change would collide with
an existing character of the same name in the TARGET fandom, raises
TagMergeConflict (the API turns that into a 409 merge hint) — unless
merge=True, in which case this tag is merged INTO that existing
character (a deliberate cross-fandom merge) and the surviving target
is returned. Passing fandom_id=None clears the fandom.
"""
tag = await self.session.get(Tag, tag_id)
if tag is None:
raise TagValidationError(f"Tag {tag_id} not found")
if tag.kind != TagKind.character:
raise TagValidationError("Only character tags can have a fandom")
if fandom_id is not None:
fandom = await self.session.get(Tag, fandom_id)
if fandom is None or fandom.kind != TagKind.fandom:
raise TagValidationError(
f"fandom_id {fandom_id} does not reference a fandom tag"
)
if fandom_id == tag.fandom_id:
return tag
# Collision: another character with the same name already lives in the
# target fandom. Mirrors rename's (name, kind, fandom_id) uniqueness.
clash_stmt = (
select(Tag)
.where(Tag.name == tag.name)
.where(Tag.kind == TagKind.character)
.where(
Tag.fandom_id.is_(None)
if fandom_id is None
else Tag.fandom_id == fandom_id
)
.where(Tag.id != tag_id)
)
clash = (await self.session.execute(clash_stmt)).scalar_one_or_none()
if clash is not None:
if not merge:
source_image_count = await self.session.scalar(
select(func.count())
.select_from(image_tag)
.where(image_tag.c.tag_id == tag_id)
)
will_alias = await self._keep_as_alias(tag_id)
raise TagMergeConflict(
f"A character named {tag.name!r} already exists in that fandom",
target_id=clash.id,
target_name=clash.name,
source_image_count=int(source_image_count or 0),
will_alias=will_alias,
)
await self._do_merge(tag, clash)
return clash
tag.fandom_id = fandom_id
await self.session.flush()
return tag
async def merge(self, source_id: int, target_id: int) -> MergeResult:
"""Transactionally repoint every FK from source→target, optionally
keep source's name as a tagger alias, delete source. Atomic: any
@@ -298,7 +361,14 @@ class TagService:
raise TagValidationError(
"Tags must be the same kind and fandom to merge"
)
return await self._do_merge(source, target)
async def _do_merge(self, source: Tag, target: Tag) -> MergeResult:
"""Repoint every FK source→target, optionally keep source's name as a
tagger alias, delete source. NO kind/fandom validation — callers that
need it (public merge()) validate first; set_fandom's collision
resolution calls this directly for a deliberate CROSS-fandom merge."""
source_id, target_id = source.id, target.id
keep_as_alias = await self._keep_as_alias(source_id)
source_name = source.name
source_kind = source.kind
+10 -1
View File
@@ -10,6 +10,7 @@ disposes it (``await engine.dispose()``) when its loop ends.
"""
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.pool import NullPool
from ..config import get_config
@@ -17,5 +18,13 @@ from ..config import get_config
def async_session_factory():
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
cfg = get_config()
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
# NullPool: this engine lives for ONE task (created + disposed per
# asyncio.run loop), so intra-task connection pooling buys nothing and
# actively bit us — download_source releases its phase-1 connection
# before a multi-minute gallery-dl subprocess, and a *pooled* idle
# connection would be reaped by the server and handed back dead to
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
# opens a fresh real connection on each checkout, so phase 3 always
# reconnects clean; pre_ping is then redundant.
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
+7
View File
@@ -34,3 +34,10 @@ def sync_session_factory():
)
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
return _SESSIONMAKER
def get_sync_engine():
"""The process-wide sync Engine — for raw work that needs a connection
directly (e.g. AUTOCOMMIT VACUUM, which can't run inside a transaction)."""
sync_session_factory() # ensure _ENGINE is initialized
return _ENGINE
+99 -4
View File
@@ -1,12 +1,17 @@
"""download_source Celery task — runs DownloadService for one source."""
import asyncio
import logging
from datetime import UTC, datetime
from pathlib import Path
from celery.exceptions import SoftTimeLimitExceeded
from sqlalchemy import select
from sqlalchemy.exc import DBAPIError, OperationalError
from sqlalchemy.orm import Session as SyncSession
from ..celery_app import celery
from ..models import ImportSettings
from ..models import DownloadEvent, ImportSettings, Source
from ..services.credential_crypto import CredentialCrypto
from ..services.credential_service import CredentialService
from ..services.download_service import DownloadService
@@ -16,9 +21,79 @@ from ..services.thumbnailer import Thumbnailer
from ._async_session import async_session_factory
from .import_file import _sync_session_factory
log = logging.getLogger(__name__)
IMAGES_ROOT = Path("/images")
_KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
# Celery time budget for one download_source run. The ceiling that
# governs *clean* teardown is the SOFT limit: it raises a catchable
# SoftTimeLimitExceeded in-process, whereas the HARD limit SIGKILLs the
# worker (no chance to finalize). Both gallery-dl subprocess budgets
# (gallery_dl.py: _DEFAULT_GDL_TIMEOUT_SECONDS=870 tick,
# BACKFILL_CHUNK_SECONDS=600 per backfill chunk, plan #693) MUST sit below the soft limit
# so subprocess.run raises its own TimeoutExpired first — that path
# captures partial stdout/stderr and finalizes the DownloadEvent. soft is
# max-subprocess (1170) + ~180s phase-3 persist headroom; hard is soft +
# 150s SIGKILL backstop. Audit 2026-06-03 (Anduo #39912): the old
# soft=900 sat BELOW the 1170 backfill budget, so SoftTimeLimitExceeded
# preempted TimeoutExpired and the event stranded empty. The recovery
# sweep's DOWNLOAD_STALL_THRESHOLD_MINUTES (30 min) still trails the new
# 25-min hard kill by 5 min, so it stays a true backstop. Invariant
# guarded by test_timeout_ladder_keeps_subprocess_budgets_under_soft_limit.
DOWNLOAD_SOFT_TIME_LIMIT = 1350
DOWNLOAD_HARD_TIME_LIMIT = 1500
def _finalize_soft_limited(session: SyncSession, source_id: int) -> None:
"""Defense in depth for the soft-time-limit kill path.
A SoftTimeLimitExceeded unwinds download_source before phase 3 can
finalize the DownloadEvent, leaving it 'running' until the recovery
sweep stamps a context-free "stranded" error 30 min later — AND
leaving backfill_runs_remaining undecremented so the source re-runs
and re-strands every tick (Anduo #39912, 2026-06-03). Flip the
in-flight event to error with a real reason, mirror phase 3's
source-health write, and decrement any backfill budget so a
chronically-slow source self-heals back to tick mode.
The caller owns the commit. All mutations are gated on actually
finding a running event, so a benign late soft-limit (phase 3 already
committed) is a no-op.
"""
now = datetime.now(UTC)
ev = session.execute(
select(DownloadEvent)
.where(DownloadEvent.source_id == source_id)
.where(DownloadEvent.status == "running")
.order_by(DownloadEvent.id.desc())
.limit(1)
).scalar_one_or_none()
if ev is None:
return
ev.status = "error"
ev.finished_at = now
ev.error = (
f"killed by Celery soft time limit ({DOWNLOAD_SOFT_TIME_LIMIT}s) "
"before the gallery-dl subprocess returned — the run exceeded its "
"budget and its stdout/stderr were lost with the worker thread. "
"If this recurs, the source is too large for one run; the backfill "
"budget was decremented so the next tick walks less."
)
ev.metadata_ = {
**(ev.metadata_ or {}),
"error_type": "timeout",
"soft_time_limited": True,
}
src = session.get(Source, source_id)
if src is not None:
src.consecutive_failures = (src.consecutive_failures or 0) + 1
src.last_error = "soft time limit exceeded"
src.error_type = "timeout"
src.last_checked_at = now
if (src.backfill_runs_remaining or 0) > 0:
src.backfill_runs_remaining = max(0, src.backfill_runs_remaining - 1)
@celery.task(
name="backend.app.tasks.download.download_source",
@@ -29,8 +104,8 @@ _KEY_PATH = IMAGES_ROOT / "secrets" / "credential_key.b64"
retry_backoff_max=120,
retry_jitter=True,
max_retries=3,
soft_time_limit=900,
time_limit=1200,
soft_time_limit=DOWNLOAD_SOFT_TIME_LIMIT,
time_limit=DOWNLOAD_HARD_TIME_LIMIT,
)
def download_source(self, source_id: int) -> int:
"""Returns the DownloadEvent.id."""
@@ -68,9 +143,29 @@ def download_source(self, source_id: int) -> int:
gdl=gdl,
importer=importer,
cred_service=cred_service,
# The native Patreon ingester opens its own short-lived
# sync sessions for the seen-ledger (never held across
# the walk). Same factory the importer's sync session
# comes from — a different DB connection per checkout.
sync_session_factory=SyncFactory,
)
return await svc.download_source(source_id)
finally:
await async_engine.dispose()
return asyncio.run(_run())
try:
return asyncio.run(_run())
except SoftTimeLimitExceeded:
# phase 3 never ran — salvage the in-flight event so the operator
# sees a real reason instead of the recovery sweep's generic
# "stranded" 30 min later (Anduo #39912). Best-effort: a failure
# here must not mask the timeout. Re-raise so Celery + the
# task_run signal handler still record the kill.
try:
SyncFactory = _sync_session_factory()
with SyncFactory() as session:
_finalize_soft_limited(session, source_id)
session.commit()
except Exception: # noqa: BLE001 — cleanup must not swallow the kill
log.exception("soft-limit finalize failed for source %s", source_id)
raise
+41 -4
View File
@@ -22,10 +22,27 @@ from ..models import (
TaskRun,
)
from ..utils.phash import compute_phash
from ._sync_engine import get_sync_engine
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
# High-churn tables whose dead-tuple bloat matters: the TABLESAMPLE showcase
# reads physical blocks (bloat slows it directly), and the periodic
# prune/backfill/recovery tasks generate dead tuples faster than autovacuum
# always keeps up with. VACUUM reclaims them; ANALYZE refreshes planner stats.
# Allowlist ONLY — names are interpolated into VACUUM, so they must never come
# from request input.
VACUUM_TABLES = (
"image_record",
"image_provenance",
"post_attachment",
"download_event",
"task_run",
"import_task",
"import_batch",
)
STUCK_THRESHOLD_MINUTES = 5
# Archive ImportTasks run the per-member pipeline inline for every
# member (import_archive_file: soft=30min/hard=35min). The ImportTask
@@ -46,9 +63,12 @@ MAX_RECOVERY_ATTEMPTS = 3
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
# DownloadEvent (pending|running) recovery threshold. download_source has
# time_limit=1200s (20 min); 30 min is 10 min past that, so a legitimately-
# running task is never killed by the sweep. Operator-confirmed 2026-05-29
# after 43 sources stranded at "last check never" by the in-flight guard.
# time_limit=1500s (25 min, DOWNLOAD_HARD_TIME_LIMIT); 30 min is 5 min past
# that, so a legitimately-running task is hard-killed before the sweep ever
# touches it — the sweep only catches events whose worker died without
# finalizing. Operator-confirmed 2026-05-29 after 43 sources stranded at
# "last check never" by the in-flight guard; budget bumped 2026-06-03 with
# the soft/hard limit raise (Anduo #39912).
DOWNLOAD_STALL_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7
@@ -535,7 +555,7 @@ def recover_stalled_download_events() -> int:
tasks.scan._tick_due_sources_async) inserts DownloadEvent(status='pending')
and fires download_source.delay(). If that task dies before finalizing the
event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind
on the 1200s hard time_limit — the event stays in-flight forever. The next
on the 1500s hard time_limit — the event stays in-flight forever. The next
tick then skips that source because of the in-flight guard (scan.py:168)
and Source.last_checked_at never updates; the operator sees "last check
never" in the Subscriptions health column, permanently.
@@ -758,3 +778,20 @@ def cleanup_old_download_events() -> int:
)
session.commit()
return result.rowcount or 0
@celery.task(name="backend.app.tasks.maintenance.vacuum_analyze")
def vacuum_analyze() -> dict:
"""Periodic VACUUM (ANALYZE) over the high-churn tables (VACUUM_TABLES) to
reclaim dead-tuple bloat and refresh planner statistics. VACUUM cannot run
inside a transaction block, so it runs on an AUTOCOMMIT connection.
Scheduled weekly; also operator-triggerable from Settings → Maintenance.
"""
engine = get_sync_engine()
done = []
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
for table in VACUUM_TABLES:
conn.exec_driver_sql(f"VACUUM (ANALYZE) {table}")
done.append(table)
log.info("vacuum_analyze complete: %s", done)
return {"vacuumed": done}
+46 -4
View File
@@ -11,6 +11,8 @@
<PipelineStatusChip />
</div>
<!-- Desktop: inline links, centered. Hidden on mobile (see media query),
where they fold into the hamburger menu on the right. -->
<nav class="fc-links">
<RouterLink
v-for="r in navRoutes"
@@ -20,9 +22,31 @@
>{{ r.meta.title }}</RouterLink>
</nav>
<!-- Per-view contextual actions teleport here (Showcase: Shuffle,
Gallery: Select). TopNav owns the slot, not its contents. -->
<div id="fc-nav-actions" class="fc-nav-actions" />
<div class="fc-nav-right">
<!-- Per-view contextual actions teleport here (Showcase: Shuffle,
Gallery: Select). TopNav owns the slot, not its contents. -->
<div id="fc-nav-actions" class="fc-nav-actions" />
<!-- Mobile nav: the link row collides with brand + actions below ~768px
(7+ links in one flex row), so collapse it into a menu. -->
<v-menu location="bottom end">
<template #activator="{ props: menuProps }">
<v-btn
v-bind="menuProps"
icon="mdi-menu" variant="text" size="small"
class="fc-nav-burger" aria-label="Menu"
/>
</template>
<v-list density="compact" min-width="180">
<v-list-item
v-for="r in navRoutes"
:key="r.name"
:to="{ name: r.name }"
:title="r.meta.title"
/>
</v-list>
</v-menu>
</div>
</header>
</template>
@@ -137,7 +161,7 @@ const health = computed(() => {
align-items: center;
flex-shrink: 0;
}
.fc-nav-actions {
.fc-nav-right {
flex: 1 1 0;
min-width: 0;
display: flex;
@@ -145,4 +169,22 @@ const health = computed(() => {
justify-content: flex-end;
gap: 0.5rem;
}
.fc-nav-actions {
display: flex;
align-items: center;
gap: 0.5rem;
}
/* The hamburger only exists on mobile; the inline links carry desktop. */
.fc-nav-burger { display: none; }
@media (max-width: 768px) {
.fc-topnav { gap: 0.5rem; padding: 0.6rem 0.75rem; }
/* Fold the link row into the hamburger menu. */
.fc-links { display: none; }
.fc-nav-burger { display: inline-flex; }
}
@media (max-width: 480px) {
/* Reclaim width on the smallest phones — the glyph alone still brands. */
.fc-brand__text { display: none; }
}
</style>
@@ -112,6 +112,16 @@ onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.single_color_threshold
tolerance.value = store.defaults.single_color_tolerance
// Reconnect to this rule's latest run so a scan started before navigating
// away keeps showing progress / its result on return (the scan itself runs
// backend-side regardless).
try {
const latest = await store.latestAuditForRule('single_color')
if (latest) {
audit.value = latest
if (latest.status === 'running') startPoll(latest.id)
}
} catch { /* non-fatal — card still works for a fresh scan */ }
})
onUnmounted(() => stopPoll())
@@ -97,6 +97,16 @@ let pollTimer = null
onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.transparency_threshold
// Reconnect to this rule's latest run so a scan started before navigating
// away keeps showing progress / its result on return (the scan itself runs
// backend-side regardless).
try {
const latest = await store.latestAuditForRule('transparency')
if (latest) {
audit.value = latest
if (latest.status === 'running') startPoll(latest.id)
}
} catch { /* non-fatal — card still works for a fresh scan */ }
})
onUnmounted(() => stopPoll())
@@ -8,6 +8,15 @@
<div v-if="card.preview_thumbnails.length === 0" class="fc-artistcard__noimg">
No preview
</div>
<!-- Accent pill in the corner when this artist has content imported
since the operator last opened their detail view. Caps at 99+
to keep the layout compact; the actual count appears in the
banner inside ArtistView. -->
<span
v-if="(card.unseen_count || 0) > 0"
class="fc-artistcard__unseen"
:aria-label="`${card.unseen_count} new since last visit`"
>+{{ card.unseen_count > 99 ? '99+' : card.unseen_count }}</span>
</div>
<v-card-text class="fc-artistcard__body">
<div class="fc-artistcard__name">{{ card.name }}</div>
@@ -37,6 +46,7 @@ function onCardClick() {
<style scoped>
.fc-artistcard { cursor: pointer; }
.fc-artistcard__previews {
position: relative;
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
@@ -45,6 +55,19 @@ function onCardClick() {
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-artistcard__unseen {
position: absolute;
top: 6px; right: 6px;
display: inline-flex; align-items: center;
padding: 2px 8px;
font-size: 11px; font-weight: 700; letter-spacing: 0.02em;
font-variant-numeric: tabular-nums;
color: rgb(var(--v-theme-on-accent, 0, 0, 0));
background: rgb(var(--v-theme-accent));
border-radius: 999px;
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
pointer-events: none;
}
.fc-artistcard__previews img {
display: block;
width: 100%; height: 100%;
@@ -6,7 +6,6 @@
v-for="item in col" :key="item.id"
class="fc-masonry__item"
:class="{ 'fc-masonry__item--anim': shouldAnimate(item) }"
:style="itemStyle(item)"
type="button"
@click="$emit('open', item.id)"
>
@@ -66,12 +65,6 @@ function shouldAnimate(item) {
return idx !== undefined && idx >= props.animateFromIndex
}
function itemStyle(item) {
if (!shouldAnimate(item)) return {}
const idx = idxById.value.get(item.id) - props.animateFromIndex
return { '--stagger-index': idx }
}
function aspectStyle(item) {
const w = Number(item.width)
const h = Number(item.height)
@@ -117,12 +110,15 @@ useInfiniteScroll(sentinelEl, () => {
.fc-masonry__end { text-align: center; padding: 32px 0; }
/* Cascade entry: each tile flips up out of a backward tilt and settles
into place, one at a time — more pronounced than a plain fade so the
showcase reads as an "experience" (operator-flagged 2026-05-28). The
`both` fill holds the hidden/tilted 0% state until each tile's staggered
turn; the cubic-bezier overshoots slightly past flat then settles.
Honors prefers-reduced-motion. Tunables: tilt (-28deg), stagger (70ms),
duration (0.6s). */
into place — more pronounced than a plain fade so the showcase reads as an
"experience" (operator-flagged 2026-05-28). The reveal is paced entirely by
the store, which pushes one fully-decoded item at a time (showcase.js); each
tile therefore animates the instant it mounts, with NO per-index CSS delay —
the old `animation-delay: index×70ms` compounded on top of the insert
cadence and made the cascade drag and desync as it grew (operator-flagged
2026-06-04). The `both` fill holds the hidden/tilted 0% state until mount;
the cubic-bezier overshoots slightly past flat then settles. Honors
prefers-reduced-motion. Tunables: tilt (-28deg), duration (0.6s). */
@keyframes fc-masonry-item-in {
0% {
opacity: 0;
@@ -138,7 +134,6 @@ useInfiniteScroll(sentinelEl, () => {
transform-origin: center top;
backface-visibility: hidden;
animation: fc-masonry-item-in 0.6s cubic-bezier(0.34, 1.45, 0.64, 1) both;
animation-delay: calc(var(--stagger-index, 0) * 70ms);
}
@media (prefers-reduced-motion: reduce) {
.fc-masonry__item--anim {
@@ -55,6 +55,12 @@
/>
</template>
<v-list density="compact">
<v-list-item
v-if="card.kind === 'character'"
title="Set fandom…"
prepend-icon="mdi-book-open-page-variant"
@click="$emit('set-fandom', card)"
/>
<v-list-item
title="Merge with…"
prepend-icon="mdi-call-merge"
@@ -78,7 +84,9 @@
import { ref } from 'vue'
const props = defineProps({ card: { type: Object, required: true } })
const emit = defineEmits(['open', 'rename', 'manage', 'read', 'merge-with', 'delete'])
const emit = defineEmits([
'open', 'rename', 'manage', 'read', 'merge-with', 'delete', 'set-fandom',
])
const editing = ref(false)
const draft = ref('')
@@ -186,7 +186,9 @@ async function onDeleteConfirm(token) {
<style scoped>
.fc-bulk-panel {
position: fixed; top: 0; right: 0;
width: 320px; height: 100vh; z-index: 1100;
/* min(320px, 90vw): on phones the panel never swallows the whole screen —
leaves a sliver of the gallery visible behind it. */
width: min(320px, 90vw); height: 100vh; z-index: 1100;
background: rgb(var(--v-theme-surface));
border-left: 1px solid rgb(var(--v-theme-surface-light));
box-shadow: -2px 0 16px rgba(0, 0, 0, 0.4);
@@ -0,0 +1,153 @@
<template>
<div class="fc-facets">
<v-progress-linear
v-if="store.facetsLoading" indeterminate color="accent"
class="fc-facets__bar" height="2"
/>
<div class="fc-facets__group">
<span class="fc-facets__label">Platform</span>
<div class="fc-facets__chips">
<v-chip
v-for="p in platformOptions" :key="p.key"
size="small" label
:variant="p.key === store.filter.platform ? 'flat' : 'tonal'"
:color="p.key === store.filter.platform ? 'accent' : undefined"
@click="selectPlatform(p.key)"
>{{ p.label }}<span class="fc-facets__count">{{ p.count }}</span></v-chip>
<span v-if="!platformOptions.length" class="fc-facets__empty">none</span>
</div>
</div>
<div class="fc-facets__group">
<span class="fc-facets__label">Curation</span>
<div class="fc-facets__chips">
<v-chip
size="small" label
:variant="store.filter.untagged ? 'flat' : 'tonal'"
:color="store.filter.untagged ? 'accent' : undefined"
@click="toggleFlag('untagged')"
>Untagged<span class="fc-facets__count">{{ facetCount('untagged') }}</span></v-chip>
<v-chip
size="small" label
:variant="store.filter.no_artist ? 'flat' : 'tonal'"
:color="store.filter.no_artist ? 'accent' : undefined"
@click="toggleFlag('no_artist')"
>No artist<span class="fc-facets__count">{{ facetCount('no_artist') }}</span></v-chip>
</div>
</div>
<div class="fc-facets__group">
<span class="fc-facets__label">Date</span>
<v-text-field
type="date" density="compact" hide-details variant="outlined"
:model-value="store.filter.date_from" :min="dateMin" :max="dateMax"
class="fc-facets__date"
@update:model-value="setDate('date_from', $event)"
/>
<span class="fc-facets__dash"></span>
<v-text-field
type="date" density="compact" hide-details variant="outlined"
:model-value="store.filter.date_to" :min="dateMin" :max="dateMax"
class="fc-facets__date"
@update:model-value="setDate('date_to', $event)"
/>
</div>
</div>
</template>
<script setup>
import { computed, onBeforeUnmount, onMounted, watch } from 'vue'
import { useRouter } from 'vue-router'
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
// Mirrors backend gallery_service.UNSOURCED_PLATFORM — the sentinel that
// selects filesystem-imported content (the null/"no platform" bucket).
const UNSOURCED = '__unsourced__'
const store = useGalleryStore()
const router = useRouter()
const platformOptions = computed(() =>
(store.facets?.platforms || []).map((p) => ({
key: p.value === null ? UNSOURCED : p.value,
label: p.value === null ? 'No platform' : p.value,
count: p.count,
}))
)
function facetCount(flag) {
const v = store.facets?.[flag]
return v == null ? '' : v
}
const dateMin = computed(() => (store.facets?.date_min || '').slice(0, 10) || undefined)
const dateMax = computed(() => (store.facets?.date_max || '').slice(0, 10) || undefined)
// Single write path, shared format with the bar: clone → patch → URL. The
// route watcher in GalleryView reloads the store (and our filter-watch below
// refetches the facet counts for the new scope).
function pushPatch(patch) {
const n = cloneFilter(store.filter)
Object.assign(n, patch)
router.push({ name: 'gallery', query: filterToQuery(n) })
}
function selectPlatform(key) {
// Single-select, click-active-to-clear (the platform param is one value).
pushPatch({ platform: store.filter.platform === key ? null : key })
}
function toggleFlag(name) {
pushPatch({ [name]: !store.filter[name] })
}
function setDate(field, val) {
pushPatch({ [field]: val || null })
}
// Fetch when the panel opens; refetch (debounced) whenever the active filter
// changes so the live counts track the current scope. Never fires on scroll —
// the panel is the only caller of loadFacets.
onMounted(() => store.loadFacets())
let debounce = null
watch(() => store.filter, () => {
if (debounce) clearTimeout(debounce)
debounce = setTimeout(() => store.loadFacets(), 250)
})
onBeforeUnmount(() => { if (debounce) clearTimeout(debounce) })
</script>
<style scoped>
.fc-facets {
position: relative;
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 16px;
padding: 4px 12px 12px;
}
.fc-facets__bar { position: absolute; top: 0; left: 0; right: 0; }
.fc-facets__group { display: flex; align-items: center; gap: 8px; }
.fc-facets__label {
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-facets__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
/* The count rides inside the chip as a dimmer trailing number. */
.fc-facets__count {
margin-left: 6px;
opacity: 0.7;
font-variant-numeric: tabular-nums;
font-size: 0.78em;
}
.fc-facets__empty { color: rgb(var(--v-theme-on-surface-variant)); font-size: 0.8rem; }
.fc-facets__date { max-width: 160px; }
.fc-facets__dash { color: rgb(var(--v-theme-on-surface-variant)); }
/* Phones: let each facet group wrap and the side-by-side date inputs grow to
full width so they don't overflow a ~360px viewport. */
@media (max-width: 480px) {
.fc-facets__group { flex-wrap: wrap; }
.fc-facets__date { max-width: none; flex: 1 1 9rem; }
}
</style>
@@ -0,0 +1,269 @@
<template>
<div class="fc-filterbar-wrap">
<div class="fc-filterbar">
<v-autocomplete
v-model="selected"
:items="searchItems"
:loading="searchLoading"
item-title="name" item-value="value"
no-filter clearable hide-details density="compact" variant="outlined"
placeholder="Filter by tag or artist…"
prepend-inner-icon="mdi-filter-variant"
class="fc-filterbar__search"
@update:search="onSearch"
@update:model-value="onPick"
>
<template #item="{ props: itemProps, item }">
<v-list-item v-bind="itemProps" :title="item.raw.name">
<template #prepend>
<v-icon size="small">{{ iconFor(item.raw) }}</v-icon>
</template>
<template #subtitle>
{{ item.raw.kind === 'artist' ? 'artist'
: (item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind) }}
</template>
</v-list-item>
</template>
</v-autocomplete>
<div class="fc-filterbar__chips">
<v-chip
v-for="id in store.filter.tag_ids" :key="`t${id}`"
size="small" closable :color="chipColor(id)" variant="tonal"
@click:close="removeTag(id)"
>{{ store.tagLabels[id] || `#${id}` }}</v-chip>
<v-chip
v-if="store.filter.artist_id"
size="small" closable color="accent" variant="tonal"
prepend-icon="mdi-account"
@click:close="clearArtist"
>{{ store.artistLabel || `Artist #${store.filter.artist_id}` }}</v-chip>
<v-chip
v-if="store.filter.similar_to"
size="small" closable color="accent" variant="tonal"
prepend-icon="mdi-image-multiple"
@click:close="clearSimilar"
>Similar to #{{ store.filter.similar_to }}</v-chip>
</div>
<v-spacer />
<v-btn-toggle
:model-value="store.filter.media_type ?? 'all'"
density="compact" mandatory variant="outlined" divided
@update:model-value="(v) => setMedia(v === 'all' ? null : v)"
>
<v-btn value="all" size="small">All</v-btn>
<v-btn value="image" size="small">Images</v-btn>
<v-btn value="video" size="small">Videos</v-btn>
</v-btn-toggle>
<!-- Sort is meaningless in similar-mode (results are distance-ranked). -->
<v-select
v-if="!store.filter.similar_to"
:model-value="store.filter.sort"
:items="SORTS"
density="compact" hide-details variant="outlined"
class="fc-filterbar__sort"
@update:model-value="setSort"
/>
<v-btn
:color="refineOpen ? 'accent' : undefined"
:variant="refineOpen || hasRefineFilters ? 'tonal' : 'text'"
size="small"
:append-icon="refineOpen ? 'mdi-chevron-up' : 'mdi-chevron-down'"
@click="toggleRefine"
>Refine{{ refineCount ? ` (${refineCount})` : '' }}</v-btn>
<v-btn
v-if="hasActiveFilters" variant="text" size="small"
prepend-icon="mdi-close" @click="clearAll"
>Clear</v-btn>
</div>
<GalleryFacetPanel v-if="refineOpen" />
</div>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { useRouter } from 'vue-router'
import { useApi } from '../../composables/useApi.js'
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
import { useTagStore } from '../../stores/tags.js'
import GalleryFacetPanel from './GalleryFacetPanel.vue'
const store = useGalleryStore()
const tagStore = useTagStore()
const api = useApi()
const router = useRouter()
const SORTS = [
{ title: 'Newest first', value: 'newest' },
{ title: 'Oldest first', value: 'oldest' },
]
const selected = ref(null)
const searchItems = ref([])
const searchLoading = ref(false)
let debounce = null
// The faceted-refine sub-filters (platform / curation flags / date range).
const refineCount = computed(() => {
const f = store.filter
return (f.platform ? 1 : 0) + (f.untagged ? 1 : 0) + (f.no_artist ? 1 : 0)
+ (f.date_from ? 1 : 0) + (f.date_to ? 1 : 0)
})
const hasRefineFilters = computed(() => refineCount.value > 0)
const hasActiveFilters = computed(() =>
store.filter.tag_ids.length > 0 ||
store.filter.artist_id != null ||
store.filter.media_type != null ||
store.filter.sort !== 'newest' ||
store.filter.similar_to != null ||
hasRefineFilters.value
)
// Auto-open the panel when refine filters are present in the URL (deep-link /
// back button). The parent applies the query in its onMounted — after this
// child has set up — so watch for the transition rather than reading the
// initial (still-default) filter state.
const refineOpen = ref(false)
watch(hasRefineFilters, (v) => { if (v) refineOpen.value = true })
function toggleRefine() {
// The panel fetches facets itself on mount (and refetches on filter change);
// opening it is enough.
refineOpen.value = !refineOpen.value
}
function iconFor(raw) {
if (raw.kind === 'artist') return 'mdi-account'
return { character: 'mdi-account-circle', fandom: 'mdi-book-open-page-variant',
series: 'mdi-bookshelf' }[raw.kind] || 'mdi-tag'
}
function chipColor(id) {
// Tag chips use the same per-kind palette as the rest of the app; we only
// know the kind from the autocomplete pick, so fall back to a neutral tone.
return tagStore.colorFor(pickedKind.value[id] || 'general')
}
const pickedKind = ref({})
function onSearch(q) {
if (debounce) clearTimeout(debounce)
if (!q || !q.trim()) { searchItems.value = []; return }
debounce = setTimeout(async () => {
searchLoading.value = true
try {
const [tags, artists] = await Promise.all([
api.get('/api/tags/autocomplete', { params: { q, limit: 10 } }),
api.get('/api/artists/autocomplete', { params: { q, limit: 10 } }),
])
searchItems.value = [
...(artists || []).map((a) => ({
kind: 'artist', id: a.id, name: a.name, value: `artist:${a.id}`,
})),
...(tags || []).map((t) => ({
kind: t.kind, id: t.id, name: t.name, value: `tag:${t.id}`,
fandom_name: t.fandom_name,
})),
]
} catch {
searchItems.value = []
} finally {
searchLoading.value = false
}
}, 250)
}
function onPick(value) {
const item = searchItems.value.find((i) => i.value === value)
selected.value = null
searchItems.value = []
if (!item) return
if (item.kind === 'artist') {
store.noteArtistLabel(item.name)
pushFilter((n) => { n.artist_id = item.id })
} else {
store.noteTagLabel(item.id, item.name)
pickedKind.value = { ...pickedKind.value, [item.id]: item.kind }
pushFilter((n) => { if (!n.tag_ids.includes(item.id)) n.tag_ids.push(item.id) })
}
}
function removeTag(id) {
pushFilter((n) => { n.tag_ids = n.tag_ids.filter((t) => t !== id) })
}
function clearArtist() {
store.noteArtistLabel(null)
pushFilter((n) => { n.artist_id = null })
}
function clearSimilar() { pushFilter((n) => { n.similar_to = null }) }
function setMedia(m) { pushFilter((n) => { n.media_type = m }) }
function setSort(s) { pushFilter((n) => { n.sort = s }) }
function clearAll() { router.push({ name: 'gallery', query: {} }) }
// Single write path: clone the current filter, mutate, serialize to the URL.
// The route watcher in GalleryView applies it to the store and reloads.
// cloneFilter/filterToQuery are shared with GalleryFacetPanel so the refine
// sub-filters survive a bar push and vice versa.
function pushFilter(mutate) {
const n = cloneFilter(store.filter)
mutate(n)
router.push({ name: 'gallery', query: filterToQuery(n) })
}
</script>
<style scoped>
/* The whole chrome (bar row + expandable refine panel) is one sticky,
frosted block pinned directly under the 64px TopNav and continuous with it. */
.fc-filterbar-wrap {
position: sticky;
top: 64px;
z-index: 5;
/* Attach to the TopNav: cancel the v-container's top padding (pt-2 = 8px)
so the bar sits flush at 64px even at scroll 0 — without this it detaches
and a gap shows through when scrolled to the top. */
margin-top: -8px;
margin-bottom: 12px;
/* EXACT same gradiated obsidian (#14171A = 20,23,26) frost as the TopNav so
the two read as one continuous piece of chrome — images scroll visibly
under both. The nav's gradient fades to transparent at ITS bottom; this
bar re-darkens at its top, so a faint seam (the page/image showing through
the nav's transparent edge) separates them when scrolled to the very top,
while under-scroll they frost as one. */
background: linear-gradient(
to bottom,
rgba(20, 23, 26, 0.92) 0%,
rgba(20, 23, 26, 0.65) 60%,
rgba(20, 23, 26, 0) 100%
);
backdrop-filter: blur(2px);
-webkit-backdrop-filter: blur(2px);
}
.fc-filterbar {
display: flex;
align-items: center;
gap: 10px;
flex-wrap: wrap;
padding: 10px 12px;
}
/* The inputs/toggles float on the haze: still translucent, but a touch more
opaque than the bar/nav so the controls stay legible. */
.fc-filterbar-wrap :deep(.v-field),
.fc-filterbar-wrap :deep(.v-btn-group) {
background-color: rgba(20, 23, 26, 0.72);
}
.fc-filterbar__search { max-width: 320px; min-width: 200px; }
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
.fc-filterbar__sort { max-width: 150px; }
/* Phones: the search's 200px min-width jams the wrapping bar. Give search its
own full-width row and let sort grow; everything else wraps under it. */
@media (max-width: 600px) {
.fc-filterbar { gap: 8px; }
.fc-filterbar__search { min-width: 100%; max-width: none; }
.fc-filterbar__sort { max-width: none; flex: 1 1 auto; }
}
</style>
@@ -15,6 +15,19 @@
</div>
</template>
<!-- Similar-mode (and any non-date-grouped result set) returns no date
groups the results are ranked, not chronological. Render them as a
single flat list in their given order rather than nothing. -->
<div
v-if="!store.dateGroups.length && store.images.length"
class="fc-gallery-grid__items"
>
<GalleryItem
v-for="img in store.images"
:key="img.id" :image="img" @open="$emit('open', img.id)"
/>
</div>
<div v-if="store.loading" class="fc-gallery-grid__sentinel">
<v-progress-circular indeterminate color="accent" size="28" />
</div>
@@ -9,11 +9,16 @@
>
<div class="fc-gallery-item__media">
<img
v-if="!isVideo" :src="image.thumbnail_url" :alt="`Image ${image.id}`"
loading="lazy" @error="onThumbError"
v-if="!isVideo" ref="imgEl" :src="image.thumbnail_url"
:alt="`Image ${image.id}`" loading="lazy"
:class="{ 'is-loaded': loaded }"
@load="loaded = true" @error="onThumbError"
>
<div v-else class="fc-gallery-item__video-thumb">
<img :src="image.thumbnail_url" :alt="`Video ${image.id}`" loading="lazy">
<img
ref="imgEl" :src="image.thumbnail_url" :alt="`Video ${image.id}`"
loading="lazy" :class="{ 'is-loaded': loaded }" @load="loaded = true"
>
<v-icon class="fc-gallery-item__video-badge" icon="mdi-play-circle" />
</div>
<div v-if="thumbError" class="fc-gallery-item__placeholder">
@@ -38,7 +43,7 @@
</template>
<script setup>
import { computed, ref } from 'vue'
import { computed, onMounted, ref } from 'vue'
import { useGallerySelectionStore } from '../../stores/gallerySelection.js'
const props = defineProps({ image: { type: Object, required: true } })
@@ -46,6 +51,18 @@ const emit = defineEmits(['open'])
const sel = useGallerySelectionStore()
const thumbError = ref(false)
// Reveal each tile when ITS OWN thumbnail finishes loading (`@load`), not
// when its metadata batch lands — so tiles fade/flip in individually in
// load order instead of popping in together (operator-flagged 2026-06-04).
const loaded = ref(false)
const imgEl = ref(null)
onMounted(() => {
// A cached thumbnail can already be complete before @load binds; reflect
// that so the tile reveals instead of sitting invisible at opacity 0.
const el = imgEl.value
if (el && el.complete && el.naturalWidth > 0) loaded.value = true
})
const isVideo = computed(
() => props.image.mime && props.image.mime.startsWith('video/')
)
@@ -82,6 +99,30 @@ function onThumbError() { thumbError.value = true }
}
.fc-gallery-item__media img {
width: 100%; height: 100%; object-fit: cover; display: block;
opacity: 0;
}
/* Entrance: each thumbnail flips up out of a slight backward tilt and
settles into place, played WHEN ITS OWN image finishes loading (the
`is-loaded` class) rather than on a batch/timer — so tiles cascade in
natural load order instead of popping in together. Mirrors the showcase
MasonryGrid entrance styling (operator-flagged 2026-06-04). */
.fc-gallery-item__media img.is-loaded {
animation: fc-gallery-item-in 0.5s cubic-bezier(0.34, 1.45, 0.64, 1) both;
}
@keyframes fc-gallery-item-in {
0% {
opacity: 0;
transform: perspective(1000px) rotateX(-22deg) translateY(20px) scale(0.96);
}
55% { opacity: 1; }
100% {
opacity: 1;
transform: perspective(1000px) rotateX(0) translateY(0) scale(1);
}
}
@media (prefers-reduced-motion: reduce) {
.fc-gallery-item__media img { opacity: 1; }
.fc-gallery-item__media img.is-loaded { animation: none; }
}
.fc-gallery-item__video-thumb { position: relative; height: 100%; }
.fc-gallery-item__video-badge {
@@ -17,6 +17,10 @@
</div>
</v-card-text>
<v-card-actions>
<!-- Not every character belongs to a fandom (original characters,
unsorted, etc.). "No fandom" creates the character unassigned;
a fandom can still be set later from the chip's kebab menu. -->
<v-btn variant="text" @click="onNoFandom">No fandom</v-btn>
<v-spacer />
<v-btn @click="$emit('cancel')">Cancel</v-btn>
<v-btn :disabled="!selectedId" color="primary" rounded="pill" @click="onConfirm">Use this fandom</v-btn>
@@ -45,4 +49,9 @@ function onConfirm() {
const f = store.fandomCache.find(x => x.id === selectedId.value)
if (f) emit('confirm', f)
}
// Create the character with no fandom. Emits null so the caller knows this
// was a deliberate "unassigned", not a cancel.
function onNoFandom() {
emit('confirm', null)
}
</script>
@@ -0,0 +1,128 @@
<template>
<v-card>
<v-card-title class="text-body-1">Fandom for {{ tag.name }}</v-card-title>
<v-card-text>
<template v-if="!collision">
<v-autocomplete
v-model="selectedId"
:items="store.fandomCache"
:item-title="(f) => f.name" :item-value="(f) => f.id"
label="Fandom" clearable density="compact"
:hint="selectedId == null
? 'No fandom — the character will be unassigned.' : ''"
persistent-hint
/>
<v-divider class="my-3" />
<p class="text-caption mb-2">Or create a new fandom:</p>
<div class="d-flex" style="gap: 8px;">
<v-text-field
v-model="newName" placeholder="New fandom name"
density="compact" hide-details
@keydown.enter.prevent="onCreate"
/>
<v-btn
:disabled="!newName.trim() || busy" rounded="pill"
@click="onCreate"
>Create</v-btn>
</div>
<v-alert
v-if="error" type="error" variant="tonal" density="compact"
class="mt-3"
>{{ error }}</v-alert>
</template>
<template v-else>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
A character named “{{ tag.name }}” already exists in that fandom.
</v-alert>
<p class="text-body-2">
Merge this tag into “{{ collision.target.name }}”?
{{ collision.source_image_count }} image
association{{ collision.source_image_count === 1 ? '' : 's' }}
will move over and this tag will be deleted{{
collision.will_alias ? ' (its name kept as a tagger alias)' : '' }}.
</p>
</template>
</v-card-text>
<v-card-actions>
<v-spacer />
<template v-if="!collision">
<v-btn variant="text" :disabled="busy" @click="$emit('cancel')">
Cancel
</v-btn>
<v-btn
color="primary" rounded="pill" :loading="busy"
:disabled="selectedId === (tag.fandom_id ?? null)"
@click="onSave"
>Save</v-btn>
</template>
<template v-else>
<v-btn variant="text" :disabled="busy" @click="collision = null">
Back
</v-btn>
<v-btn
color="warning" variant="flat" rounded="pill" :loading="busy"
@click="onConfirmMerge"
>Merge</v-btn>
</template>
</v-card-actions>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { useTagStore } from '../../stores/tags.js'
const props = defineProps({ tag: { type: Object, required: true } })
const emit = defineEmits(['updated', 'cancel'])
const store = useTagStore()
const selectedId = ref(props.tag.fandom_id ?? null)
const newName = ref('')
const busy = ref(false)
const error = ref(null)
const collision = ref(null)
onMounted(() => { if (store.fandomCache.length === 0) store.loadFandoms() })
async function onCreate() {
const name = newName.value.trim()
if (!name) return
busy.value = true
error.value = null
try {
const f = await store.createFandom(name)
selectedId.value = f.id
newName.value = ''
} catch (e) {
error.value = e.message || String(e)
} finally {
busy.value = false
}
}
async function save(merge) {
busy.value = true
error.value = null
try {
const body = await store.setFandom(props.tag.id, selectedId.value, { merge })
emit('updated', body)
} catch (e) {
// 409 on first attempt → surface the merge confirmation; cross-fandom
// collisions can't go through the regular /merge endpoint, so the
// resolution is a second setFandom with merge: true.
if (!merge && e.status === 409 && e.body && e.body.target) {
collision.value = e.body
} else {
error.value = e.message || String(e)
collision.value = null
}
} finally {
busy.value = false
}
}
function onSave() { save(false) }
function onConfirmMerge() { save(true) }
</script>
+46 -12
View File
@@ -51,6 +51,9 @@
<aside v-if="modal.current" class="fc-viewer__side">
<ProvenancePanel />
<TagPanel />
<!-- Non-blocking: fetches its own similar set after the modal is up;
collapses silently if empty/slow/failed (see RelatedStrip). -->
<RelatedStrip />
</aside>
</div>
</div>
@@ -60,10 +63,12 @@
<script setup>
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
import { useModalStore } from '../../stores/modal.js'
import { arrowNavAllowed } from '../../utils/textEntry.js'
import ImageCanvas from './ImageCanvas.vue'
import VideoCanvas from './VideoCanvas.vue'
import TagPanel from './TagPanel.vue'
import ProvenancePanel from './ProvenancePanel.vue'
import RelatedStrip from './RelatedStrip.vue'
const emit = defineEmits(['close'])
@@ -105,11 +110,13 @@ function onKeyDown(ev) {
ev.preventDefault()
emit('close')
} else if (ev.key === 'ArrowLeft') {
if (isTextEntry(ev.target)) return
// Navigate unless the caret is in a non-empty text field (then let it move
// through the text). An empty tag-entry field still navigates.
if (!arrowNavAllowed(ev.target)) return
ev.preventDefault()
modal.goPrev()
} else if (ev.key === 'ArrowRight') {
if (isTextEntry(ev.target)) return
if (!arrowNavAllowed(ev.target)) return
ev.preventDefault()
modal.goNext()
}
@@ -135,17 +142,14 @@ watch(() => modal.currentImageId, async () => {
function nextFrame() {
return new Promise(resolve => requestAnimationFrame(resolve))
}
function isTextEntry(el) {
if (!el) return false
const tag = el.tagName
return tag === 'INPUT' || tag === 'TEXTAREA' || el.isContentEditable
}
</script>
<style scoped>
.fc-viewer {
position: fixed; inset: 0; z-index: 2000;
/* Single source of truth for the metadata side-panel width — the next
arrow offsets off it so it never overlaps the panel. */
--fc-side-w: 320px;
/* Obsidian haze (#14171A = 20,23,26) — same palette as TopNav,
mid-opacity + blur so the page behind shows through faintly. */
background: rgba(20, 23, 26, 0.65);
@@ -174,7 +178,12 @@ function isTextEntry(el) {
position: absolute; top: 72px; right: 16px; z-index: 3;
}
.fc-viewer__nav--prev { left: 16px; transform: translateY(-50%); }
.fc-viewer__nav--next { right: 16px; transform: translateY(-50%); }
/* Sit just inside the image area, clear of the metadata side panel —
not floating over it (operator-flagged 2026-06-04). */
.fc-viewer__nav--next {
right: calc(var(--fc-side-w) + 16px);
transform: translateY(-50%);
}
.fc-viewer__body {
flex: 1; display: flex; min-height: 0;
}
@@ -187,19 +196,44 @@ function isTextEntry(el) {
min-width: 0; min-height: 0;
}
.fc-viewer__side {
width: 320px; flex-shrink: 0;
width: var(--fc-side-w); flex-shrink: 0;
background: rgb(var(--v-theme-surface));
border-left: 1px solid rgb(var(--v-theme-surface-light));
overflow-y: auto;
}
@media (max-width: 900px) {
.fc-viewer__body { flex-direction: column; }
/* Stacked layout (operator-flagged 2026-06-05): pin the image pane and
its controls at the top while the metadata panel scrolls beneath it.
Previously the image + a 40vh panel shared one fixed viewport and the
controls could be pushed out of view; now the body scrolls and the
media is sticky, so the image + prev/next/close (and the integrity
badge) stay visible no matter how far the panel is scrolled. */
.fc-viewer__body {
flex-direction: column;
overflow-y: auto;
}
.fc-viewer__media {
flex: none;
height: 55vh;
position: sticky;
top: 0;
z-index: 1;
/* Opaque obsidian so the scrolling panel never bleeds through the
haze behind the pinned image. */
background: rgb(20, 23, 26);
}
.fc-viewer__side {
width: 100%;
max-height: 40vh;
max-height: none;
border-left: none;
border-top: 1px solid rgb(var(--v-theme-surface-light));
}
/* Re-center the prev/next arrows over the 55vh image band (their base
top:50% would land on the scrolling panel); next uses the full width
now that the panel is below, not beside. Close + integrity badge keep
their top:16px/72px — already within the pinned band. */
.fc-viewer__nav { top: 27.5vh; }
.fc-viewer__nav--next { right: 16px; }
}
</style>
@@ -0,0 +1,133 @@
<template>
<!-- Collapses entirely unless the source has an embedding AND we're loading
or have results — so a slow/empty/failed fetch leaves no trace and never
affects the modal. -->
<div v-if="show" class="fc-related">
<div class="fc-related__head">
<span class="fc-related__title">Related</span>
<v-btn
size="x-small" variant="text" color="accent"
:disabled="loading || !results.length"
@click="seeAll"
>See all similar</v-btn>
</div>
<div class="fc-related__row">
<template v-if="loading">
<div v-for="n in 6" :key="n" class="fc-related__skel" />
</template>
<button
v-for="img in results" :key="img.id"
class="fc-related__item" type="button"
@click="openImage(img.id)"
>
<img :src="img.thumbnail_url" :alt="`image ${img.id}`" loading="lazy" />
</button>
</div>
</div>
</template>
<script setup>
import { computed, onBeforeUnmount, ref, watch } from 'vue'
import { useRouter } from 'vue-router'
import { useApi } from '../../composables/useApi.js'
import { useModalStore } from '../../stores/modal.js'
// Deferred so the modal's main image gets network/decode priority the strip
// is a nice-to-have and must never block or slow the modal load.
const DEFER_MS = 200
const STRIP_LIMIT = 12
const api = useApi()
const router = useRouter()
const modal = useModalStore()
const results = ref([])
const loading = ref(false)
const hasEmbedding = computed(() => modal.current?.has_embedding === true)
const show = computed(() => hasEmbedding.value && (loading.value || results.value.length > 0))
let seq = 0
let timer = null
async function fetchSimilar(id) {
const mine = ++seq
loading.value = true
results.value = []
try {
const body = await api.get('/api/gallery/similar', {
params: { similar_to: id, limit: STRIP_LIMIT },
})
if (mine !== seq) return
results.value = body.images || []
} catch {
if (mine === seq) results.value = [] // quietly collapse on error
} finally {
if (mine === seq) loading.value = false
}
}
// Re-fetch whenever the modal lands on a new embedded image. modal.current is
// null while the next image loads, so this also clears the strip during
// prev/next nav and repopulates once the new payload arrives.
watch(
() => (hasEmbedding.value ? modal.current?.id : null),
(id) => {
if (timer) { clearTimeout(timer); timer = null }
seq++ // cancel any in-flight fetch
results.value = []
loading.value = false
if (!id) return
timer = setTimeout(() => fetchSimilar(id), DEFER_MS)
},
{ immediate: true },
)
onBeforeUnmount(() => { if (timer) clearTimeout(timer) })
function openImage(id) {
modal.open(id)
}
function seeAll() {
const id = modal.current?.id
if (!id) return
modal.close()
router.push({ name: 'gallery', query: { similar_to: String(id) } })
}
</script>
<style scoped>
.fc-related {
padding: 12px;
border-top: 1px solid rgb(var(--v-theme-surface-light));
}
.fc-related__head {
display: flex; align-items: center; justify-content: space-between;
margin-bottom: 8px;
}
.fc-related__title {
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-related__row {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 6px;
}
.fc-related__item {
display: block; padding: 0; border: 0; background: none;
cursor: pointer; border-radius: 4px; overflow: hidden;
aspect-ratio: 1; width: 100%;
}
.fc-related__item img {
width: 100%; height: 100%; object-fit: cover; display: block;
background: rgb(var(--v-theme-surface-light));
transition: transform 0.2s ease, filter 0.2s ease;
}
.fc-related__item:hover img { transform: scale(1.05); filter: brightness(1.1); }
.fc-related__skel {
aspect-ratio: 1; border-radius: 4px;
background: rgb(var(--v-theme-surface-light));
opacity: 0.5;
}
</style>
@@ -19,24 +19,25 @@
>
Accept
</v-btn>
<!-- Operator-flagged 2026-06-02: the kebab menu wasn't opening.
Wrapping in a <span @click.stop> matches the TagPanel chip
fix — even though there's no parent click capture here today,
the wrap is harmless and keeps both kebabs on the same
pattern. Click bubbles from the v-btn opens menu via
activator props bubble continues to span stopPropagation
halts it. -->
<span class="fc-suggestion__menu-wrap" @click.stop>
<v-menu>
<template #activator="{ props }">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
v-bind="props"
/>
</template>
<!-- Operator-flagged 2026-06-04: the kebab still wasn't opening. The
prior `#activator` + `v-bind="props"` path never toggled the menu
inside this teleported modal, while v-model-driven overlays (the
dialogs in this modal) work fine. So drive the menu explicitly:
the button toggles `menuOpen` with @click.stop (also shields any
parent), and `activator="parent"` anchors the menu for positioning
only — `:open-on-click="false"` keeps Vuetify's activator-click out
of it, so there's a single, reliable opener. -->
<span class="fc-suggestion__menu-wrap">
<v-btn
class="fc-suggestion__menu"
icon="mdi-dots-vertical" size="small"
variant="outlined" density="compact"
:aria-label="`More actions for ${suggestion.display_name}`"
@click.stop="menuOpen = !menuOpen"
/>
<v-menu
v-model="menuOpen" activator="parent" :open-on-click="false"
>
<v-list density="compact">
<v-list-item @click="$emit('alias', suggestion)">
<v-list-item-title>Treat as alias for…</v-list-item-title>
@@ -51,11 +52,12 @@
</template>
<script setup>
import { computed } from 'vue'
import { computed, ref } from 'vue'
const props = defineProps({ suggestion: { type: Object, required: true } })
defineEmits(['accept', 'alias', 'dismiss'])
const menuOpen = ref(false)
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
</script>
@@ -65,8 +65,17 @@ const store = useTagStore()
// Vuetify's v-text-field exposes .focus() on the component instance;
// nextTick waits for the modal's mount to finish so the inner <input>
// element exists.
//
// EXCEPTION — not on mobile (operator-flagged 2026-06-05): focusing the
// field pops the soft keyboard, which shrinks the visual viewport and
// shoves the pinned image + its nav/close controls out of view. 900px is
// the modal's stacked-layout breakpoint (ImageViewer.vue). matchMedia is
// safe on plain HTTP (not a secure-context API).
const inputRef = ref(null)
onMounted(() => { nextTick(() => inputRef.value?.focus?.()) })
onMounted(() => {
if (window.matchMedia?.('(max-width: 900px)')?.matches) return
nextTick(() => inputRef.value?.focus?.())
})
// Single text input; no kind dropdown. Client-side mirror of the
// backend's parse_kind_prefix lives below — kept in sync with
@@ -147,10 +156,14 @@ function onCreate () {
reset()
}
// fandom is null when the user picked "No fandom" — characters don't all
// belong to a fandom. The backend already accepts fandom_id: null for the
// character kind (tag.kind check + nullable fandom_id), and a fandom can be
// assigned later from the chip kebab's "Set fandom…".
function onFandomChosen (fandom) {
fandomDialog.value = false
emit('pick-new', {
name: pendingNewName, kind: 'character', fandom_id: fandom.id,
name: pendingNewName, kind: 'character', fandom_id: fandom ? fandom.id : null,
})
pendingNewName = null
reset()
+47 -14
View File
@@ -10,24 +10,33 @@
>
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
{{ tag.name }}<span v-if="tag.fandom_id"></span>
<!-- Operator-flagged 2026-06-02: the previous activator had
`@click.stop` directly on the v-icon, which silently
overrode Vuetify's onClick from `v-bind="mp"` — the menu
never opened. Now the v-icon receives the activator
onClick cleanly, and the wrapping span absorbs the
bubbled click so the chip's close button isn't tripped. -->
<span class="kebab-wrap" @click.stop>
<v-menu>
<template #activator="{ props: mp }">
<v-icon
v-bind="mp" size="x-small" class="ml-1"
icon="mdi-dots-vertical"
/>
</template>
<!-- Operator-flagged 2026-06-04: the `#activator` + `v-bind` menu
never opened inside this teleported modal. Drive it explicitly
instead (same mechanism as the dialogs below, which work): the
icon toggles `openTagId` with @click.stop (shielding the chip's
close button), and `activator="parent"` + `:open-on-click=false`
anchors the menu for positioning only. One tag's menu open at a
time, so a single id is enough. -->
<span class="kebab-wrap">
<v-icon
size="x-small" class="ml-1 kebab-icon"
icon="mdi-dots-vertical"
@click.stop="openTagId = openTagId === tag.id ? null : tag.id"
/>
<v-menu
:model-value="openTagId === tag.id"
activator="parent" :open-on-click="false"
@update:model-value="v => { if (!v) openTagId = null }"
>
<v-list density="compact">
<v-list-item @click="openRename(tag)">
<v-list-item-title>Rename…</v-list-item-title>
</v-list-item>
<v-list-item
v-if="tag.kind === 'character'" @click="openSetFandom(tag)"
>
<v-list-item-title>Set fandom…</v-list-item-title>
</v-list-item>
</v-list>
</v-menu>
</span>
@@ -57,6 +66,13 @@
@renamed="onRenamed" @cancel="renameDialog = false"
/>
</v-dialog>
<v-dialog v-model="fandomDialog" max-width="460">
<FandomSetDialog
v-if="fandomTarget" :tag="fandomTarget"
@updated="onFandomUpdated" @cancel="fandomDialog = false"
/>
</v-dialog>
</aside>
</template>
@@ -67,10 +83,14 @@ import { useTagStore } from '../../stores/tags.js'
import TagAutocomplete from './TagAutocomplete.vue'
import SuggestionsPanel from './SuggestionsPanel.vue'
import TagRenameDialog from './TagRenameDialog.vue'
import FandomSetDialog from './FandomSetDialog.vue'
const modal = useModalStore()
const store = useTagStore()
const errorMsg = ref(null)
// Which tag chip's kebab menu is open (only one at a time). Drives each
// chip menu's v-model so opening never depends on Vuetify's activator click.
const openTagId = ref(null)
const KIND_ICONS = {
general: 'mdi-tag', character: 'mdi-account-circle',
@@ -107,6 +127,18 @@ async function onRenamed() {
// Reflect the new name in the modal's current tag list without a full reload.
await modal.reloadTags()
}
const fandomDialog = ref(false)
const fandomTarget = ref(null)
function openSetFandom(tag) {
fandomTarget.value = tag
fandomDialog.value = true
}
async function onFandomUpdated() {
fandomDialog.value = false
// A fandom change can merge the tag away; reload to reflect the new state.
await modal.reloadTags()
}
</script>
<style scoped>
@@ -122,4 +154,5 @@ async function onRenamed() {
}
.fc-tag-panel__chips { display: flex; flex-wrap: wrap; gap: 6px; }
.kebab-wrap { display: inline-flex; align-items: center; }
.kebab-icon { cursor: pointer; }
</style>
+156 -202
View File
@@ -1,29 +1,18 @@
<template>
<v-card
:class="['fc-post-card', expanded && 'fc-post-card--expanded']"
variant="outlined"
:tabindex="expanded ? -1 : 0"
@click="onCardClick"
@keydown.enter="onCardClick"
>
<v-card class="fc-post-card" variant="outlined">
<div class="fc-post-card__head">
<!-- Posts with no live subscription have source=null (alembic
0030); show a "filesystem import" affordance instead of a
platform chip. -->
<!-- Posts with no live subscription have source=null (alembic 0030);
show a "filesystem import" affordance instead of a platform chip. -->
<v-chip size="x-small" variant="tonal">
{{ post.source?.platform ?? 'filesystem import' }}
</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-card__artist"
@click.stop
>{{ post.artist.name }}</RouterLink>
<span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span>
<span v-if="expanded && images.length" class="fc-post-card__meta">
· {{ images.length }} image{{ images.length === 1 ? '' : 's' }}
</span>
<span v-if="expanded && attachments.length" class="fc-post-card__meta">
· {{ attachments.length }} attachment{{ attachments.length === 1 ? '' : 's' }}
<span v-if="totalImages" class="fc-post-card__meta">
· {{ totalImages }} image{{ totalImages === 1 ? '' : 's' }}
</span>
<v-spacer />
<v-btn
@@ -31,140 +20,108 @@
:href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="x-small" variant="text"
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
@click.stop
/>
<v-btn
:icon="expanded ? 'mdi-chevron-up' : 'mdi-chevron-down'"
size="x-small" variant="text"
:aria-label="expanded ? 'Collapse post' : 'Expand post'"
@click.stop="toggleExpanded"
/>
</div>
<!-- Compact body: collapsed card. Hero + thumb rail + truncated text. -->
<div v-if="!expanded" class="fc-post-card__body">
<div class="fc-post-card__body">
<div class="fc-post-card__media">
<template v-if="images.length">
<div class="fc-post-card__hero">
<img :src="hero.thumbnail_url" :alt="`hero thumbnail`" loading="lazy" />
</div>
<div v-if="rail.length" class="fc-post-card__rail">
<div v-for="t in rail" :key="t.image_id" class="fc-post-card__rail-cell">
<img :src="t.thumbnail_url" :alt="`thumbnail`" loading="lazy" />
</div>
<div v-if="moreCount > 0" class="fc-post-card__rail-more">
+{{ moreCount }}
</div>
<!-- Images open the post-scoped image modal (look bigger + arrow
through ALL the post's images) the card never expands. -->
<button
type="button" class="fc-post-card__hero"
aria-label="Open images" @click="openModal(hero.image_id)"
>
<img :src="hero.thumbnail_url" alt="hero thumbnail" loading="lazy" />
</button>
<div v-if="rail.length || moreCount" class="fc-post-card__rail">
<button
v-for="t in rail" :key="t.image_id" type="button"
class="fc-post-card__rail-cell"
aria-label="Open image" @click="openModal(t.image_id)"
>
<img :src="t.thumbnail_url" alt="thumbnail" loading="lazy" />
</button>
<button
v-if="moreCount > 0" type="button"
class="fc-post-card__rail-more"
:aria-label="`Open ${moreCount} more images`"
@click="openModalAtMore"
>+{{ moreCount }}</button>
</div>
</template>
<PostEmptyThumbs v-else />
</div>
<div class="fc-post-card__text">
<h3 v-if="plainTitle" class="fc-post-card__title">
{{ plainTitle }}
</h3>
<h3 v-if="plainTitle" class="fc-post-card__title">{{ plainTitle }}</h3>
<h3 v-else class="fc-post-card__title fc-post-card__title--missing">
Post {{ post.external_post_id }}
</h3>
<p v-if="post.description_plain" class="fc-post-card__desc">
{{ post.description_plain }}
</p>
<p
v-if="hasDescription" ref="descEl"
class="fc-post-card__desc"
:class="{ 'fc-post-card__desc--clamped': !descExpanded }"
>{{ descText }}</p>
<p v-else class="fc-post-card__desc fc-post-card__desc--missing">
(no description)
</p>
<div v-if="post.attachments?.length" class="fc-post-card__atts">
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
{{ post.attachments.length }} attachment{{ post.attachments.length === 1 ? '' : 's' }}
</div>
</div>
</div>
<!-- The ONLY in-place expansion: the post text, and only when it's
actually truncated (server flag or a CSS-clamp overflow). -->
<button
v-if="canExpand" type="button" class="fc-post-card__more"
@click="toggleDesc"
>{{ descExpanded ? 'Show less' : 'Show more' }}</button>
<!-- Expanded body: title, full mosaic, full sanitized HTML description,
attachments. Lazy-loaded detail via getPostFull. -->
<div v-else class="fc-post-card__expanded">
<h2 v-if="plainTitle" class="fc-post-card__title-full">
{{ plainTitle }}
</h2>
<h2 v-else class="fc-post-card__title-full fc-post-card__title--missing">
Post {{ post.external_post_id }}
</h2>
<section v-if="images.length" class="fc-post-card__sec">
<PostImageGrid :thumbnails="images" />
<div v-if="!detailLoaded" class="fc-post-card__loading-hint">
Loading full image list
</div>
</section>
<section v-if="descriptionHtml" class="fc-post-card__sec">
<div class="fc-post-card__desc-full" v-html="descriptionHtml" />
</section>
<section v-else-if="detailLoaded" class="fc-post-card__sec">
<p class="fc-post-card__desc fc-post-card__desc--missing">(no description)</p>
</section>
<section v-if="attachments.length" class="fc-post-card__sec">
<h3 class="fc-post-card__h3">Attachments</h3>
<div class="fc-post-card__atts-full">
<div v-if="attachments.length" class="fc-post-card__atts">
<a
v-for="att in attachments" :key="att.id"
:href="att.download_url" download
class="fc-post-card__att"
@click.stop
:href="att.download_url" download class="fc-post-card__att"
>
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
<span>{{ att.original_filename }}</span>
<span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span>
</a>
</div>
</section>
</div>
</div>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { computed, nextTick, onBeforeUnmount, onMounted, ref } from 'vue'
import { RouterLink } from 'vue-router'
import { useModalStore } from '../../stores/modal.js'
import { usePostsStore } from '../../stores/posts.js'
import { sanitizeHtml, toPlainText } from '../../utils/htmlSanitize.js'
import { toPlainText } from '../../utils/htmlSanitize.js'
import PostEmptyThumbs from './PostEmptyThumbs.vue'
import PostImageGrid from './PostImageGrid.vue'
const props = defineProps({
post: { type: Object, required: true },
})
const postsStore = usePostsStore()
const modal = useModalStore()
// Per-card expand state. No global modal — each PostCard owns its own
// view-mode and lazy-loaded detail.
const expanded = ref(false)
// Full detail (uncapped thumbnails + full description), fetched lazily — only
// when opening the modal for a post with >6 images, or expanding a
// server-truncated description.
const detail = ref(null)
const detailLoaded = ref(false)
const detailError = ref(null)
// When expanded + detail loaded, prefer the uncapped detail thumbnails +
// full description. Falls back to feed shape if detail fetch is in flight
// or failed.
const merged = computed(() => detail.value || props.post)
const images = computed(() => merged.value.thumbnails || [])
const attachments = computed(() => merged.value.attachments || [])
// Titles can arrive as stored HTML (e.g. "<strong>…</strong>"); render as
// plain text — the CSS makes the title bold.
const attachments = computed(() => props.post.attachments || [])
const images = computed(() => props.post.thumbnails || [])
const totalImages = computed(() => images.value.length + (props.post.thumbnails_more || 0))
const plainTitle = computed(() => toPlainText(props.post.post_title))
// Compact-view hero+rail derived from the feed-shape (capped 6).
const hero = computed(() => props.post.thumbnails?.[0])
const rail = computed(() => (props.post.thumbnails || []).slice(1, 4))
const hero = computed(() => images.value[0])
const rail = computed(() => images.value.slice(1, 4))
const visibleCount = computed(() => (images.value.length ? 1 + rail.value.length : 0))
const moreCount = computed(() => {
const more = props.post.thumbnails_more || 0
const railLen = rail.value.length
const extraShown = Math.max(0, (props.post.thumbnails?.length || 0) - 1 - railLen)
const extraShown = Math.max(0, images.value.length - visibleCount.value)
return more + extraShown
})
@@ -180,49 +137,77 @@ const relativeDate = computed(() => {
return new Date(sortDateIso.value).toLocaleDateString()
})
const descriptionHtml = computed(() => {
// Detail endpoint returns description_full as plain text (the service
// uses html_to_plain on the stored description). Render plain text in
// <p> wrappers; sanitize defensively in case the backend ever returns
// raw HTML.
const raw = merged.value.description_full || merged.value.description_plain
if (!raw) return ''
if (/[<>]/.test(raw)) return sanitizeHtml(raw)
const esc = raw
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
return esc
.split(/\n\s*\n/)
.map((p) => `<p>${p.replace(/\n/g, '<br>')}</p>`)
.join('')
})
async function loadDetailIfNeeded () {
if (detailLoaded.value || detail.value) return
try {
detail.value = await postsStore.getPostFull(props.post.id)
detailLoaded.value = true
} catch (e) {
detailError.value = e.message
// Leave merged on feed-shape; the card still renders the truncated
// body so the operator isn't staring at a blank panel.
// --- images → post-scoped modal ---------------------------------------
async function fullImageIds () {
// Feed caps thumbnails at 6; load detail for the complete id list only when
// there are more, so the modal can arrow through ALL of the post's images.
if ((props.post.thumbnails_more || 0) === 0) {
return images.value.map((t) => t.image_id)
}
if (!detail.value) {
try {
detail.value = await postsStore.getPostFull(props.post.id)
} catch { /* fall back to the capped feed list */ }
}
return (detail.value?.thumbnails || images.value).map((t) => t.image_id)
}
function toggleExpanded () {
expanded.value = !expanded.value
if (expanded.value) loadDetailIfNeeded()
async function openModal (imageId) {
modal.open(imageId, { postImageIds: await fullImageIds() })
}
function onCardClick (e) {
// Inner interactive elements use @click.stop so they never reach here.
// Whole-card click expands a collapsed card; collapsing is chevron-only
// so a mosaic-image click on an expanded card can never accidentally
// collapse the surrounding card.
if (expanded.value) return
expanded.value = true
loadDetailIfNeeded()
async function openModalAtMore () {
const ids = await fullImageIds()
const first = ids[visibleCount.value] ?? ids[0]
if (first != null) modal.open(first, { postImageIds: ids })
}
// --- description "Show more" (text-only, in place, only when truncated) ----
const descExpanded = ref(false)
const cssOverflow = ref(false)
const descEl = ref(null)
const hasDescription = computed(() => !!props.post.description_plain)
const fullDescription = computed(() => detail.value?.description_full || null)
const descText = computed(() =>
descExpanded.value
? (fullDescription.value || props.post.description_plain)
: props.post.description_plain,
)
// Show the toggle iff the server truncated the text OR the clamp is cutting it.
const canExpand = computed(
() => props.post.description_truncated === true || cssOverflow.value,
)
function measureOverflow () {
const el = descEl.value
cssOverflow.value = !!el && el.scrollHeight > el.clientHeight + 1
}
let ro = null
onMounted(() => {
nextTick(measureOverflow)
// Re-measure when the card resizes (the container-query clamp differs by
// width). Guarded for happy-dom / older runtimes without ResizeObserver.
if (typeof ResizeObserver !== 'undefined' && descEl.value) {
ro = new ResizeObserver(() => { if (!descExpanded.value) measureOverflow() })
ro.observe(descEl.value)
}
})
onBeforeUnmount(() => { if (ro) { ro.disconnect(); ro = null } })
async function toggleDesc () {
if (!descExpanded.value) {
if (props.post.description_truncated && !fullDescription.value && !detail.value) {
try {
detail.value = await postsStore.getPostFull(props.post.id)
} catch { /* render the truncated text rather than nothing */ }
}
descExpanded.value = true
} else {
descExpanded.value = false
nextTick(measureOverflow)
}
}
function formatBytes (n) {
@@ -239,20 +224,6 @@ function formatBytes (n) {
padding: 1rem;
margin-bottom: 1rem;
container-type: inline-size;
transition: border-color 0.15s ease;
}
.fc-post-card:not(.fc-post-card--expanded) {
cursor: pointer;
}
.fc-post-card:not(.fc-post-card--expanded):hover {
border-color: rgb(var(--v-theme-accent));
}
.fc-post-card:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 2px;
}
.fc-post-card--expanded {
border-color: rgb(var(--v-theme-accent) / 0.6);
}
.fc-post-card__head {
@@ -273,7 +244,6 @@ function formatBytes (n) {
.fc-post-card__date,
.fc-post-card__meta { white-space: nowrap; }
/* ---- COMPACT BODY ---- */
.fc-post-card__body {
display: flex;
flex-direction: column;
@@ -288,6 +258,13 @@ function formatBytes (n) {
.fc-post-card__text { flex: 1 1 0; min-width: 0; }
}
/* Image tiles are buttons (open the post-scoped modal) — reset button chrome. */
.fc-post-card__hero,
.fc-post-card__rail-cell,
.fc-post-card__rail-more {
display: block; padding: 0; border: 0; background: none;
cursor: pointer;
}
.fc-post-card__hero {
width: 100%;
aspect-ratio: 16 / 10;
@@ -297,10 +274,13 @@ function formatBytes (n) {
.fc-post-card__hero img {
width: 100%; height: 100%;
object-fit: cover; display: block;
transition: transform 0.2s ease, filter 0.2s ease;
}
.fc-post-card__hero:hover img,
.fc-post-card__rail-cell:hover img { transform: scale(1.03); filter: brightness(1.08); }
.fc-post-card__rail {
display: flex; gap: 6px; margin-top: 6px;
display: flex; gap: 6px; margin-top: 6px; flex-wrap: wrap;
}
.fc-post-card__rail-cell {
width: 80px; height: 80px;
@@ -318,6 +298,10 @@ function formatBytes (n) {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem;
}
.fc-post-card__rail-more:hover {
border-color: rgb(var(--v-theme-accent));
color: rgb(var(--v-theme-accent));
}
.fc-post-card__title {
font-family: 'Fraunces', Georgia, serif;
@@ -346,67 +330,36 @@ function formatBytes (n) {
font-size: 0.9rem;
line-height: 1.5;
color: rgb(var(--v-theme-on-surface));
margin: 0 0 12px 0;
margin: 0;
white-space: pre-wrap;
}
/* Clamp ONLY while collapsed; expanding drops the clamp to show it all. */
.fc-post-card__desc--clamped {
display: -webkit-box;
-webkit-line-clamp: 3;
-webkit-box-orient: vertical;
overflow: hidden;
}
@container (min-width: 800px) {
.fc-post-card__desc--clamped { -webkit-line-clamp: 5; }
}
.fc-post-card__desc--missing {
font-style: italic;
color: rgb(var(--v-theme-on-surface-variant));
}
@container (min-width: 800px) {
.fc-post-card__desc { -webkit-line-clamp: 5; }
.fc-post-card__more {
margin-top: 6px;
padding: 0;
background: none; border: 0; cursor: pointer;
color: rgb(var(--v-theme-accent));
font-size: 0.85rem; font-weight: 600;
}
.fc-post-card__more:hover { text-decoration: underline; }
.fc-post-card__atts {
display: inline-flex;
align-items: center;
gap: 6px;
font-size: 0.85rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
/* ---- EXPANDED BODY ---- */
.fc-post-card__expanded {
display: flex;
flex-direction: column;
gap: 20px;
}
.fc-post-card__title-full {
font-family: 'Fraunces', Georgia, serif;
font-size: 22px;
font-weight: 700;
margin: 0;
color: rgb(var(--v-theme-on-surface));
}
@container (min-width: 800px) {
.fc-post-card__title-full { font-size: 26px; }
}
.fc-post-card__sec { margin: 0; }
.fc-post-card__h3 {
font-family: 'Fraunces', Georgia, serif;
font-size: 16px;
font-weight: 500;
margin: 0 0 8px 0;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-card__loading-hint {
margin-top: 8px;
font-size: 0.8rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__desc-full {
font-size: 0.95rem;
line-height: 1.55;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-card__desc-full :deep(p) { margin: 0 0 12px 0; }
.fc-post-card__desc-full :deep(a) { color: rgb(var(--v-theme-accent)); }
.fc-post-card__atts-full {
display: flex; flex-wrap: wrap; gap: 8px;
margin-top: 12px;
}
.fc-post-card__att {
display: inline-flex;
@@ -423,5 +376,6 @@ function formatBytes (n) {
color: rgb(var(--v-theme-accent));
border-color: rgb(var(--v-theme-accent));
}
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -1,63 +0,0 @@
<template>
<div class="fc-post-grid">
<button
v-for="(t, idx) in thumbnails"
:key="t.image_id"
type="button"
class="fc-post-grid__cell"
:aria-label="`Open image ${idx + 1} of ${thumbnails.length}`"
@click="openImage(t.image_id, idx)"
>
<img
:src="t.thumbnail_url"
:alt="`thumbnail ${idx + 1}`"
loading="lazy"
/>
</button>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { useModalStore } from '../../stores/modal.js'
const props = defineProps({
thumbnails: { type: Array, required: true }, // [{ image_id, thumbnail_url, ... }]
})
const modal = useModalStore()
const imageIds = computed(() => props.thumbnails.map(t => t.image_id))
function openImage (id, idx) {
modal.open(id, { postImageIds: imageIds.value, initialIndex: idx })
}
</script>
<style scoped>
.fc-post-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 6px;
}
.fc-post-grid__cell {
aspect-ratio: 4 / 3;
overflow: hidden;
border-radius: 4px;
cursor: pointer;
border: 0;
padding: 0;
background: rgb(var(--v-theme-background));
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.fc-post-grid__cell:hover {
transform: scale(1.02);
box-shadow: 0 0 0 2px rgb(var(--v-theme-accent));
}
.fc-post-grid__cell img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
</style>
@@ -13,7 +13,7 @@
clearable
no-filter
return-object
style="min-width: 240px"
class="fc-posts-filters__artist"
@update:search="onArtistSearch"
@update:model-value="onArtistPicked"
/>
@@ -25,7 +25,7 @@
density="compact"
hide-details
clearable
style="min-width: 180px"
class="fc-posts-filters__platform"
@update:model-value="emitFilters"
/>
@@ -135,6 +135,14 @@ watch(() => props.platform, (val) => {
display: flex;
gap: 0.75rem;
align-items: center;
flex-wrap: wrap;
padding-bottom: 1rem;
}
.fc-posts-filters__artist { min-width: 240px; flex: 1 1 240px; max-width: 360px; }
.fc-posts-filters__platform { min-width: 180px; flex: 1 1 180px; max-width: 240px; }
/* Phones: each field takes a full-width row instead of overflowing. */
@media (max-width: 600px) {
.fc-posts-filters__artist,
.fc-posts-filters__platform { min-width: 100%; max-width: none; flex-basis: 100%; }
}
</style>
@@ -0,0 +1,84 @@
<template>
<v-card>
<v-card-title>Database maintenance</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3">
VACUUM (ANALYZE) reclaims dead-tuple bloat which slows the random
showcase, since it samples physical blocks and refreshes the query
planner's statistics. Runs automatically each week; trigger a pass
here after a large import or cleanup.
</p>
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
<v-icon start>mdi-database-cog</v-icon> Run VACUUM ANALYZE now
</v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
<v-table
v-if="store.tables.length" density="compact" class="mt-4 fc-dbm__table"
>
<thead>
<tr>
<th>Table</th>
<th class="text-right">Live rows</th>
<th class="text-right">Dead</th>
<th class="text-right">Dead %</th>
<th>Last vacuum</th>
</tr>
</thead>
<tbody>
<tr v-for="t in store.tables" :key="t.table">
<td>{{ t.table }}</td>
<td class="text-right">{{ t.live.toLocaleString() }}</td>
<td class="text-right">{{ t.dead.toLocaleString() }}</td>
<td
class="text-right"
:class="{ 'text-warning': t.dead_pct >= 20 }"
>{{ t.dead_pct }}%</td>
<td class="text-caption">{{ fmt(t.last_vacuum || t.last_autovacuum) }}</td>
</tr>
</tbody>
</v-table>
<p v-else class="text-caption mt-3" style="opacity: 0.6;">
No table statistics yet.
</p>
</v-card-text>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import { toast } from '../../utils/toast.js'
import { useDbMaintenanceStore } from '../../stores/dbMaintenance.js'
import QueueStatusBar from './QueueStatusBar.vue'
const store = useDbMaintenanceStore()
const busy = ref(false)
const queued = ref(false)
onMounted(() => store.loadStats())
async function run () {
busy.value = true
queued.value = false
try {
await store.runVacuum()
queued.value = true
// pg_stat updates after the vacuum lands — refresh shortly after.
setTimeout(() => store.loadStats(), 5000)
} catch (e) {
toast({ text: e.message, type: 'error' })
} finally {
busy.value = false
}
}
function fmt (iso) {
return iso ? new Date(iso).toLocaleString() : ''
}
</script>
<style scoped>
.fc-dbm__table { background: transparent; }
</style>
@@ -2,6 +2,42 @@
<v-card>
<v-card-title>Import filters</v-card-title>
<v-card-text v-if="store.settings">
<!-- Near-duplicate dedup sensitivity, hoisted to the top: it's the
most-asked knob too loose and edits/variants of the same image
get dropped as duplicates on import. Slider with sane labelled
stops for the gist + a number field for precision; both bind the
same phash_threshold. -->
<div class="fc-phash">
<div class="fc-phash__title">Near-duplicate sensitivity</div>
<div class="fc-help mb-1">
How aggressively imports merge look-alike images (perceptual-hash
distance). <strong>Lower it if edits/variants of the same image are
being dropped as duplicates;</strong> raise it to collapse more
look-alikes. Applies to new imports.
</div>
<v-row align="center" no-gutters>
<v-col cols="12" sm="9">
<v-slider
v-model="local.phash_threshold"
:min="0" :max="16" :step="1"
:ticks="PHASH_TICKS" show-ticks="always" tick-size="4"
thumb-label color="accent" hide-details
class="fc-phash__slider"
@end="save"
/>
</v-col>
<v-col cols="8" sm="3" class="ps-sm-4 mt-2 mt-sm-0">
<v-text-field
v-model.number="local.phash_threshold"
label="Distance" type="number" min="0"
density="compact" hide-details @blur="save"
/>
</v-col>
</v-row>
</div>
<v-divider class="my-5" />
<v-row>
<v-col cols="12" sm="6">
<v-text-field
@@ -41,14 +77,6 @@
:disabled="!local.skip_single_color" @end="save"
/>
</v-col>
<v-col cols="12" sm="6">
<v-text-field
v-model.number="local.phash_threshold"
label="Perceptual-hash threshold" type="number" min="0"
density="compact" hide-details @blur="save"
/>
<div class="fc-help">Higher = looser near-duplicate matching.</div>
</v-col>
</v-row>
<v-alert v-if="store.settingsError" type="error" variant="tonal" class="mt-2" closable>
@@ -66,6 +94,9 @@ import { reactive, watch } from 'vue'
import { useImportStore } from '../../stores/import.js'
const store = useImportStore()
// Labelled stops so the less-initiated get the gist without knowing what a
// Hamming distance is. 0 = byte-for-byte only; 10 = the shipped default.
const PHASH_TICKS = { 0: 'Exact', 4: 'Strict', 10: 'Default', 16: 'Loose' }
// Downloader + schedule-defaults fields moved to
// /subscriptions?tab=settings (operator decision 2026-05-27). This form
// now only owns image-import filters.
@@ -89,4 +120,11 @@ async function save() {
color: rgb(var(--v-theme-on-surface-variant));
margin-top: 2px;
}
.fc-phash__title {
font-size: 0.95rem;
font-weight: 600;
color: rgb(var(--v-theme-on-surface));
}
/* Headroom so the tick labels (Exact/Strict/Default/Loose) aren't clipped. */
.fc-phash__slider { margin-bottom: 18px; }
</style>
@@ -14,6 +14,7 @@
<MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" />
<AliasTable class="mt-4" />
<DbMaintenanceCard class="mt-6" />
<BackupCard class="mt-6" />
<!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) it
operates on the existing library which fits the Cleanup-tab
@@ -31,6 +32,7 @@ import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue'
import DbMaintenanceCard from './DbMaintenanceCard.vue'
import BackupCard from './BackupCard.vue'
import { useSystemActivityStore } from '../../stores/systemActivity.js'
@@ -89,6 +89,53 @@
@click="onKindCommit"
>Delete {{ kindPreview.count }} legacy tag(s)</v-btn>
</div>
<v-divider class="my-5" />
<p class="text-body-2 mb-2">
<strong class="text-error">Reset content tagging.</strong>
Deletes every <code>general</code> and <code>character</code> tag and
removes them from every image, so you can re-tag from scratch with the
auto-suggest. <strong>Fandoms and series (with their page order) are
kept</strong>, and each image's saved predictions are untouched open
an image and its suggestions reappear.
</p>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
Irreversible there's no undo except restoring a DB backup.
Back one up first (Settings → Maintenance → Backup).
</v-alert>
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="loadingResetPreview"
class="mb-3"
@click="onResetPreview"
>Preview content-tag reset</v-btn>
<div v-if="resetPreview">
<p class="text-body-2 mb-2">
<strong>{{ resetPreview.count }}</strong> content tag(s)
<span v-for="(n, k) in resetPreview.by_kind" :key="k" class="fc-muted">
({{ k }}: {{ n }})&nbsp;
</span>
across <strong>{{ resetPreview.applications }}</strong> image
application(s).
</p>
<div v-if="resetPreview.sample_names?.length" class="fc-name-grid mb-3">
<span v-for="n in resetPreview.sample_names" :key="n" class="fc-name">
{{ n }}
</span>
</div>
<v-btn
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete-alert"
:disabled="!resetPreview.count"
:loading="resetCommitting"
@click="onResetCommit"
>Delete {{ resetPreview.count }} content tag(s) +
{{ resetPreview.applications }} application(s)</v-btn>
</div>
</v-card-text>
</v-card>
</template>
@@ -105,6 +152,9 @@ const committing = ref(false)
const kindPreview = ref(null)
const loadingKindPreview = ref(false)
const kindCommitting = ref(false)
const resetPreview = ref(null)
const loadingResetPreview = ref(false)
const resetCommitting = ref(false)
async function onPreview() {
loadingPreview.value = true
@@ -143,6 +193,25 @@ async function onKindCommit() {
kindCommitting.value = false
}
}
async function onResetPreview() {
loadingResetPreview.value = true
try {
resetPreview.value = await store.resetContentTagging({ dryRun: true })
} finally {
loadingResetPreview.value = false
}
}
async function onResetCommit() {
resetCommitting.value = true
try {
await store.resetContentTagging({ dryRun: false })
resetPreview.value = { count: 0, by_kind: {}, applications: 0, sample_names: [] }
} finally {
resetCommitting.value = false
}
}
</script>
<style scoped>
@@ -93,6 +93,7 @@ const ERROR_TYPE_COLOR = {
unsupported_url: 'error',
http_error: 'error',
unknown_error: 'error',
api_drift: 'error',
partial: 'info',
tier_limited: 'info',
no_new_content: 'info',
@@ -108,6 +109,7 @@ const ERROR_TYPE_HINT = {
http_error: 'Generic HTTP error — see Logs.',
unsupported_url: 'gallery-dl does not support this URL pattern.',
unknown_error: 'Could not classify — see Logs.',
api_drift: 'Patreon changed its API shape — the native ingester needs a code update.',
}
function errorTypeColor(t) { return ERROR_TYPE_COLOR[t] || 'error' }
function errorTypeHint(t) { return ERROR_TYPE_HINT[t] || '' }
@@ -0,0 +1,130 @@
<template>
<div class="fc-source-card">
<div class="fc-source-card__top">
<SourceHealthDot :source="source" :warning-threshold="warningThreshold" />
<v-chip size="x-small" variant="tonal" label>{{ source.platform }}</v-chip>
<v-spacer />
<v-switch
:model-value="source.enabled"
density="compact" hide-details color="accent"
@click.stop
@update:model-value="onToggleEnabled"
/>
</div>
<a
:href="source.url" target="_blank" rel="noopener"
class="fc-source-card__url" @click.stop
>{{ source.url }}</a>
<div class="fc-source-card__meta">
<span>Last {{ formatRelative(source.last_checked_at) }}</span>
<span>Next {{ formatRelative(source.next_check_at, { future: true }) }}</span>
<v-chip
v-if="(source.consecutive_failures || 0) > 0"
size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }} err</v-chip>
<v-chip
v-else-if="source.backfill_state === 'running'"
size="x-small" color="info" variant="tonal" label
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling'
}}{{ source.backfill_chunks ? ` (${source.backfill_chunks})` : '' }}</v-chip>
<v-chip
v-else-if="source.backfill_state === 'complete'"
size="x-small" color="success" variant="tonal" label
>Backfilled</v-chip>
<v-chip
v-else-if="source.backfill_state === 'stalled'"
size="x-small" color="warning" variant="tonal" label
>Stalled</v-chip>
</div>
<div class="fc-source-card__actions">
<v-btn
size="x-small" variant="text" :loading="checking"
@click.stop="$emit('check', source)"
>
<v-icon>mdi-play</v-icon>
<v-tooltip activator="parent" location="top">Check now</v-tooltip>
</v-btn>
<v-btn
size="x-small" variant="text"
:color="source.backfill_state === 'running' ? 'warning' : undefined"
@click.stop="$emit('backfill', source)"
>
<v-icon>{{ source.backfill_state === 'running' ? 'mdi-stop' : 'mdi-magnify-scan' }}</v-icon>
<v-tooltip activator="parent" location="top">
{{ source.backfill_state === 'running'
? (source.backfill_bypass_seen ? 'Stop recovery' : 'Stop backfill')
: 'Backfill full history' }}
</v-tooltip>
</v-btn>
<v-btn
v-if="source.platform === 'patreon' && source.backfill_state !== 'running'"
size="x-small" variant="text"
@click.stop="$emit('recover', source)"
>
<v-icon>mdi-backup-restore</v-icon>
<v-tooltip activator="parent" location="top">
Recover re-fetch dropped near-dups &amp; re-evaluate under the current threshold
</v-tooltip>
</v-btn>
<v-btn size="x-small" variant="text" @click.stop="$emit('edit', source)">
<v-icon>mdi-pencil</v-icon>
<v-tooltip activator="parent" location="top">Edit</v-tooltip>
</v-btn>
<v-btn
size="x-small" variant="text" color="error"
@click.stop="$emit('remove', source)"
>
<v-icon>mdi-close</v-icon>
<v-tooltip activator="parent" location="top">Remove</v-tooltip>
</v-btn>
</div>
</div>
</template>
<script setup>
import SourceHealthDot from './SourceHealthDot.vue'
import { formatRelative } from '../../utils/date.js'
// Mobile-stacked equivalent of SourceRow (the desktop <tr>) — same data and
// emits, but laid out vertically so the wide source columns never force the
// lateral scroll the operator flagged on phones.
const props = defineProps({
source: { type: Object, required: true },
checking: { type: Boolean, default: false },
warningThreshold: { type: Number, default: 5 },
})
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover'])
function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value })
}
</script>
<style scoped>
.fc-source-card {
padding: 8px 10px;
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 6px;
background: rgb(var(--v-theme-surface));
display: flex; flex-direction: column; gap: 6px;
}
.fc-source-card__top { display: flex; align-items: center; gap: 8px; }
.fc-source-card__url {
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none;
font-size: 0.8rem;
word-break: break-all;
}
.fc-source-card__url:hover { color: rgb(var(--v-theme-accent)); }
.fc-source-card__meta {
display: flex; flex-wrap: wrap; align-items: center; gap: 6px 12px;
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
.fc-source-card__actions {
display: flex; gap: 2px; justify-content: flex-end;
}
</style>
@@ -32,11 +32,18 @@
size="x-small" color="error" variant="tonal" label
>{{ source.consecutive_failures }}</v-chip>
<v-chip
v-else-if="(source.backfill_runs_remaining || 0) > 0"
v-else-if="source.backfill_state === 'running'"
size="x-small" color="info" variant="tonal" label
>
backfill ({{ source.backfill_runs_remaining }}×)
</v-chip>
>{{ source.backfill_bypass_seen ? 'Recovering' : 'Backfilling'
}}{{ source.backfill_chunks ? ` (${source.backfill_chunks})` : '' }}</v-chip>
<v-chip
v-else-if="source.backfill_state === 'complete'"
size="x-small" color="success" variant="tonal" label
>Backfilled</v-chip>
<v-chip
v-else-if="source.backfill_state === 'stalled'"
size="x-small" color="warning" variant="tonal" label
>Stalled</v-chip>
<span v-else class="fc-source-row__zero">0</span>
</td>
<td class="fc-source-row__actions">
@@ -49,13 +56,25 @@
<v-tooltip activator="parent" location="top">Check now</v-tooltip>
</v-btn>
<v-btn
icon="mdi-magnify-scan" size="x-small" variant="text"
:disabled="(source.backfill_runs_remaining || 0) > 0"
size="x-small" variant="text"
:color="source.backfill_state === 'running' ? 'warning' : undefined"
@click.stop="$emit('backfill', source)"
>
<v-icon>mdi-magnify-scan</v-icon>
<v-icon>{{ source.backfill_state === 'running' ? 'mdi-stop' : 'mdi-magnify-scan' }}</v-icon>
<v-tooltip activator="parent" location="top">
Deep scan walk full history for next few runs
{{ source.backfill_state === 'running'
? (source.backfill_bypass_seen ? 'Stop recovery' : 'Stop backfill')
: 'Backfill — walk the full post history until complete' }}
</v-tooltip>
</v-btn>
<v-btn
v-if="source.platform === 'patreon' && source.backfill_state !== 'running'"
size="x-small" variant="text"
@click.stop="$emit('recover', source)"
>
<v-icon>mdi-backup-restore</v-icon>
<v-tooltip activator="parent" location="top">
Recover re-fetch dropped near-dups &amp; re-evaluate under the current threshold
</v-tooltip>
</v-btn>
<v-btn
@@ -85,7 +104,7 @@ const props = defineProps({
checking: { type: Boolean, default: false },
warningThreshold: { type: Number, default: 5 },
})
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill'])
const emit = defineEmits(['edit', 'remove', 'toggle', 'check', 'backfill', 'recover'])
function onToggleEnabled(value) {
emit('toggle', { source: props.source, enabled: value })
@@ -26,14 +26,14 @@
:items="STATUS_OPTIONS"
:disabled="needsAttention"
density="compact" variant="outlined" hide-details
style="max-width: 180px"
class="fc-subs__status"
/>
<v-text-field
v-model="search"
density="compact" variant="outlined" hide-details clearable
prepend-inner-icon="mdi-magnify"
placeholder="Search subscriptions"
style="max-width: 320px"
class="fc-subs__search"
/>
</div>
@@ -75,7 +75,7 @@
<p v-else>No subscriptions match the current filter.</p>
</div>
<v-card v-else class="fc-subs__card" variant="outlined">
<v-card v-else-if="!isMobile" class="fc-subs__card" variant="outlined">
<v-data-table
:headers="headers"
:items="filteredGroups"
@@ -175,6 +175,7 @@
@toggle="toggleSourceEnabled"
@check="onCheck"
@backfill="onBackfill"
@recover="onRecover"
/>
<tr v-if="item.sources.length === 0">
<td colspan="8" class="fc-subs__sources-empty">
@@ -189,6 +190,77 @@
</v-data-table>
</v-card>
<!-- Mobile: compact cards (several fit per screen), expanding to STACKED
source cards so the wide source columns never force lateral scroll. -->
<div v-else class="fc-subs__mlist">
<div
v-for="item in filteredGroups" :key="item.key"
class="fc-subs__mcard"
>
<div class="fc-subs__mhead" @click="toggleExpand(item)">
<v-checkbox-btn
:model-value="isSelected(item)" density="compact" hide-details
@click.stop @update:model-value="toggleSelect(item)"
/>
<span class="fc-subs__name">{{ item.artist.name }}</span>
<v-spacer />
<SourceHealthDot
v-if="item.worstSource"
:source="item.worstSource" :warning-threshold="failureThreshold"
/>
<v-icon size="small">
{{ isExpanded(item) ? 'mdi-chevron-up' : 'mdi-chevron-down' }}
</v-icon>
</div>
<div class="fc-subs__mmeta">
<PlatformChip
v-for="p in item.platforms" :key="p" :platform="p" size="x-small"
/>
<span class="fc-subs__mmeta-text">
{{ item.sources.length }} src · {{ formatRelative(item.lastActivity) }}
</span>
</div>
<div v-if="isExpanded(item)" class="fc-subs__mbody">
<div class="fc-subs__mactions">
<v-btn
size="small" variant="text" :loading="anyChecking(item.sources)"
@click="checkAll(item)"
>
<v-icon>mdi-refresh</v-icon>
<v-tooltip activator="parent" location="top">Check all</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" @click="openAddSource(item.artist)">
<v-icon>mdi-plus</v-icon>
<v-tooltip activator="parent" location="top">Add source</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" :to="`/posts?artist_id=${item.artist.id}`">
<v-icon>mdi-rss</v-icon>
<v-tooltip activator="parent" location="top">View posts</v-tooltip>
</v-btn>
<v-btn size="small" variant="text" :to="`/artist/${item.artist.slug}`">
<v-icon>mdi-account</v-icon>
<v-tooltip activator="parent" location="top">Artist page</v-tooltip>
</v-btn>
</div>
<SourceCard
v-for="s in item.sources" :key="s.id" :source="s"
:checking="store.checkingIds.has(s.id)"
:warning-threshold="failureThreshold"
@edit="openEditSource"
@remove="removeSource"
@toggle="toggleSourceEnabled"
@check="onCheck"
@backfill="onBackfill"
@recover="onRecover"
/>
<div v-if="item.sources.length === 0" class="fc-subs__sources-empty">
No sources yet. Tap + to add one.
</div>
</div>
</div>
</div>
<SourceFormDialog
v-model="showSourceDialog"
:source="editingSource"
@@ -202,11 +274,13 @@
<script setup>
import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
import { useDisplay } from 'vuetify'
import { useRoute, useRouter } from 'vue-router'
import { useSourcesStore } from '../../stores/sources.js'
import { usePlatformsStore } from '../../stores/platforms.js'
import { useImportStore } from '../../stores/import.js'
import SourceRow from './SourceRow.vue'
import SourceCard from './SourceCard.vue'
import SourceHealthDot from './SourceHealthDot.vue'
import SourceFormDialog from './SourceFormDialog.vue'
import ArtistCreateDialog from './ArtistCreateDialog.vue'
@@ -230,6 +304,10 @@ const STATUS_OPTIONS = [
const route = useRoute()
const router = useRouter()
const display = useDisplay()
// Match the 600px breakpoint the rest of the hub uses; below it the table is
// replaced by the custom compact-card list.
const isMobile = computed(() => display.width.value < 600)
const store = useSourcesStore()
const platformsStore = usePlatformsStore()
const importStore = useImportStore()
@@ -239,6 +317,19 @@ const statusFilter = ref('all')
const needsAttention = ref(false)
const expanded = ref([])
const selected = ref([])
// Mobile card list drives the same `selected`/`expanded` key arrays the
// desktop v-data-table binds, so selection + bulk actions work identically.
function _toggleKey(arr, key) {
const i = arr.value.indexOf(key)
if (i === -1) arr.value = [...arr.value, key]
else arr.value = arr.value.filter((k) => k !== key)
}
function isSelected(item) { return selected.value.includes(item.key) }
function toggleSelect(item) { _toggleKey(selected, item.key) }
function isExpanded(item) { return expanded.value.includes(item.key) }
function toggleExpand(item) { _toggleKey(expanded, item.key) }
const showSourceDialog = ref(false)
const editingSource = ref(null)
const editingArtist = ref(null)
@@ -443,31 +534,39 @@ async function onCheck(source) {
}
}
// Plan #544: arm a source for backfill mode (gallery-dl walks the full
// post history) for the next N download runs. Default 3 — enough budget
// to finish a deep creator without re-prompting the operator across
// timeout boundaries. The chip on the row reflects the remaining count.
// Plan #693: toggle a run-until-done backfill. Starting walks the full post
// history in time-boxed chunks across ticks until it reaches the bottom (the
// row badge tracks progress / completion); stopping cancels back to tick mode.
async function onBackfill(source) {
const raw = globalThis.window?.prompt(
`Deep scan "${source.artist_name} (${source.platform})" — walk full history for the next how many download runs? (110, default 3)`,
'3',
)
if (raw == null) return
const runs = parseInt(raw, 10)
if (!Number.isFinite(runs) || runs < 1 || runs > 10) {
toast({ text: 'Deep scan: runs must be 110', type: 'error' })
return
}
const running = source.backfill_state === 'running'
try {
await store.setBackfill(source.id, runs, source.artist_id)
toast({
text: `Deep scan armed for ${runs} run${runs === 1 ? '' : 's'}`,
type: 'success',
})
if (running) {
await store.stopBackfill(source.id, source.artist_id)
toast({ text: `Backfill stopped for ${source.artist_name}`, type: 'success' })
} else {
await store.startBackfill(source.id, source.artist_id)
toast({ text: `Backfill started for ${source.artist_name}`, type: 'success' })
}
await store.loadAll()
} catch (e) {
toast({
text: `Deep scan failed: ${e?.detail || e?.message || e}`,
text: `Backfill ${running ? 'stop' : 'start'} failed: ${e?.detail || e?.message || e}`,
type: 'error',
})
}
}
// Plan #697: arm a recovery walk (Patreon-only) — re-fetches dropped-and-deleted
// near-dups and re-evaluates them under the current pHash threshold. Reuses the
// backfill lifecycle/badge; stop via the same Stop control (onBackfill).
async function onRecover(source) {
try {
await store.recoverSource(source.id, source.artist_id)
toast({ text: `Recovery started for ${source.artist_name}`, type: 'success' })
await store.loadAll()
} catch (e) {
toast({
text: `Recovery start failed: ${e?.detail || e?.message || e}`,
type: 'error',
})
}
@@ -568,6 +667,18 @@ async function bulkDelete() {
padding: 8px 0 1rem;
}
.fc-subs__bar .v-chip { cursor: pointer; }
.fc-subs__status { max-width: 180px; }
.fc-subs__search { max-width: 320px; flex: 1 1 200px; }
/* Phones: status + search each take a full-width row; drop the spacer that
would otherwise eat a row pushing them around. */
@media (max-width: 600px) {
.fc-subs__status, .fc-subs__search { max-width: none; flex-basis: 100%; }
.fc-subs__bar :deep(.v-spacer) { display: none; }
/* The table renders as stacked cards (mobile-breakpoint); reclaim the
desktop indent on the expanded sources detail (it keeps its own
horizontal scroll for the wide source columns). */
.fc-subs__sources-cell { padding-left: 0.5rem !important; }
}
.fc-subs__loading, .fc-subs__empty {
display: flex; justify-content: center; padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant));
@@ -575,6 +686,30 @@ async function bulkDelete() {
.fc-subs__card {
background: rgb(var(--v-theme-surface));
}
/* Mobile compact-card list (replaces the data-table <600px). */
.fc-subs__mlist { display: flex; flex-direction: column; gap: 8px; }
.fc-subs__mcard {
border: 1px solid rgb(var(--v-theme-surface-light));
border-radius: 8px;
background: rgb(var(--v-theme-surface));
padding: 8px 10px;
}
.fc-subs__mhead { display: flex; align-items: center; gap: 6px; cursor: pointer; }
.fc-subs__mmeta {
display: flex; flex-wrap: wrap; align-items: center; gap: 6px;
margin-top: 4px;
}
.fc-subs__mmeta-text {
font-size: 0.78rem; color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
.fc-subs__mbody {
margin-top: 8px; padding-top: 8px;
border-top: 1px solid rgb(var(--v-theme-surface-light));
display: flex; flex-direction: column; gap: 6px;
}
.fc-subs__mactions { display: flex; gap: 2px; }
.fc-subs__name { font-weight: 600; }
.fc-subs__chips {
display: flex; flex-wrap: wrap; gap: 4px;
+16
View File
@@ -127,6 +127,21 @@ export const useAdminStore = defineStore('admin', () => {
}
}
// Destructive: deletes ALL general + character tags so the operator can
// re-tag from scratch via auto-suggest. fandom + series preserved.
async function resetContentTagging({ dryRun = true } = {}) {
lastError.value = null
try {
return await api.post(
'/api/admin/tags/reset-content',
{ body: { dry_run: dryRun } },
)
} catch (e) {
lastError.value = e.message
throw e
}
}
// --- Task progress polling (taps FC-3i activity dashboard) --------
/**
@@ -162,6 +177,7 @@ export const useAdminStore = defineStore('admin', () => {
tagUsageCount,
pruneUnusedTags,
purgeLegacyTags,
resetContentTagging,
pollTaskUntilDone,
}
})
+9 -1
View File
@@ -55,6 +55,14 @@ export const useCleanupStore = defineStore('cleanup', () => {
return body.runs
}
// The most recent audit run for a given rule, or null. Cards call this on
// mount to reconnect to a scan that's still running (or to show the last
// completed result) after the user navigates away and back.
async function latestAuditForRule(rule) {
const body = await api.get('/api/cleanup/audit', { params: { rule, limit: 1 } })
return (body.runs && body.runs[0]) || null
}
async function applyAudit(id, confirm) {
return await api.post(`/api/cleanup/audit/${id}/apply`, { body: { confirm } })
}
@@ -67,6 +75,6 @@ export const useCleanupStore = defineStore('cleanup', () => {
defaults, recentRuns,
loadDefaults,
previewMinDim, deleteMinDim,
startAudit, getAudit, loadHistory, applyAudit, cancelAudit,
startAudit, getAudit, loadHistory, latestAuditForRule, applyAudit, cancelAudit,
}
})
+25
View File
@@ -0,0 +1,25 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useDbMaintenanceStore = defineStore('dbMaintenance', () => {
const api = useApi()
const tables = ref([])
const loading = ref(false)
async function loadStats() {
loading.value = true
try {
const body = await api.get('/api/admin/maintenance/db-stats')
tables.value = body.tables || []
} finally {
loading.value = false
}
}
async function runVacuum() {
return await api.post('/api/admin/maintenance/vacuum')
}
return { tables, loading, loadStats, runVacuum }
})
+186 -30
View File
@@ -3,12 +3,15 @@ import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { useInflightToken } from '../composables/useInflightToken.js'
// Operator-confirmed 2026-05-30: fetch PAGE-sized chunks instead of one
// 50-item request so items render as each batch lands. Total initial
// count is unchanged (PAGE * INITIAL_BATCHES = 50). Infinite-scroll also
// pulls PAGE per trigger to keep appends progressive.
const PAGE = 5
const INITIAL_BATCHES = 10
// Initial paint is a SINGLE request (INITIAL_LIMIT). The old 10×serial-
// batch loop (2026-05-30) only staggered METADATA, which isn't the visual
// bottleneck — thumbnails load as independent `<img>` requests, and
// GalleryItem now reveals each tile on its own image `@load`. One fetch is
// far fewer round-trips and faster to first paint; the reveal-on-load is
// what makes appearance progressive. Infinite scroll pulls PAGE per trigger.
// Reworked 2026-06-04.
const PAGE = 25
const INITIAL_LIMIT = 50
export const useGalleryStore = defineStore('gallery', () => {
const api = useApi()
@@ -18,7 +21,26 @@ export const useGalleryStore = defineStore('gallery', () => {
const nextCursor = ref(null)
const loading = ref(false)
const error = ref(null)
const filter = ref({ tag_id: null, post_id: null })
const filter = ref({
tag_ids: [], artist_id: null, media_type: null,
sort: 'newest', post_id: null,
// Phase-2 faceted refine params.
platform: null, untagged: false, no_artist: false,
date_from: null, date_to: null,
// Phase-3 visual similarity: when set, the gallery is in "similar mode" —
// ranked by cosine distance to this image, bounded top-N, no cursor.
similar_to: null,
})
// Live facet counts for the refine panel; fetched on-demand (panel open +
// filter change), never on plain scroll. Null until first load.
const facets = ref(null)
const facetsLoading = ref(false)
const facetsInflight = useInflightToken()
// Display names for the active filter chips — resolved by id on deep-link
// and pre-noted by the filter bar when a user picks from autocomplete.
const tagLabels = ref({}) // tagId -> name
const artistLabel = ref(null)
const timelineBuckets = ref([])
const timelineLoading = ref(false)
@@ -32,22 +54,16 @@ export const useGalleryStore = defineStore('gallery', () => {
images.value = []
dateGroups.value = []
nextCursor.value = null
// Sequentially fetch INITIAL_BATCHES chunks so items render as each
// batch lands rather than blocking on one big response. Stop early
// when the backend reports no more pages.
for (let i = 0; i < INITIAL_BATCHES; i++) {
if (i > 0 && nextCursor.value === null) break
await loadMore()
}
await loadMore(INITIAL_LIMIT)
}
async function loadMore() {
async function loadMore(limit = PAGE) {
if (loading.value) return
loading.value = true
error.value = null
const t = inflight.claim()
try {
const params = { limit: PAGE, ...activeFilterParam() }
const params = { limit, ...activeFilterParam() }
if (nextCursor.value) params.cursor = nextCursor.value
const body = await api.get('/api/gallery/scroll', { params })
if (!t.isCurrent()) return
@@ -71,6 +87,39 @@ export const useGalleryStore = defineStore('gallery', () => {
}
}
// Visual "more like this": ranked top-N by cosine distance, scope filters
// composed (AND). No cursor / no timeline — bounded result set.
async function loadSimilar() {
inflight.cancel()
images.value = []
dateGroups.value = []
nextCursor.value = null
timelineBuckets.value = []
loading.value = true
error.value = null
const t = inflight.claim()
try {
const f = filter.value
const params = { similar_to: f.similar_to, limit: 100 }
if (f.tag_ids.length) params.tag_id = f.tag_ids.join(',')
if (f.artist_id) params.artist_id = f.artist_id
if (f.media_type) params.media = f.media_type
if (f.platform) params.platform = f.platform
if (f.untagged) params.untagged = '1'
if (f.no_artist) params.no_artist = '1'
if (f.date_from) params.date_from = f.date_from
if (f.date_to) params.date_to = f.date_to
const body = await api.get('/api/gallery/similar', { params })
if (!t.isCurrent()) return
images.value = body.images
// ranked + bounded → no next page (nextCursor stays null → hasMore false)
} catch (e) {
error.value = e.message
} finally {
if (t.isCurrent()) loading.value = false
}
}
async function jumpTo(year, month) {
// Rapid timeline-jump clicks need the same race guard as
// loadMore — first jump's late body could clobber the second
@@ -89,23 +138,101 @@ export const useGalleryStore = defineStore('gallery', () => {
}
function activeFilterParam() {
if (filter.value.tag_id) return { tag_id: filter.value.tag_id }
// post_id is the exclusive post-detail view.
if (filter.value.post_id) return { post_id: filter.value.post_id }
return {}
const p = {}
if (filter.value.tag_ids.length) p.tag_id = filter.value.tag_ids.join(',')
if (filter.value.artist_id) p.artist_id = filter.value.artist_id
if (filter.value.media_type) p.media = filter.value.media_type
if (filter.value.sort && filter.value.sort !== 'newest') p.sort = filter.value.sort
if (filter.value.platform) p.platform = filter.value.platform
if (filter.value.untagged) p.untagged = '1'
if (filter.value.no_artist) p.no_artist = '1'
if (filter.value.date_from) p.date_from = filter.value.date_from
if (filter.value.date_to) p.date_to = filter.value.date_to
return p
}
function setTagFilter(tagId) {
filter.value.tag_id = tagId
filter.value.post_id = null
loadInitial()
loadTimeline()
// Live facet counts, scoped to the current filter (panel-gated — callers
// are the refine panel only). Single-flighted so rapid toggles don't let a
// stale response clobber a newer one.
async function loadFacets() {
facetsLoading.value = true
const t = facetsInflight.claim()
try {
const body = await api.get('/api/gallery/facets', { params: activeFilterParam() })
if (!t.isCurrent()) return
facets.value = body
} catch (e) {
if (t.isCurrent()) error.value = e.message
} finally {
if (t.isCurrent()) facetsLoading.value = false
}
}
function setPostFilter(postId) {
filter.value.post_id = postId
filter.value.tag_id = null
loadInitial()
loadTimeline()
// URL is the source of truth for filters. GalleryView calls this on mount
// and on every route-query change; the filter bar mutates the URL
// (router.push) rather than the store directly, so deep-links, the back
// button, and bar actions all funnel through one path.
async function applyFilterFromQuery(q) {
filter.value = {
tag_ids: _parseIds(q.tag_id),
artist_id: _toId(q.artist_id),
media_type: ['image', 'video'].includes(q.media) ? q.media : null,
sort: q.sort === 'oldest' ? 'oldest' : 'newest',
post_id: _toId(q.post_id),
platform: q.platform || null,
untagged: _truthy(q.untagged),
no_artist: _truthy(q.no_artist),
date_from: _parseDate(q.date_from),
date_to: _parseDate(q.date_to),
similar_to: _toId(q.similar_to),
}
if (filter.value.similar_to) {
// Similar mode: ranked, no timeline scroll.
await loadSimilar()
} else {
await loadInitial()
await loadTimeline()
}
_resolveLabels()
}
function _truthy(v) { return v === '1' || v === 'true' || v === true }
function _parseDate(v) {
return typeof v === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(v) ? v : null
}
function _toId(v) {
const n = Number(v)
return Number.isInteger(n) && n > 0 ? n : null
}
function _parseIds(raw) {
if (!raw) return []
return String(raw).split(',').map(Number).filter((n) => Number.isInteger(n) && n > 0)
}
// Pre-seed a label so a freshly-picked chip shows its name without a
// round-trip; the bar calls these before pushing the new URL.
function noteTagLabel(id, name) { tagLabels.value = { ...tagLabels.value, [id]: name } }
function noteArtistLabel(name) { artistLabel.value = name || null }
async function _resolveLabels() {
for (const id of filter.value.tag_ids) {
if (tagLabels.value[id]) continue
try {
const t = await api.get(`/api/tags/${id}`)
tagLabels.value = { ...tagLabels.value, [id]: t.name }
} catch { /* chip falls back to #id */ }
}
if (filter.value.artist_id && !artistLabel.value) {
// The filtered set is this artist — derive the chip label from a tile.
const hit = images.value.find(
(i) => i.artist && i.artist.id === filter.value.artist_id
)
artistLabel.value = hit?.artist?.name || null
}
if (!filter.value.artist_id) artistLabel.value = null
}
const hasMore = computed(() => nextCursor.value !== null)
@@ -113,11 +240,40 @@ export const useGalleryStore = defineStore('gallery', () => {
return {
images, dateGroups, hasMore, isEmpty, loading, error,
filter, timelineBuckets, timelineLoading,
loadInitial, loadMore, loadTimeline, jumpTo, setTagFilter, setPostFilter
filter, tagLabels, artistLabel, timelineBuckets, timelineLoading,
facets, facetsLoading,
loadInitial, loadMore, loadTimeline, jumpTo, loadFacets, loadSimilar,
applyFilterFromQuery, noteTagLabel, noteArtistLabel,
}
})
// Shared by GalleryFilterBar and GalleryFacetPanel so both write the URL in
// one format. post_id is intentionally absent — the bar/panel are hidden in
// the exclusive post-detail view.
export function cloneFilter(f) {
return {
tag_ids: [...f.tag_ids], artist_id: f.artist_id, media_type: f.media_type,
sort: f.sort, platform: f.platform, untagged: f.untagged,
no_artist: f.no_artist, date_from: f.date_from, date_to: f.date_to,
similar_to: f.similar_to,
}
}
export function filterToQuery(f) {
const q = {}
if (f.tag_ids?.length) q.tag_id = f.tag_ids.join(',')
if (f.artist_id) q.artist_id = String(f.artist_id)
if (f.media_type) q.media = f.media_type
if (f.sort && f.sort !== 'newest') q.sort = f.sort
if (f.platform) q.platform = f.platform
if (f.untagged) q.untagged = '1'
if (f.no_artist) q.no_artist = '1'
if (f.date_from) q.date_from = f.date_from
if (f.date_to) q.date_to = f.date_to
if (f.similar_to) q.similar_to = String(f.similar_to)
return q
}
function mergeGroups(existing, incoming) {
// Merge sequential groups with the same (year, month) instead of duplicating.
const merged = [...existing]
+3 -3
View File
@@ -18,9 +18,9 @@ export const useModalStore = defineStore('modal', () => {
const inflight = useInflightToken()
// Post-scoped cycle. When set, prev/next cycles within this array
// (used by PostCard's expanded-mosaic PostImageGrid clicks). When
// null, prev/next falls back to current.value.neighbors (the
// gallery-store-driven /api/gallery/image/<id> neighbors).
// (used by PostCard image clicks — the modal is scoped to that post's
// images). When null, prev/next falls back to current.value.neighbors
// (the gallery-store-driven /api/gallery/image/<id> neighbors).
const postImageIds = ref(null)
const postImageIndex = ref(0)
+124 -89
View File
@@ -1,37 +1,42 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { preloadImage } from '../utils/preloadImage.js'
// Operator-flagged 2026-05-30 (round 3): the all-parallel fetch was fast
// but risked later chunks arriving first — undesirable even when each
// chunk is a random sample. Switched to a PIPELINE: only one fetch in
// flight at any moment, but the next fetch kicks off as soon as the
// previous one resolves (NOT after its trickle finishes). The next RTT
// overlaps with the current batch's trickle, hiding the per-batch
// round-trip behind the visible animation cadence. Responses arrive in
// fire-order, so no out-of-order rendering surprises.
//
// Smaller PAGE (3 vs 5) → first chunk's items appear sooner: a chunk of
// 3 trickles in 240 ms, well within one RTT, so by the time chunk 2 is
// in-hand the trickle is just finishing. Total wall-clock is roughly
// RTT + N × max(trickle_time, RTT); APPEND_DELAY_MS keeps the visible
// cadence smooth throughout.
// Buffered producer/consumer so the cascade cadence is decoupled from fetch
// latency (operator-flagged 2026-06-04). The OLD pipeline trickled each batch
// right after its fetch and bet the next round-trip would finish inside the
// ~240ms trickle window; when a fetch ran long (TABLESAMPLE hits random,
// sometimes-cold blocks; RTT jitter) the animation starved and the view
// "burped" out a clump of images. Now:
// - PRODUCER (_fill): races ahead fetching batches into `queue` up to a
// target depth, refilling whenever the queue dips below BUFFER_MIN. It
// also kicks off the image PRELOAD for each queued item so decoding
// pipelines ahead of the reveal.
// - CONSUMER (_drain): pops ONE item, WAITS for its thumbnail to be fully
// decoded, then reveals it — at most one per CADENCE_MS. Because the
// producer preloads ahead, the decode-wait is usually already satisfied,
// so the reveal stays evenly paced without idling the network.
// The decode-gate is what makes the showcase's signature cascade land each
// tile fully-loaded (the flip-in animates a real image, never a gray
// placeholder); the single-item reveal keeps it strictly one-at-a-time
// (operator-flagged 2026-06-04). The very first image still waits on the first
// fetch + decode (a cold TABLESAMPLE is a separate, query-side concern);
// everything after it is buffer-smoothed.
const PAGE = 3
const INITIAL_BATCHES = 20
const APPEND_DELAY_MS = 80 // ≈ the MasonryGrid stagger animation (70 ms)
// Operator-flagged 2026-06-01: scrolling the showcase eventually hit a
// premature "End." because /api/showcase returns a *random sample* and
// after enough scrolling the `seen` Set accumulated enough to fully
// collide with a 3-item batch. The showcase is supposed to be endless;
// only a genuinely empty API response (library has zero images) should
// mark it exhausted. Retry up to FETCH_RETRY_CAP times on all-dupe
// batches; only flip `exhausted` when the API returns 0 items OR every
// retry came back dupe-only (graceful fallback for tiny libraries
// where retries will keep returning the same handful of items).
const CADENCE_MS = 160 // floor between fully-loaded reveals (doubled
// 2026-06-04 — slower, more deliberate cadence)
const PRIME = 6 // items buffered before the drain starts
const BUFFER_TARGET = 30 // producer tops the queue up to this
const BUFFER_MIN = 12 // ...and refills once the queue dips below this
const INITIAL_COUNT = 60 // cascade length on load / shuffle
const SCROLL_COUNT = 15 // cascade length per infinite-scroll demand
// Showcase is endless by design (random sample); an unlucky all-duplicate
// batch must be retried — only a genuinely empty API response is exhaustion.
const FETCH_RETRY_CAP = 8
function _sleep(ms) { return new Promise(r => setTimeout(r, ms)) }
function _sleep(ms) { return new Promise((r) => setTimeout(r, ms)) }
export const useShowcaseStore = defineStore('showcase', () => {
@@ -42,95 +47,125 @@ export const useShowcaseStore = defineStore('showcase', () => {
const exhausted = ref(false)
const seen = new Set()
// Sequence token: every call to loadInitial bumps this. _trickleAppend
// bails between items if its captured seq is no longer current — guards
// against a fast shuffle / mount-then-shuffle from interleaving two
// trickles into the same images.value.
// Internal buffer (not reactive — the consumer is what feeds the UI via
// images.value).
let queue = []
// id -> Promise that settles when the thumbnail is paint-ready. Started by
// the producer so decoding runs ahead of the reveal; awaited by the consumer
// so no tile is shown before its image is loaded.
let _preloads = new Map()
// Sequence token: shuffle / re-mount bumps it so in-flight producers and
// the drain bail instead of interleaving two runs into one images list.
let _seq = 0
let _filling = false
let _draining = false
async function _trickleAppend(items, mySeq) {
for (const item of items) {
if (mySeq !== _seq) return
if (seen.has(item.id)) continue
seen.add(item.id)
images.value.push(item)
await _sleep(APPEND_DELAY_MS)
}
function _preload(item) {
if (!_preloads.has(item.id)) _preloads.set(item.id, preloadImage(item.thumbnail_url))
return _preloads.get(item.id)
}
// Single batch — used by infinite-scroll appends. Trickles its items
// in for the same one-at-a-time cadence as the initial load. Retries
// up to FETCH_RETRY_CAP times when the API's random sample comes back
// all-duplicates (the showcase is endless by design; only a genuinely
// empty API response should mark it exhausted, not an unlucky sample).
async function fetchPage() {
if (loading.value) return
loading.value = true
error.value = null
// One batch, retried while the random sample comes back all-duplicates.
// Returns the fresh items, or null when the API is genuinely empty.
async function _fetchFresh(mySeq) {
for (let attempt = 0; attempt < FETCH_RETRY_CAP; attempt++) {
if (mySeq !== _seq) return []
const body = await api
.get('/api/showcase', { params: { limit: PAGE } })
.catch((e) => {
error.value = error.value || (e.message || String(e))
return null
})
if (mySeq !== _seq) return []
const items = (body && body.images) || []
if (items.length === 0) return null
const fresh = items.filter((i) => !seen.has(i.id))
fresh.forEach((f) => seen.add(f.id))
if (fresh.length) return fresh
}
return null // retry cap hit → tiny library, treat as exhausted
}
// Producer: top the buffer up to `target`. Single-flight.
async function _fill(mySeq, target) {
if (_filling) return
_filling = true
try {
for (let attempt = 0; attempt < FETCH_RETRY_CAP; attempt++) {
const body = await api.get('/api/showcase', { params: { limit: PAGE } })
const items = body.images || []
// API genuinely empty → library is empty / endpoint exhausted.
if (items.length === 0) { exhausted.value = true; return }
const fresh = items.filter(i => !seen.has(i.id))
if (fresh.length > 0) {
await _trickleAppend(fresh, _seq)
return
}
// All-dupes batch — keep trying. Showcase is endless by intent.
while (mySeq === _seq && !exhausted.value && queue.length < target) {
const batch = await _fetchFresh(mySeq)
if (mySeq !== _seq) return
if (batch === null) { exhausted.value = true; return }
queue.push(...batch)
batch.forEach(_preload) // pipeline decoding ahead of the reveal
}
// Retry cap hit with zero fresh items: library is probably much
// smaller than the running `seen` set, fall back to exhausted so
// the UI stops trying. Operator can shuffle to reset `seen`.
exhausted.value = true
} catch (e) {
error.value = e.message || String(e)
} finally {
loading.value = false
_filling = false
}
}
function _fetchOne() {
return api.get('/api/showcase', { params: { limit: PAGE } }).catch(e => {
error.value = error.value || (e.message || String(e))
return null
})
// Consumer: show `count` items at a fixed cadence, topping up the buffer as
// it drains. Single-flight so the initial cascade and scroll appends can't
// interleave.
async function _drain(mySeq, count) {
if (_draining) return
_draining = true
loading.value = true
try {
for (let shown = 0; shown < count && mySeq === _seq; shown++) {
if (!exhausted.value && queue.length < BUFFER_MIN) {
_fill(mySeq, BUFFER_TARGET) // topup, no await
}
let guard = 0
while (queue.length === 0 && !exhausted.value && mySeq === _seq) {
await _sleep(20)
if (++guard > 500) break // 10s starvation safety net
}
if (mySeq !== _seq) return
if (queue.length === 0) return // exhausted + empty
const item = queue.shift()
await _preload(item) // reveal only once the thumbnail is fully decoded
if (mySeq !== _seq) return
images.value.push(item)
await _sleep(CADENCE_MS)
}
} finally {
if (mySeq === _seq) {
_draining = false
loading.value = false
}
}
}
// Reset state and pipeline INITIAL_BATCHES fetches: only one in flight
// at a time, but kick off the next one as soon as the previous resolves
// (NOT after its trickle finishes), so the next RTT runs concurrently
// with the current batch's trickle. Responses arrive in fire-order, so
// items always render in the order they were fetched — no out-of-order
// surprises from parallel races.
async function loadInitial() {
_seq += 1
const mySeq = _seq
images.value = []
queue = []
_preloads = new Map()
seen.clear()
exhausted.value = false
error.value = null
_filling = false
_draining = false
loading.value = true
try {
let nextFetch = _fetchOne()
for (let i = 0; i < INITIAL_BATCHES; i++) {
if (mySeq !== _seq) return
const body = await nextFetch
// Fire the NEXT fetch immediately so its RTT overlaps the trickle.
if (i + 1 < INITIAL_BATCHES) nextFetch = _fetchOne()
if (!body || !body.images || body.images.length === 0) {
exhausted.value = true
break
}
await _trickleAppend(body.images, mySeq)
}
if (mySeq === _seq && images.value.length === 0) exhausted.value = true
// Prime a small buffer before the cascade starts so it doesn't starve
// at the front; the drain's own topup grows it to BUFFER_TARGET.
await _fill(mySeq, PRIME)
if (mySeq !== _seq) return
if (queue.length === 0 && exhausted.value) return // empty library
await _drain(mySeq, INITIAL_COUNT)
} finally {
if (mySeq === _seq) loading.value = false
}
}
async function fetchPage() {
// Infinite-scroll demand — append another cascade of SCROLL_COUNT.
if (_draining || exhausted.value) return
await _drain(_seq, SCROLL_COUNT)
}
async function shuffle() {
await loadInitial()
}
+21 -6
View File
@@ -85,11 +85,24 @@ export const useSourcesStore = defineStore('sources', () => {
}
}
// Plan #544: arm a source for backfill mode. The next `runs` download
// runs (default 3) walk gallery-dl's full post history instead of
// exiting early at the first contiguous archived block.
async function setBackfill(id, runs = 3, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { runs } })
// Plan #693: start/stop a run-until-done backfill. 'start' walks the full
// post history in time-boxed chunks until it reaches the bottom (then the
// source shows backfill_state 'complete'); 'stop' cancels back to tick mode.
async function startBackfill(id, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'start' } })
_invalidate(artistIdHint ?? body.artist_id)
return body
}
async function stopBackfill(id, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'stop' } })
_invalidate(artistIdHint ?? body.artist_id)
return body
}
// Plan #697: arm a recovery walk — a backfill that bypasses the Patreon
// seen-ledger to re-fetch dropped-and-deleted near-dups and re-evaluate them
// under the current pHash threshold. Stop with stopBackfill (shared lifecycle).
async function recoverSource(id, artistIdHint = null) {
const body = await api.post(`/api/sources/${id}/backfill`, { body: { action: 'recover' } })
_invalidate(artistIdHint ?? body.artist_id)
return body
}
@@ -120,7 +133,9 @@ export const useSourcesStore = defineStore('sources', () => {
loadAll, loadForArtist,
create, update, remove,
checkNow,
setBackfill,
startBackfill,
stopBackfill,
recoverSource,
findOrCreateArtist, autocompleteArtist,
loadScheduleStatus,
sourcesByArtistGrouped,
+14 -1
View File
@@ -49,8 +49,21 @@ export const useTagStore = defineStore('tags', () => {
return fandom
}
// Set / change / clear a character tag's fandom. fandomId null clears it.
// Throws ApiError (status 409, body.target) on a name collision in the
// target fandom; pass { merge: true } to resolve it by merging this tag
// into the existing character. Returns the updated/surviving tag.
async function setFandom(tagId, fandomId, { merge = false } = {}) {
const body = { fandom_id: fandomId ?? null }
if (merge) body.merge = true
return await api.patch(`/api/tags/${tagId}`, { body })
}
function kindOptions() { return KIND_OPTIONS }
function colorFor(kind) { return KIND_COLOR[kind] || 'on-surface' }
return { fandomCache, autocomplete, loadFandoms, createFandom, kindOptions, colorFor }
return {
fandomCache, autocomplete, loadFandoms, createFandom, setFandom,
kindOptions, colorFor
}
})
+21
View File
@@ -0,0 +1,21 @@
// Resolve once the image at `url` is fully loaded AND decoded (paint-ready),
// or after `timeoutMs` as a safety net so a broken/slow image never stalls a
// cascade. Never rejects — the caller only cares that it's safe to reveal.
//
// Used by the showcase cascade to gate each tile's entry animation on the
// thumbnail actually being ready, so the flip-in plays on a real image rather
// than on a gray placeholder (operator-flagged 2026-06-04).
export function preloadImage (url, timeoutMs = 4000) {
return new Promise((resolve) => {
let done = false
const finish = () => { if (!done) { done = true; resolve() } }
const img = new Image()
img.onload = finish
img.onerror = finish
img.src = url
// decode() resolves at paint-ready (a beat after onload); prefer it when
// available, but onload/onerror/timeout all still settle the promise.
if (typeof img.decode === 'function') img.decode().then(finish).catch(() => {})
setTimeout(finish, timeoutMs)
})
}
+21
View File
@@ -0,0 +1,21 @@
// Helpers for deciding whether a keyboard shortcut should fire or yield to a
// focused text field.
export function isTextEntry (el) {
if (!el) return false
const tag = el.tagName
return tag === 'INPUT' || tag === 'TEXTAREA' || el.isContentEditable
}
export function hasText (el) {
if (!el) return false
if (el.isContentEditable) return (el.textContent || '').length > 0
return (el.value || '').length > 0
}
// Prev/next arrow navigation should fire UNLESS focus is in a text entry that
// already has content — in that case the arrows belong to the caret so it can
// move through the text. An empty (or non-text) target still navigates.
export function arrowNavAllowed (target) {
return !(isTextEntry(target) && hasText(target))
}
+25 -1
View File
@@ -20,6 +20,22 @@
:last-added="store.lastAdded"
/>
<v-container fluid class="pt-2 pb-4">
<!-- "N new since last visit" banner. Visible only on the initial
load that triggered the visit-mark; dismissable via close
button or by switching tabs. Re-entry only re-shows if more
content has arrived (overview returns 0 immediately after a
previous visit). -->
<v-alert
v-if="unseenBanner"
type="info" variant="tonal" density="compact"
class="mb-3" closable
@click:close="unseenBanner = false"
>
<span class="fc-artist__unseen-msg">
<strong>{{ store.overview.unseen_count_at_visit }}</strong>
new since last visit
</span>
</v-alert>
<v-window v-model="tab">
<v-window-item value="posts">
<ArtistPostsTab
@@ -39,7 +55,7 @@
</template>
<script setup>
import { computed, watch } from 'vue'
import { computed, ref, watch } from 'vue'
import { useRoute } from 'vue-router'
import { useArtistStore } from '../stores/artist.js'
@@ -60,6 +76,10 @@ const { tab, resolve } = useTabQuery(
() => ((store.postCount ?? 0) > 0 ? 'posts' : 'gallery'),
)
// One-shot banner — reset on each new artist-slug load so it re-appears
// when navigating between artists that each have unseen content.
const unseenBanner = ref(false)
watch(slug, async (s) => {
if (!s) return
await store.load(s)
@@ -67,6 +87,7 @@ watch(slug, async (s) => {
? `${store.overview.name} — FabledCurator`
: 'FabledCurator'
tab.value = resolve()
unseenBanner.value = (store.overview?.unseen_count_at_visit || 0) > 0
}, { immediate: true })
</script>
@@ -74,4 +95,7 @@ watch(slug, async (s) => {
.fc-artist__loading {
display: flex; justify-content: center; padding: 64px 0;
}
.fc-artist__unseen-msg {
font-size: 14px;
}
</style>
+3 -1
View File
@@ -76,7 +76,9 @@ onMounted(async () => {
}
.fc-artists__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
/* min(440px, 100%) so a card never exceeds the viewport — on phones the
min-track collapses to 100% (single column) instead of overflowing. */
grid-template-columns: repeat(auto-fill, minmax(min(440px, 100%), 1fr));
gap: 12px;
}
.fc-artists__sentinel {
+16 -18
View File
@@ -12,10 +12,14 @@
<div class="fc-gallery-layout">
<div class="fc-gallery-layout__main">
<PostInfoHeader />
<GalleryFilterBar v-if="store.filter.post_id == null" />
<EmptyState v-if="store.isEmpty" />
<GalleryGrid v-else @open="openImage" />
</div>
<TimelineSidebar v-if="store.images.length > 0" class="fc-gallery-layout__sidebar" />
<TimelineSidebar
v-if="store.images.length > 0 && store.filter.similar_to == null"
class="fc-gallery-layout__sidebar"
/>
</div>
<BulkEditorPanel />
@@ -28,6 +32,7 @@ import { useRoute } from 'vue-router'
import { useGalleryStore } from '../stores/gallery.js'
import { useModalStore } from '../stores/modal.js'
import GalleryGrid from '../components/gallery/GalleryGrid.vue'
import GalleryFilterBar from '../components/gallery/GalleryFilterBar.vue'
import TimelineSidebar from '../components/gallery/TimelineSidebar.vue'
import EmptyState from '../components/gallery/EmptyState.vue'
import PostInfoHeader from '../components/gallery/PostInfoHeader.vue'
@@ -39,25 +44,13 @@ const modal = useModalStore()
const sel = useGallerySelectionStore()
const route = useRoute()
onMounted(async () => {
const postId = parseInt(route.query.post_id, 10)
const tagId = parseInt(route.query.tag_id, 10)
if (!isNaN(postId)) store.setPostFilter(postId)
else if (!isNaN(tagId)) store.setTagFilter(tagId)
await store.loadInitial()
await store.loadTimeline()
})
// The URL query is the single source of truth for filters. Apply it on
// mount and on any query change (filter bar pushes, back button, deep-link).
onMounted(() => store.applyFilterFromQuery(route.query))
watch(() => route.query.tag_id, (q) => {
watch(() => route.query, (q) => {
sel.clear() // result set changed — selected ids are no longer valid
const tagId = parseInt(q, 10)
store.setTagFilter(isNaN(tagId) ? null : tagId)
})
watch(() => route.query.post_id, (q) => {
sel.clear() // result set changed — selected ids are no longer valid
const postId = parseInt(q, 10)
store.setPostFilter(isNaN(postId) ? null : postId)
store.applyFilterFromQuery(q)
})
function openImage(id) {
@@ -78,4 +71,9 @@ function openImage(id) {
.fc-gallery-layout { flex-direction: column-reverse; }
.fc-gallery-layout__sidebar { width: 100%; max-height: 200px; }
}
/* Phones: the year/month timeline strip eats vertical space and reads poorly
as a horizontal band — drop it; the gallery scroll is the primary nav here. */
@media (max-width: 600px) {
.fc-gallery-layout__sidebar { display: none; }
}
</style>
+6 -5
View File
@@ -44,14 +44,15 @@
</v-window-item>
<v-window-item value="import">
<!-- Order: trigger → recent tasks → filters. Tasks sit directly
below the trigger so operator sees hit/miss feedback without
scrolling past the filter card (operator-flagged 2026-05-25). -->
<!-- Order: filters → trigger → recent tasks. Filters hoisted above the
trigger (operator-flagged 2026-06-04); the task list stays
directly below the trigger so hit/miss feedback is adjacent to the
button that produced it (operator-flagged 2026-05-25). -->
<ImportFiltersForm />
<v-divider class="my-6" />
<ImportTriggerPanel />
<v-divider class="my-6" />
<ImportTaskList />
<v-divider class="my-6" />
<ImportFiltersForm />
</v-window-item>
<v-window-item value="cleanup">
+22 -1
View File
@@ -8,7 +8,7 @@
placeholder="Search tags" clearable style="max-width: 320px;"
/>
<v-chip-group
v-model="kind" selected-class="text-accent" mandatory="false"
v-model="kind" selected-class="text-accent" :mandatory="false"
>
<v-chip v-for="k in KINDS" :key="k" :value="k" filter size="small">
{{ k }}
@@ -28,6 +28,7 @@
v-for="c in store.cards" :key="c.id" :card="c"
@open="openTag" @rename="onRename" @manage="onManage" @read="onRead"
@merge-with="onMergeWith" @delete="onDeleteTag"
@set-fandom="onSetFandom"
/>
</div>
@@ -85,6 +86,13 @@
: ''"
@confirm="onDeleteTagConfirm"
/>
<v-dialog v-model="fandomDialogOpen" max-width="460">
<FandomSetDialog
v-if="fandomTarget" :tag="fandomTarget"
@updated="onFandomUpdated" @cancel="fandomDialogOpen = false"
/>
</v-dialog>
</v-container>
</template>
@@ -98,6 +106,7 @@ import { useInfiniteScroll } from '../composables/useInfiniteScroll.js'
import TagCard from '../components/discovery/TagCard.vue'
import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue'
import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue'
import FandomSetDialog from '../components/modal/FandomSetDialog.vue'
// Must stay a subset of the backend TagKind enum (character, fandom,
// general, series, archive, post). 'fandom' is this model's
@@ -140,6 +149,18 @@ function openTag(tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } })
}
// Character fandom editing (dots-menu → FandomSetDialog).
const fandomDialogOpen = ref(false)
const fandomTarget = ref(null)
function onSetFandom(card) {
fandomTarget.value = card
fandomDialogOpen.value = true
}
function onFandomUpdated() {
fandomDialogOpen.value = false
store.reset() // reload so the card reflects the new fandom (or its removal)
}
function onManage(id) {
router.push({ name: 'series-manage', params: { tagId: id } })
}
@@ -0,0 +1,53 @@
// @vitest-environment happy-dom
import { describe, it, expect, vi, afterEach } from 'vitest'
import GalleryFacetPanel from '../../src/components/gallery/GalleryFacetPanel.vue'
import { useGalleryStore } from '../../src/stores/gallery.js'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
const FACETS = {
total: 5,
platforms: [
{ value: 'patreon', count: 3 },
{ value: null, count: 2 }, // unsourced → "No platform" bucket
],
untagged: 4,
no_artist: 1,
date_min: '2020-01-01T00:00:00+00:00',
date_max: '2026-06-01T00:00:00+00:00',
}
function stubFetchOk(body) {
globalThis.fetch = vi.fn(async () => ({
ok: true, status: 200, statusText: 'OK',
text: async () => JSON.stringify(body),
}))
}
describe('GalleryFacetPanel', () => {
afterEach(() => vi.restoreAllMocks())
it('renders platform options with counts and the curation flags', () => {
const pinia = freshPinia()
stubFetchOk(FACETS) // covers the onMounted loadFacets() call
const s = useGalleryStore()
s.facets = FACETS // seed so counts render synchronously
const w = mountComponent(GalleryFacetPanel, { pinia })
const t = w.text()
expect(t).toContain('patreon')
expect(t).toContain('3')
expect(t).toContain('No platform')
expect(t).toContain('Untagged')
expect(t).toContain('4')
expect(t).toContain('No artist')
})
it('shows an en-dash placeholder for a flag count before facets load', () => {
const pinia = freshPinia()
stubFetchOk(FACETS)
useGalleryStore().facets = null // not loaded yet
const w = mountComponent(GalleryFacetPanel, { pinia })
expect(w.text()).toContain('Untagged')
expect(w.text()).toContain('')
})
})
@@ -0,0 +1,47 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import { mount } from '@vue/test-utils'
import GalleryGrid from '../../src/components/gallery/GalleryGrid.vue'
import { useGalleryStore } from '../../src/stores/gallery.js'
import { freshPinia } from '../support/mountComponent.js'
const GIStub = { name: 'GalleryItem', props: ['image'], template: '<div class="gi" />' }
function mountGrid(pinia) {
return mount(GalleryGrid, {
global: {
plugins: [pinia],
stubs: { GalleryItem: GIStub, RouterLink: { template: '<a><slot /></a>' } },
},
})
}
describe('GalleryGrid', () => {
it('renders a flat ranked list when there are no date groups (similar mode)', () => {
const pinia = freshPinia()
const store = useGalleryStore()
// similar-mode shape: images present, date_groups empty, no cursor.
store.images = [
{ id: 1, thumbnail_url: '/a' },
{ id: 2, thumbnail_url: '/b' },
{ id: 3, thumbnail_url: '/c' },
]
store.dateGroups = []
const w = mountGrid(pinia)
expect(w.findAll('.gi').length).toBe(3)
})
it('renders grouped by date when date groups are present', () => {
const pinia = freshPinia()
const store = useGalleryStore()
store.images = [
{ id: 1, thumbnail_url: '/a' },
{ id: 2, thumbnail_url: '/b' },
]
store.dateGroups = [{ year: 2026, month: 6, image_ids: [1, 2] }]
const w = mountGrid(pinia)
expect(w.find('.fc-gallery-grid__date-header').exists()).toBe(true)
expect(w.findAll('.gi').length).toBe(2)
})
})
@@ -0,0 +1,28 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import GalleryItem from '../../src/components/gallery/GalleryItem.vue'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
describe('GalleryItem', () => {
const image = {
id: 7, sha256: 'a'.repeat(64), mime: 'image/jpeg',
width: 100, height: 100,
thumbnail_url: '/images/thumbs/aa/abc.jpg', artist: null,
}
it('reveals the thumbnail only once it has loaded, so tiles fade in individually', async () => {
const pinia = freshPinia()
const w = mountComponent(GalleryItem, { props: { image }, pinia })
const img = w.find('img')
expect(img.exists()).toBe(true)
// Pre-load: the tile must NOT be revealed, so it can fade in when its
// own thumbnail lands rather than pop together with its API batch.
expect(img.classes()).not.toContain('is-loaded')
await img.trigger('load')
expect(img.classes()).toContain('is-loaded')
})
})
@@ -0,0 +1,73 @@
// @vitest-environment happy-dom
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import { flushPromises } from '@vue/test-utils'
import RelatedStrip from '../../src/components/modal/RelatedStrip.vue'
import { useModalStore } from '../../src/stores/modal.js'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
function stubFetch(handler) {
globalThis.fetch = vi.fn(async (url) => {
const { status, body } = handler(url)
return {
ok: status >= 200 && status < 300,
status, statusText: String(status),
text: async () => JSON.stringify(body),
}
})
}
describe('RelatedStrip (modal "more like this")', () => {
beforeEach(() => vi.useFakeTimers())
afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks() })
it('stays hidden and never fetches when the image has no embedding', async () => {
const pinia = freshPinia()
useModalStore().current = { id: 1, has_embedding: false }
const fetchSpy = vi.fn()
globalThis.fetch = fetchSpy
const w = mountComponent(RelatedStrip, { pinia })
await vi.advanceTimersByTimeAsync(500)
expect(fetchSpy).not.toHaveBeenCalled()
expect(w.find('.fc-related').exists()).toBe(false)
})
it('deferred-fetches and renders similar thumbs for an embedded image', async () => {
const pinia = freshPinia()
useModalStore().current = { id: 1, has_embedding: true }
stubFetch(() => ({
status: 200,
body: { images: [{ id: 2, thumbnail_url: '/t2' }, { id: 3, thumbnail_url: '/t3' }] },
}))
const w = mountComponent(RelatedStrip, { pinia })
// Nothing before the defer elapses (modal image gets priority).
expect(globalThis.fetch).not.toHaveBeenCalled()
await vi.advanceTimersByTimeAsync(250)
await flushPromises()
expect(globalThis.fetch).toHaveBeenCalledTimes(1)
expect(w.findAll('.fc-related__item').length).toBe(2)
})
it('collapses silently when the similar fetch returns nothing', async () => {
const pinia = freshPinia()
useModalStore().current = { id: 1, has_embedding: true }
stubFetch(() => ({ status: 200, body: { images: [] } }))
const w = mountComponent(RelatedStrip, { pinia })
await vi.advanceTimersByTimeAsync(250)
await flushPromises()
expect(w.find('.fc-related').exists()).toBe(false)
})
it('clicking a thumb opens that image in the modal', async () => {
const pinia = freshPinia()
const modal = useModalStore()
modal.current = { id: 1, has_embedding: true }
const openSpy = vi.spyOn(modal, 'open').mockResolvedValue()
stubFetch(() => ({ status: 200, body: { images: [{ id: 7, thumbnail_url: '/t7' }] } }))
const w = mountComponent(RelatedStrip, { pinia })
await vi.advanceTimersByTimeAsync(250)
await flushPromises()
await w.find('.fc-related__item').trigger('click')
expect(openSpy).toHaveBeenCalledWith(7)
})
})
+48 -13
View File
@@ -1,26 +1,61 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import { describe, it, expect, vi } from 'vitest'
import { flushPromises } from '@vue/test-utils'
import PostCard from '../../src/components/posts/PostCard.vue'
import { useModalStore } from '../../src/stores/modal.js'
import { freshPinia, mountComponent } from '../support/mountComponent.js'
const now = new Date().toISOString()
const BASE = {
id: 1, external_post_id: 'P1',
post_title: '<strong>Hello World</strong>',
post_url: 'https://x/1', post_date: now, downloaded_at: now,
artist: { id: 1, name: 'Sabu', slug: 'sabu' },
source: { id: 1, platform: 'subscribestar' },
thumbnails: [], thumbnails_more: 0, attachments: [],
}
describe('PostCard', () => {
it('renders the HTML-stripped title and the artist', () => {
const pinia = freshPinia()
const now = new Date().toISOString()
const post = {
id: 1, external_post_id: 'P1',
post_title: '<strong>Hello World</strong>',
post_url: 'https://x/1', post_date: now, downloaded_at: now,
description_plain: 'a description',
artist: { id: 1, name: 'Sabu', slug: 'sabu' },
source: { id: 1, platform: 'subscribestar' },
thumbnails: [], thumbnails_more: 0, attachments: [],
}
const w = mountComponent(PostCard, { props: { post }, pinia })
const post = { ...BASE, description_plain: 'a description' }
const w = mountComponent(PostCard, { props: { post }, pinia: freshPinia() })
const t = w.text()
expect(t).toContain('Hello World') // plain title
expect(t).not.toContain('<strong>') // tags stripped
expect(t).toContain('Sabu') // artist (RouterLink slot)
})
it('shows a "Show more" toggle only when the description is truncated', () => {
const truncated = mountComponent(PostCard, {
props: { post: { ...BASE, description_plain: 'lots of text…', description_truncated: true } },
pinia: freshPinia(),
})
expect(truncated.text()).toContain('Show more')
const full = mountComponent(PostCard, {
props: { post: { ...BASE, description_plain: 'short', description_truncated: false } },
pinia: freshPinia(),
})
expect(full.text()).not.toContain('Show more')
})
it('clicking an image opens the post-scoped modal (never expands the card)', async () => {
const pinia = freshPinia()
const modal = useModalStore()
const openSpy = vi.spyOn(modal, 'open').mockResolvedValue()
const post = {
...BASE,
description_plain: 'd',
thumbnails: [
{ image_id: 10, thumbnail_url: '/a' },
{ image_id: 11, thumbnail_url: '/b' },
],
thumbnails_more: 0,
}
const w = mountComponent(PostCard, { props: { post }, pinia })
await w.find('.fc-post-card__hero').trigger('click')
await flushPromises()
expect(openSpy).toHaveBeenCalledWith(10, { postImageIds: [10, 11] })
})
})
+44
View File
@@ -0,0 +1,44 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
import { useDbMaintenanceStore } from '../src/stores/dbMaintenance.js'
function stubFetch(handler) {
globalThis.fetch = vi.fn(async (url, init) => {
const { status, body } = handler(url, init)
return {
ok: status >= 200 && status < 300,
status,
statusText: String(status),
text: async () => (body == null ? '' : JSON.stringify(body)),
}
})
}
describe('dbMaintenance store', () => {
beforeEach(() => setActivePinia(createPinia()))
afterEach(() => vi.restoreAllMocks())
it('loadStats populates the bloat table', async () => {
const s = useDbMaintenanceStore()
stubFetch(() => ({
status: 200,
body: { tables: [{ table: 'image_record', live: 5, dead: 1, dead_pct: 16.7, last_vacuum: null }] },
}))
await s.loadStats()
expect(s.tables).toHaveLength(1)
expect(s.tables[0].table).toBe('image_record')
})
it('runVacuum POSTs the trigger endpoint', async () => {
const s = useDbMaintenanceStore()
const calls = []
stubFetch((url, init) => {
calls.push({ url, init })
return { status: 202, body: { status: 'queued' } }
})
await s.runVacuum()
const c = calls.at(-1)
expect(c.url).toContain('/api/admin/maintenance/vacuum')
expect(c.init.method).toBe('POST')
})
})
+170 -25
View File
@@ -1,6 +1,6 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
import { useGalleryStore } from '../src/stores/gallery.js'
import { cloneFilter, filterToQuery, useGalleryStore } from '../src/stores/gallery.js'
function stubFetch(handler) {
globalThis.fetch = vi.fn(async (url, init) => {
@@ -9,44 +9,189 @@ function stubFetch(handler) {
ok: status >= 200 && status < 300,
status,
statusText: String(status),
text: async () => (body == null ? '' : JSON.stringify(body))
text: async () => (body == null ? '' : JSON.stringify(body)),
}
})
}
const EMPTY = { images: [], date_groups: [], next_cursor: null }
describe('gallery store: tag/post filter exclusivity', () => {
describe('gallery store: composable filter', () => {
beforeEach(() => setActivePinia(createPinia()))
afterEach(() => vi.restoreAllMocks())
it('setPostFilter sets post_id and clears tag_id', () => {
it('applyFilterFromQuery parses the query and loadMore sends composable params', async () => {
const s = useGalleryStore()
stubFetch(() => ({ status: 200, body: EMPTY }))
s.setTagFilter(3)
expect(s.filter.tag_id).toBe(3)
s.setPostFilter(7)
expect(s.filter.post_id).toBe(7)
expect(s.filter.tag_id).toBe(null)
const urls = []
stubFetch((url) => {
urls.push(url)
if (url.includes('/api/tags/')) {
return { status: 200, body: { id: 1, name: 'X', kind: 'general' } }
}
return { status: 200, body: EMPTY }
})
await s.applyFilterFromQuery({ tag_id: '1,2', artist_id: '5', media: 'video', sort: 'oldest' })
expect(s.filter.tag_ids).toEqual([1, 2])
expect(s.filter.artist_id).toBe(5)
expect(s.filter.media_type).toBe('video')
expect(s.filter.sort).toBe('oldest')
const scroll = decodeURIComponent(urls.find((u) => u.includes('/api/gallery/scroll')))
expect(scroll).toContain('tag_id=1,2')
expect(scroll).toContain('artist_id=5')
expect(scroll).toContain('media=video')
expect(scroll).toContain('sort=oldest')
})
it('setTagFilter clears post_id', () => {
const s = useGalleryStore()
stubFetch(() => ({ status: 200, body: EMPTY }))
s.setPostFilter(7)
s.setTagFilter(3)
expect(s.filter.tag_id).toBe(3)
expect(s.filter.post_id).toBe(null)
})
it('loadMore sends exactly the active filter param', async () => {
it('omits sort=newest and sends no filter params when empty', async () => {
const s = useGalleryStore()
const urls = []
stubFetch((url) => { urls.push(url); return { status: 200, body: EMPTY } })
s.setPostFilter(7)
await s.loadMore()
const scrollCall = urls.filter(u => u.includes('/api/gallery/scroll')).pop()
expect(scrollCall).toContain('post_id=7')
expect(scrollCall).not.toContain('tag_id=')
await s.applyFilterFromQuery({})
const scroll = urls.find((u) => u.includes('/api/gallery/scroll'))
expect(scroll).not.toContain('sort=')
expect(scroll).not.toContain('tag_id=')
expect(scroll).not.toContain('media=')
})
it('treats post_id as the exclusive filter param', async () => {
const s = useGalleryStore()
const urls = []
stubFetch((url) => { urls.push(url); return { status: 200, body: EMPTY } })
await s.applyFilterFromQuery({ post_id: '7', tag_id: '1' })
expect(s.filter.post_id).toBe(7)
const scroll = urls.find((u) => u.includes('/api/gallery/scroll'))
expect(scroll).toContain('post_id=7')
expect(scroll).not.toContain('tag_id=')
})
it('noteTagLabel and noteArtistLabel pre-seed chip labels', () => {
const s = useGalleryStore()
s.noteTagLabel(9, 'Rukia')
expect(s.tagLabels[9]).toBe('Rukia')
s.noteArtistLabel('Kubo')
expect(s.artistLabel).toBe('Kubo')
})
it('parses faceted refine params and loadMore forwards them', async () => {
const s = useGalleryStore()
const urls = []
stubFetch((url) => { urls.push(url); return { status: 200, body: EMPTY } })
await s.applyFilterFromQuery({
platform: 'pixiv', untagged: '1', no_artist: '1',
date_from: '2024-01-01', date_to: '2024-12-31',
})
expect(s.filter.platform).toBe('pixiv')
expect(s.filter.untagged).toBe(true)
expect(s.filter.no_artist).toBe(true)
expect(s.filter.date_from).toBe('2024-01-01')
expect(s.filter.date_to).toBe('2024-12-31')
const scroll = decodeURIComponent(urls.find((u) => u.includes('/api/gallery/scroll')))
expect(scroll).toContain('platform=pixiv')
expect(scroll).toContain('untagged=1')
expect(scroll).toContain('no_artist=1')
expect(scroll).toContain('date_from=2024-01-01')
expect(scroll).toContain('date_to=2024-12-31')
})
it('drops a malformed date in the query', async () => {
const s = useGalleryStore()
stubFetch(() => ({ status: 200, body: EMPTY }))
await s.applyFilterFromQuery({ date_from: 'garbage' })
expect(s.filter.date_from).toBe(null)
})
it('loadFacets fetches counts scoped to the active filter', async () => {
const s = useGalleryStore()
const urls = []
const FACETS = {
total: 4, platforms: [{ value: 'patreon', count: 2 }],
untagged: 1, no_artist: 0,
date_min: '2020-01-01T00:00:00+00:00', date_max: '2026-06-01T00:00:00+00:00',
}
stubFetch((url) => {
urls.push(url)
if (url.includes('/api/gallery/facets')) return { status: 200, body: FACETS }
return { status: 200, body: EMPTY }
})
await s.applyFilterFromQuery({ platform: 'patreon', untagged: '1' })
await s.loadFacets()
const facetsUrl = decodeURIComponent(urls.find((u) => u.includes('/api/gallery/facets')))
expect(facetsUrl).toContain('platform=patreon')
expect(facetsUrl).toContain('untagged=1')
expect(s.facets.total).toBe(4)
})
it('similar_to routes applyFilterFromQuery to the /similar endpoint, not scroll', async () => {
const s = useGalleryStore()
const urls = []
stubFetch((url) => {
urls.push(url)
if (url.includes('/api/gallery/similar')) {
return { status: 200, body: { images: [{ id: 9 }], next_cursor: null, date_groups: [] } }
}
return { status: 200, body: EMPTY }
})
await s.applyFilterFromQuery({ similar_to: '5', tag_id: '3' })
expect(s.filter.similar_to).toBe(5)
const sim = decodeURIComponent(urls.find((u) => u.includes('/api/gallery/similar')))
expect(sim).toContain('similar_to=5')
expect(sim).toContain('limit=100')
expect(sim).toContain('tag_id=3') // scope composes
expect(s.images.map((i) => i.id)).toEqual([9])
expect(s.hasMore).toBe(false) // ranked + bounded → no pages
expect(urls.some((u) => u.includes('/api/gallery/scroll'))).toBe(false)
expect(urls.some((u) => u.includes('/api/gallery/timeline'))).toBe(false)
})
it('loadInitial issues exactly one scroll request at the initial limit', async () => {
const s = useGalleryStore()
const urls = []
// Non-null cursor: the old 10×serial-batch loop would have fired ten
// requests here. The single-fetch path must fire exactly one.
stubFetch((url) => {
urls.push(url)
return { status: 200, body: { images: [], date_groups: [], next_cursor: 'c1' } }
})
await s.loadInitial()
const scrollCalls = urls.filter((u) => u.includes('/api/gallery/scroll'))
expect(scrollCalls).toHaveLength(1)
expect(scrollCalls[0]).toContain('limit=50')
})
})
describe('filterToQuery / cloneFilter', () => {
it('serializes faceted params incl. platform sentinel and flags', () => {
const q = filterToQuery({
tag_ids: [3], artist_id: null, media_type: null, sort: 'newest',
platform: '__unsourced__', untagged: true, no_artist: false,
date_from: '2024-01-01', date_to: null,
})
expect(q.tag_id).toBe('3')
expect(q.platform).toBe('__unsourced__')
expect(q.untagged).toBe('1')
expect(q.no_artist).toBeUndefined()
expect(q.date_from).toBe('2024-01-01')
expect(q.date_to).toBeUndefined()
expect(q.sort).toBeUndefined() // newest is the default → omitted
})
it('serializes similar_to', () => {
expect(filterToQuery({ tag_ids: [], similar_to: 42 }).similar_to).toBe('42')
expect(filterToQuery({ tag_ids: [], similar_to: null }).similar_to).toBeUndefined()
})
it('cloneFilter copies refine fields and detaches tag_ids', () => {
const orig = {
tag_ids: [1], artist_id: 2, media_type: 'image', sort: 'oldest',
platform: 'patreon', untagged: true, no_artist: true,
date_from: '2024-01-01', date_to: '2024-02-01',
}
const c = cloneFilter(orig)
c.tag_ids.push(9)
expect(orig.tag_ids).toEqual([1]) // detached
expect(c.platform).toBe('patreon')
expect(c.untagged).toBe(true)
expect(c.date_to).toBe('2024-02-01')
})
})
+97
View File
@@ -0,0 +1,97 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
// Mock the decode-gate so the cascade is paced purely by the store's cadence
// in tests (no real <img> load events in happy-dom). Individual tests can
// override per-call to exercise the gating itself.
vi.mock('../src/utils/preloadImage.js', () => ({
preloadImage: vi.fn(() => Promise.resolve()),
}))
import { preloadImage } from '../src/utils/preloadImage.js'
import { useShowcaseStore } from '../src/stores/showcase.js'
let nextId
function stubShowcase({ empty = false } = {}) {
globalThis.fetch = vi.fn(async () => {
const images = empty
? []
: Array.from({ length: 3 }, () => ({
id: nextId++, sha256: 's', mime: 'image/jpeg',
width: 1, height: 1, thumbnail_url: '/t',
}))
return {
ok: true,
status: 200,
statusText: '200',
text: async () => JSON.stringify({ images }),
}
})
}
describe('showcase store: buffered cascade', () => {
beforeEach(() => {
setActivePinia(createPinia())
nextId = 1
preloadImage.mockReset()
preloadImage.mockImplementation(() => Promise.resolve())
vi.useFakeTimers()
})
afterEach(() => {
vi.useRealTimers()
vi.restoreAllMocks()
})
it('drains buffered items in fire-order at a fixed cadence, no dupes', async () => {
stubShowcase()
const s = useShowcaseStore()
const p = s.loadInitial()
// Past INITIAL_COUNT (60) × CADENCE (160ms = 9.6s) plus fetch microtasks.
await vi.advanceTimersByTimeAsync(12000)
await p
expect(s.images.length).toBe(60)
const ids = s.images.map((i) => i.id)
expect(new Set(ids).size).toBe(60) // dedup held
expect(ids).toEqual([...ids].sort((a, b) => a - b)) // shown in fire order
})
it('reveals at most one item per cadence tick (decoupled from fetch)', async () => {
stubShowcase()
const s = useShowcaseStore()
const p = s.loadInitial()
await vi.advanceTimersByTimeAsync(1000)
const early = s.images.length
// Even though fetches resolve instantly, the consumer is rate-limited by
// the cadence — a 1s window must not dump the whole buffer at once.
expect(early).toBeGreaterThan(0)
expect(early).toBeLessThan(60)
await vi.advanceTimersByTimeAsync(12000)
await p
expect(s.images.length).toBe(60)
})
it('does not reveal a tile until its image has decoded', async () => {
stubShowcase()
let release
// The first tile's preload hangs until we release it; all others resolve.
preloadImage.mockImplementationOnce(() => new Promise((r) => { release = r }))
const s = useShowcaseStore()
const p = s.loadInitial()
await vi.advanceTimersByTimeAsync(1000)
expect(s.images.length).toBe(0) // gated on the un-decoded first image
release()
await vi.advanceTimersByTimeAsync(12000)
await p
expect(s.images.length).toBe(60) // cascade proceeds once it decodes
})
it('flags an empty library without starting the cascade', async () => {
stubShowcase({ empty: true })
const s = useShowcaseStore()
const p = s.loadInitial()
await vi.advanceTimersByTimeAsync(1000)
await p
expect(s.images.length).toBe(0)
expect(s.isEmpty).toBe(true)
})
})
+61
View File
@@ -0,0 +1,61 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { setActivePinia, createPinia } from 'pinia'
import { useTagStore } from '../src/stores/tags.js'
function stubFetch(handler) {
globalThis.fetch = vi.fn(async (url, init) => {
const { status, body } = handler(url, init)
return {
ok: status >= 200 && status < 300,
status,
statusText: String(status),
text: async () => (body == null ? '' : JSON.stringify(body)),
}
})
}
describe('tags store: setFandom', () => {
beforeEach(() => setActivePinia(createPinia()))
afterEach(() => vi.restoreAllMocks())
it('PATCHes fandom_id, and null to clear', async () => {
const s = useTagStore()
const calls = []
stubFetch((url, init) => {
calls.push({ url, init })
return { status: 200, body: { id: 5, name: 'Ichigo', kind: 'character', fandom_id: 9 } }
})
const body = await s.setFandom(5, 9)
expect(body.fandom_id).toBe(9)
const last = calls.at(-1)
expect(last.url).toContain('/api/tags/5')
expect(last.init.method).toBe('PATCH')
expect(JSON.parse(last.init.body)).toEqual({ fandom_id: 9 })
await s.setFandom(5, null)
expect(JSON.parse(calls.at(-1).init.body)).toEqual({ fandom_id: null })
})
it('sends merge: true when requested', async () => {
const s = useTagStore()
const calls = []
stubFetch((url, init) => {
calls.push({ url, init })
return { status: 200, body: { id: 2 } }
})
await s.setFandom(7, 3, { merge: true })
expect(JSON.parse(calls.at(-1).init.body)).toEqual({ fandom_id: 3, merge: true })
})
it('throws ApiError carrying the 409 collision body', async () => {
const s = useTagStore()
stubFetch(() => ({
status: 409,
body: { error: 'exists', target: { id: 42, name: 'Renji' } },
}))
await expect(s.setFandom(1, 2)).rejects.toMatchObject({
status: 409,
body: { target: { id: 42 } },
})
})
})
+31
View File
@@ -0,0 +1,31 @@
// @vitest-environment happy-dom
import { describe, it, expect } from 'vitest'
import { arrowNavAllowed, hasText, isTextEntry } from '../src/utils/textEntry.js'
describe('textEntry helpers', () => {
it('navigates when the target is not a text field', () => {
expect(arrowNavAllowed(document.createElement('div'))).toBe(true)
expect(arrowNavAllowed(null)).toBe(true)
})
it('navigates when a focused tag input is empty', () => {
const input = document.createElement('input')
input.value = ''
expect(arrowNavAllowed(input)).toBe(true)
})
it('does NOT navigate when the input has text (caret moves instead)', () => {
const input = document.createElement('input')
input.value = 'naruto'
expect(arrowNavAllowed(input)).toBe(false)
})
it('isTextEntry / hasText recognise inputs and textareas', () => {
const ta = document.createElement('textarea')
ta.value = 'x'
expect(isTextEntry(ta)).toBe(true)
expect(hasText(ta)).toBe(true)
expect(isTextEntry(document.createElement('div'))).toBe(false)
})
})
+42 -27
View File
@@ -104,32 +104,51 @@ async def client(app):
_SEED_SNAPSHOT: dict[str, list[dict]] | None = None
@pytest.fixture(scope="session")
def _truncate_engine():
"""Session-scoped sync engine reused by the per-test DB-reset teardown.
Creating a fresh engine + Postgres connection for EVERY test's teardown was
the integration suite's dominant cost — `--durations` showed the 15 slowest
operations in both long shards were all ~1.5-2s teardowns, i.e. the
connect+SCRAM handshake, not test logic. A single pooled connection reused
across teardowns cuts that to the TRUNCATE itself. `pool_pre_ping` guards
against Postgres reaping the idle connection mid-suite (a stale-connection
failure would be a nasty flaky bounce; the ping is sub-millisecond).
`create_engine` is lazy — it opens no connection until first `.begin()` —
so the no-DB unit job instantiates this object but never connects.
"""
eng = create_engine(_sync_database_url(), future=True, pool_pre_ping=True)
yield eng
eng.dispose()
@pytest.fixture(autouse=True)
def _reset_db_after_integration(request):
def _reset_db_after_integration(request, _truncate_engine):
"""Integration tests exercise app/celery code that commits on its own
connection, which the rollback fixtures above can't undo — so data
would leak between tests. Capture the seeded baseline at the first
integration test, then after each integration-marked test truncate
every model table and restore that baseline. Connects to the DB ONLY
for integration-marked tests, so the no-DB fast unit job is untouched.
every model table and restore that baseline. Touches the DB ONLY for
integration-marked tests, so the no-DB fast unit job is untouched.
Uses the session-scoped `_truncate_engine` (pooled, reused) rather than
building a fresh engine per test — see that fixture's docstring.
"""
global _SEED_SNAPSHOT
is_integration = request.node.get_closest_marker("integration") is not None
if is_integration and _SEED_SNAPSHOT is None:
eng = create_engine(_sync_database_url(), future=True)
snap: dict[str, list[dict]] = {}
try:
with eng.connect() as conn:
for t in Base.metadata.sorted_tables:
rows = [
dict(m)
for m in conn.execute(t.select()).mappings().all()
]
if rows:
snap[t.name] = rows
finally:
eng.dispose()
with _truncate_engine.connect() as conn:
for t in Base.metadata.sorted_tables:
rows = [
dict(m)
for m in conn.execute(t.select()).mappings().all()
]
if rows:
snap[t.name] = rows
_SEED_SNAPSHOT = snap
yield
@@ -139,18 +158,14 @@ def _reset_db_after_integration(request):
tables = ", ".join(t.name for t in Base.metadata.sorted_tables)
if not tables:
return
eng = create_engine(_sync_database_url(), future=True)
try:
with eng.begin() as conn:
conn.exec_driver_sql(
f"TRUNCATE {tables} RESTART IDENTITY CASCADE"
)
for t in Base.metadata.sorted_tables:
rows = (_SEED_SNAPSHOT or {}).get(t.name)
if rows:
conn.execute(t.insert(), rows)
finally:
eng.dispose()
with _truncate_engine.begin() as conn:
conn.exec_driver_sql(
f"TRUNCATE {tables} RESTART IDENTITY CASCADE"
)
for t in Base.metadata.sorted_tables:
rows = (_SEED_SNAPSHOT or {}).get(t.name)
if rows:
conn.execute(t.insert(), rows)
@pytest_asyncio.fixture(autouse=True)
+121
View File
@@ -0,0 +1,121 @@
{
"data": [
{
"id": "1001",
"type": "post",
"attributes": {
"title": "Image gallery post",
"published_at": "2026-05-01T12:00:00.000+00:00",
"post_type": "image_file",
"content": "<p>just a gallery</p>",
"url": "https://www.patreon.com/posts/1001",
"image": {
"large_url": "https://c10.patreonusercontent.com/4/large/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/gallery-one.jpg",
"url": "https://c10.patreonusercontent.com/4/orig/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/gallery-one.jpg"
}
},
"relationships": {
"images": {
"data": [
{"id": "9001", "type": "media"},
{"id": "9002", "type": "media"}
]
}
}
},
{
"id": "1002",
"type": "post",
"attributes": {
"title": "Attachment post",
"published_at": "2026-04-15T09:30:00.000+00:00",
"post_type": "text_only",
"content": "<p>here is a zip</p>",
"url": "https://www.patreon.com/posts/1002",
"post_file": {
"name": "bonus-pack.zip",
"url": "https://www.patreon.com/file?h=cccccccccccccccccccccccccccccccc&i=1002"
}
},
"relationships": {
"attachments_media": {
"data": [
{"id": "9003", "type": "media"}
]
}
}
},
{
"id": "1003",
"type": "post",
"attributes": {
"title": "Inline content post",
"published_at": "2026-04-01T08:00:00.000+00:00",
"post_type": "text_only",
"content": "<p>look</p><figure><img src=\"https://c10.patreonusercontent.com/4/orig/dddddddddddddddddddddddddddddddd/inline.png?token=x&amp;v=2\" alt=\"inline\"></figure>",
"url": "https://www.patreon.com/posts/1003"
},
"relationships": {}
},
{
"id": "1004",
"type": "post",
"attributes": {
"title": "Video post",
"published_at": "2026-03-20T18:45:00.000+00:00",
"post_type": "video_external_file",
"content": "<p>watch</p>",
"url": "https://www.patreon.com/posts/1004",
"post_file": {
"name": "clip.m3u8",
"url": "https://stream.mux.com/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee.m3u8?token=jwt"
}
},
"relationships": {}
}
],
"included": [
{
"id": "9001",
"type": "media",
"attributes": {
"file_name": "gallery-one.jpg",
"download_url": "https://c10.patreonusercontent.com/4/orig/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/gallery-one.jpg",
"image_urls": {
"original": "https://c10.patreonusercontent.com/4/orig/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/gallery-one.jpg",
"default": "https://c10.patreonusercontent.com/4/default/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/gallery-one.jpg"
}
}
},
{
"id": "9002",
"type": "media",
"attributes": {
"file_name": "gallery-two.jpg",
"download_url": "https://c10.patreonusercontent.com/4/orig/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/gallery-two.jpg",
"image_urls": {
"original": "https://c10.patreonusercontent.com/4/orig/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/gallery-two.jpg"
}
}
},
{
"id": "9003",
"type": "media",
"attributes": {
"file_name": "bonus-pack.zip",
"download_url": "https://www.patreon.com/file/download?h=cccccccccccccccccccccccccccccccc"
}
},
{
"id": "5555",
"type": "campaign",
"attributes": {
"name": "Example Creator",
"url": "https://www.patreon.com/example"
}
}
],
"links": {
"next": "https://www.patreon.com/api/posts?filter%5Bcampaign_id%5D=5555&page%5Bcursor%5D=NEXTCURSOR123&sort=-published_at"
}
}
+49
View File
@@ -409,3 +409,52 @@ async def test_purge_legacy_commit_deletes_only_legacy(client, db):
)
)).scalar_one()
assert gone == 0
# --- DB maintenance: bloat readout + manual VACUUM trigger ----------
@pytest.mark.asyncio
async def test_db_stats_returns_table_bloat_shape(client):
resp = await client.get("/api/admin/maintenance/db-stats")
assert resp.status_code == 200
body = await resp.get_json()
assert "tables" in body
names = {t["table"] for t in body["tables"]}
assert "image_record" in names
for t in body["tables"]:
assert {"table", "live", "dead", "dead_pct", "last_vacuum"} <= set(t)
@pytest.mark.asyncio
async def test_trigger_vacuum_queues_the_task(client, monkeypatch):
from backend.app.tasks import maintenance
calls = []
monkeypatch.setattr(maintenance.vacuum_analyze, "delay", lambda: calls.append(1))
resp = await client.post("/api/admin/maintenance/vacuum")
assert resp.status_code == 202
assert calls == [1]
# --- Tier-A: POST /tags/reset-content -------------------------------
@pytest.mark.asyncio
async def test_reset_content_tagging_dry_run_returns_counts(client, db):
db.add_all([
Tag(name="solo", kind=TagKind.general),
Tag(name="naruto", kind=TagKind.character),
Tag(name="Naruto", kind=TagKind.fandom),
Tag(name="my-series", kind=TagKind.series),
])
await db.commit()
resp = await client.post(
"/api/admin/tags/reset-content", json={"dry_run": True}
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["count"] == 2
assert body["by_kind"] == {"general": 1, "character": 1}
# dry-run leaves the rows in place — fandom + series untouched too.
assert "deleted" not in body
+2 -1
View File
@@ -43,7 +43,8 @@ async def test_directory_card_shape(client, seeded):
body = await resp.get_json()
card = next(c for c in body["cards"] if c["name"] == "alice-api")
assert set(card.keys()) == {
"id", "name", "slug", "is_subscription", "image_count", "preview_thumbnails",
"id", "name", "slug", "is_subscription", "image_count",
"unseen_count", "preview_thumbnails",
}
assert card["is_subscription"] is True
assert card["image_count"] == 1
+21
View File
@@ -155,6 +155,27 @@ async def test_audit_history_returns_recent_runs(client, db):
assert len(body["runs"]) >= 3
@pytest.mark.asyncio
async def test_audit_history_filters_by_rule(client, db):
db.add(LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="applied", matched_ids=[], finished_at=datetime.now(UTC),
))
db.add(LibraryAuditRun(
rule="single_color", params={"threshold": 0.95, "tolerance": 30},
status="ready", matched_count=2, matched_ids=[1, 2],
finished_at=datetime.now(UTC),
))
await db.commit()
# ?rule=&limit=1 → just THIS rule's latest run (the card-reconnect query).
resp = await client.get("/api/cleanup/audit?rule=single_color&limit=1")
assert resp.status_code == 200
runs = (await resp.get_json())["runs"]
assert len(runs) == 1
assert runs[0]["rule"] == "single_color"
assert runs[0]["matched_count"] == 2
@pytest.mark.asyncio
async def test_audit_apply_with_token_deletes(client, db, tmp_path):
rec = await _seed_image(db, tmp_path, w=100, h=100, name="apply.png")
+41 -4
View File
@@ -155,6 +155,8 @@ async def test_verify_no_enabled_source_is_untestable(client):
@pytest.mark.asyncio
async def test_verify_runs_gallery_dl_and_stamps_on_success(client, db, monkeypatch):
"""A gallery-dl platform routes through GalleryDLService.verify; success
stamps last_verified (platform-agnostic endpoint behavior)."""
from backend.app.models import Artist, Source
from backend.app.services import gallery_dl as gdl_mod
@@ -163,6 +165,45 @@ async def test_verify_runs_gallery_dl_and_stamps_on_success(client, db, monkeypa
return (True, "Credentials valid — the feed authenticated.")
monkeypatch.setattr(gdl_mod.GalleryDLService, "verify", _fake_verify)
await client.post("/api/credentials", json={
"platform": "subscribestar", "credential_type": "cookies", "data": _NETSCAPE,
})
artist = Artist(name="Maewix", slug="maewix")
db.add(artist)
await db.flush()
db.add(Source(
artist_id=artist.id, platform="subscribestar",
url="https://www.subscribestar.com/maewix", enabled=True, config_overrides={},
))
await db.commit()
resp = await client.post("/api/credentials/subscribestar/verify")
body = await resp.get_json()
assert body["valid"] is True
assert body["last_verified"] is not None
# The stamp is persisted on the credential record.
rec = await (await client.get("/api/credentials/subscribestar")).get_json()
assert rec["last_verified"] is not None
@pytest.mark.asyncio
async def test_verify_patreon_uses_native_ingester_not_gallery_dl(client, db, monkeypatch):
"""plan #697 cutover: Patreon credential verify routes through the native
ingester (verify_patreon_credential), NOT gallery-dl --simulate."""
from backend.app.models import Artist, Source
from backend.app.services import gallery_dl as gdl_mod
from backend.app.services import patreon_ingester as pi_mod
async def _native_verify(url, cookies_path, overrides):
return (True, "Credentials valid — the Patreon feed authenticated.")
monkeypatch.setattr(pi_mod, "verify_patreon_credential", _native_verify)
# gallery-dl must NOT be consulted for Patreon.
async def _boom(self, *args, **kwargs):
raise AssertionError("gallery-dl verify must not run for patreon")
monkeypatch.setattr(gdl_mod.GalleryDLService, "verify", _boom)
await client.post("/api/credentials", json={
"platform": "patreon", "credential_type": "cookies", "data": _NETSCAPE,
})
@@ -180,10 +221,6 @@ async def test_verify_runs_gallery_dl_and_stamps_on_success(client, db, monkeypa
assert body["valid"] is True
assert body["last_verified"] is not None
# The stamp is persisted on the credential record.
rec = await (await client.get("/api/credentials/patreon")).get_json()
assert rec["last_verified"] is not None
@pytest.mark.asyncio
async def test_verify_reports_auth_failure(client, db, monkeypatch):
+56
View File
@@ -17,6 +17,7 @@ async def _seed(db, count: int = 3):
origin="imported_filesystem", integrity_status="unknown",
)
r.created_at = base - timedelta(minutes=i)
r.effective_date = r.created_at # no post → tracks created_at (0035)
db.add(r)
await db.flush()
await db.commit()
@@ -58,3 +59,58 @@ async def test_jump_requires_year_month(client):
async def test_image_detail_404_when_missing(client):
resp = await client.get("/api/gallery/image/99999")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_scroll_sort_param_reverses(client, db):
await _seed(db, 3)
newest = await (await client.get("/api/gallery/scroll?limit=10&sort=newest")).get_json()
oldest = await (await client.get("/api/gallery/scroll?limit=10&sort=oldest")).get_json()
ids_new = [i["id"] for i in newest["images"]]
ids_old = [i["id"] for i in oldest["images"]]
assert ids_old == list(reversed(ids_new))
@pytest.mark.asyncio
async def test_scroll_media_param(client, db):
await _seed(db, 2) # only image/jpeg seeded
resp = await client.get("/api/gallery/scroll?limit=10&media=video")
assert resp.status_code == 200
assert (await resp.get_json())["images"] == []
@pytest.mark.asyncio
async def test_facets_endpoint(client, db):
await _seed(db, 3) # filesystem images: no artist, no tags, no platform
resp = await client.get("/api/gallery/facets")
assert resp.status_code == 200
body = await resp.get_json()
assert body["total"] == 3
assert body["untagged"] == 3
assert body["no_artist"] == 3
assert "platforms" in body
assert body["date_min"] is not None
assert body["date_max"] is not None
@pytest.mark.asyncio
async def test_facets_rejects_bad_date(client):
resp = await client.get("/api/gallery/facets?date_from=notadate")
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_scroll_untagged_param(client, db):
await _seed(db, 2)
resp = await client.get("/api/gallery/scroll?untagged=1&limit=10")
assert resp.status_code == 200
assert len((await resp.get_json())["images"]) == 2
@pytest.mark.asyncio
async def test_scroll_date_from_param(client, db):
await _seed(db, 2) # all created ~now
# A future date_from excludes everything.
resp = await client.get("/api/gallery/scroll?date_from=2099-01-01&limit=10")
assert resp.status_code == 200
assert (await resp.get_json())["images"] == []
+38 -14
View File
@@ -199,7 +199,7 @@ async def test_list_derives_next_check_at_when_last_checked_set(
@pytest.mark.asyncio
async def test_backfill_endpoint_arms_source(client, artist, db):
async def test_backfill_endpoint_start_and_stop(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill", enabled=True,
@@ -208,18 +208,21 @@ async def test_backfill_endpoint_arms_source(client, artist, db):
await db.commit()
sid = src.id
resp = await client.post(f"/api/sources/{sid}/backfill", json={"runs": 5})
resp = await client.post(f"/api/sources/{sid}/backfill", json={"action": "start"})
assert resp.status_code == 200
body = await resp.get_json()
assert body["backfill_runs_remaining"] == 5
assert body["backfill_state"] == "running"
# GET reflects the new state.
# GET reflects the running state.
one = await client.get(f"/api/sources/{sid}")
assert (await one.get_json())["backfill_runs_remaining"] == 5
assert (await one.get_json())["backfill_state"] == "running"
stopped = await client.post(f"/api/sources/{sid}/backfill", json={"action": "stop"})
assert (await stopped.get_json())["backfill_state"] is None
@pytest.mark.asyncio
async def test_backfill_endpoint_defaults_to_three(client, artist, db):
async def test_backfill_endpoint_defaults_to_start(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill-default", enabled=True,
@@ -228,26 +231,47 @@ async def test_backfill_endpoint_defaults_to_three(client, artist, db):
await db.commit()
resp = await client.post(f"/api/sources/{src.id}/backfill", json={})
body = await resp.get_json()
assert body["backfill_runs_remaining"] == 3
assert body["backfill_state"] == "running"
@pytest.mark.asyncio
async def test_backfill_endpoint_rejects_out_of_range(client, artist, db):
async def test_backfill_endpoint_rejects_bad_action(client, artist, db):
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-backfill-bad", enabled=True,
)
db.add(src)
await db.commit()
bad = await client.post(f"/api/sources/{src.id}/backfill", json={"runs": 0})
bad = await client.post(f"/api/sources/{src.id}/backfill", json={"action": "nope"})
assert bad.status_code == 400
too_big = await client.post(
f"/api/sources/{src.id}/backfill", json={"runs": 99}
)
assert too_big.status_code == 400
@pytest.mark.asyncio
async def test_backfill_endpoint_404_when_source_missing(client):
resp = await client.post("/api/sources/999999/backfill", json={"runs": 3})
resp = await client.post("/api/sources/999999/backfill", json={"action": "start"})
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_backfill_endpoint_recover_arms_bypass(client, artist, db):
"""Plan #697: action='recover' arms a backfill that bypasses the Patreon
seen-ledger; the response exposes backfill_bypass_seen=True so the UI badge
can label it 'Recovering'. Stop clears it."""
src = Source(
artist_id=artist.id, platform="patreon",
url="https://patreon.com/alice-recover", enabled=True,
)
db.add(src)
await db.commit()
sid = src.id
resp = await client.post(f"/api/sources/{sid}/backfill", json={"action": "recover"})
assert resp.status_code == 200
body = await resp.get_json()
assert body["backfill_state"] == "running"
assert body["backfill_bypass_seen"] is True
stopped = await client.post(f"/api/sources/{sid}/backfill", json={"action": "stop"})
stopped_body = await stopped.get_json()
assert stopped_body["backfill_state"] is None
assert stopped_body["backfill_bypass_seen"] is False
+48 -2
View File
@@ -3,11 +3,57 @@ import pytest
pytestmark = pytest.mark.integration
async def _mk(client, name, kind):
r = await client.post("/api/tags", json={"name": name, "kind": kind})
async def _mk(client, name, kind, fandom_id=None):
body = {"name": name, "kind": kind}
if fandom_id is not None:
body["fandom_id"] = fandom_id
r = await client.post("/api/tags", json=body)
return (await r.get_json())["id"]
@pytest.mark.asyncio
async def test_get_tag_returns_shape_and_404(client):
tid = await _mk(client, "Lookup", "general")
resp = await client.get(f"/api/tags/{tid}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["id"] == tid
assert body["name"] == "Lookup"
assert body["kind"] == "general"
resp = await client.get("/api/tags/99999999")
assert resp.status_code == 404
@pytest.mark.asyncio
async def test_patch_sets_and_clears_character_fandom(client):
fandom = await _mk(client, "Bleach", "fandom")
char = await _mk(client, "Ichigo", "character")
resp = await client.patch(f"/api/tags/{char}", json={"fandom_id": fandom})
assert resp.status_code == 200
assert (await resp.get_json())["fandom_id"] == fandom
resp = await client.patch(f"/api/tags/{char}", json={"fandom_id": None})
assert resp.status_code == 200
assert (await resp.get_json())["fandom_id"] is None
@pytest.mark.asyncio
async def test_patch_fandom_collision_then_merge(client):
f1 = await _mk(client, "Bleach", "fandom")
f2 = await _mk(client, "Naruto", "fandom")
c1 = await _mk(client, "Renji", "character", fandom_id=f1)
c2 = await _mk(client, "Renji", "character", fandom_id=f2)
# Moving c1 into f2 collides with c2 → rich 409 (same shape as rename).
resp = await client.patch(f"/api/tags/{c1}", json={"fandom_id": f2})
assert resp.status_code == 409
assert (await resp.get_json())["target"]["id"] == c2
# Confirm → merge c1 into c2, c2 survives.
resp = await client.patch(
f"/api/tags/{c1}", json={"fandom_id": f2, "merge": True}
)
assert resp.status_code == 200
assert (await resp.get_json())["id"] == c2
@pytest.mark.asyncio
async def test_rename_collision_returns_rich_409(client):
tgt = await _mk(client, "Canonical", "general")
+175
View File
@@ -0,0 +1,175 @@
"""ArtistVisit unseen-count badge + ArtistService.overview banner data.
Covers:
- Directory cards include `unseen_count`
- LEFT JOIN keeps artists without a visit row (treats NULL as "never
visited" → all images unseen)
- overview() returns `unseen_count_at_visit` and stamps the visit
- find_or_create autoseeds a visit row so freshly imported content
doesn't show up as unseen
- Repeat overview() returns 0 (since the previous visit just stamped
last_viewed_at = NOW())
"""
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import select
from backend.app.models import Artist, ArtistVisit, ImageRecord
from backend.app.services.artist_directory_service import ArtistDirectoryService
from backend.app.services.artist_service import ArtistService
pytestmark = pytest.mark.integration
_LONG_AGO = datetime(2000, 1, 1, tzinfo=UTC)
_RECENTLY = datetime(2099, 1, 1, tzinfo=UTC)
async def _seed_artist(db, name: str) -> Artist:
a = Artist(name=name, slug=name.lower().replace(" ", "-"))
db.add(a)
await db.flush()
return a
async def _seed_visit(db, artist_id: int, when: datetime) -> None:
db.add(ArtistVisit(artist_id=artist_id, last_viewed_at=when))
await db.flush()
async def _seed_image(db, artist_id: int, suffix: str, *, created_at: datetime) -> None:
db.add(ImageRecord(
path=f"/images/visit-{suffix}.jpg",
sha256=f"visit{suffix}".ljust(64, "0")[:64],
size_bytes=10, mime="image/jpeg", width=10, height=10,
origin="downloaded", artist_id=artist_id,
created_at=created_at,
))
await db.flush()
# --- Directory unseen_count -----------------------------------------------
@pytest.mark.asyncio
async def test_directory_unseen_count_zero_when_no_images(db):
await _seed_artist(db, "zoey-empty-visit")
await db.commit()
page = await ArtistDirectoryService(db).list_artists(
q="zoey-empty-visit", platform=None, cursor=None, limit=60,
)
target = next(c for c in page.cards if c["name"] == "zoey-empty-visit")
assert target["unseen_count"] == 0
@pytest.mark.asyncio
async def test_directory_unseen_counts_only_images_after_visit(db):
a = await _seed_artist(db, "alice-visit-mix")
await _seed_visit(db, a.id, _LONG_AGO + timedelta(days=365))
# Two images BEFORE the visit (seen), three AFTER (unseen).
for i in range(2):
await _seed_image(db, a.id, f"old-{i}", created_at=_LONG_AGO)
for i in range(3):
await _seed_image(db, a.id, f"new-{i}", created_at=_RECENTLY)
await db.commit()
page = await ArtistDirectoryService(db).list_artists(
q="alice-visit-mix", platform=None, cursor=None, limit=60,
)
target = next(c for c in page.cards if c["name"] == "alice-visit-mix")
assert target["image_count"] == 5
assert target["unseen_count"] == 3
@pytest.mark.asyncio
async def test_directory_treats_missing_visit_as_never_visited(db):
"""No ArtistVisit row → defensive count of all images as unseen.
Shouldn't happen in practice (migration 0034 seeds existing
artists, find_or_create autoseeds new ones), but the directory
query must not regress to "0 unseen" if a row is missing.
"""
a = await _seed_artist(db, "bob-no-visit")
for i in range(4):
await _seed_image(db, a.id, f"orphan-{i}", created_at=_RECENTLY)
await db.commit()
page = await ArtistDirectoryService(db).list_artists(
q="bob-no-visit", platform=None, cursor=None, limit=60,
)
target = next(c for c in page.cards if c["name"] == "bob-no-visit")
assert target["unseen_count"] == 4
# --- overview() marks visit + returns count -------------------------------
@pytest.mark.asyncio
async def test_overview_returns_unseen_count_at_visit_and_stamps_now(db):
a = await _seed_artist(db, "carol-stamp")
await _seed_visit(db, a.id, _LONG_AGO)
await _seed_image(db, a.id, "stamp-1", created_at=_RECENTLY)
await _seed_image(db, a.id, "stamp-2", created_at=_RECENTLY)
await db.commit()
data = await ArtistService(db).overview("carol-stamp")
assert data is not None
assert data["unseen_count_at_visit"] == 2
# last_viewed_at advanced to NOW() — directly check the row.
visit_at = (await db.execute(
select(ArtistVisit.last_viewed_at).where(ArtistVisit.artist_id == a.id)
)).scalar_one()
assert visit_at > _LONG_AGO
@pytest.mark.asyncio
async def test_overview_repeat_call_returns_zero(db):
a = await _seed_artist(db, "dana-repeat")
await _seed_visit(db, a.id, _LONG_AGO)
await _seed_image(db, a.id, "repeat-1", created_at=_LONG_AGO + timedelta(days=1))
await db.commit()
first = await ArtistService(db).overview("dana-repeat")
assert first is not None
assert first["unseen_count_at_visit"] == 1
# Second call: no new images, visit just stamped → count is 0.
second = await ArtistService(db).overview("dana-repeat")
assert second is not None
assert second["unseen_count_at_visit"] == 0
# --- find_or_create autoseeds the visit row ------------------------------
@pytest.mark.asyncio
async def test_find_or_create_autoseeds_visit_row(db):
artist, created = await ArtistService(db).find_or_create("Eve-Autoseed")
assert created is True
row = (await db.execute(
select(ArtistVisit.last_viewed_at).where(ArtistVisit.artist_id == artist.id)
)).scalar_one_or_none()
assert row is not None
@pytest.mark.asyncio
async def test_find_or_create_existing_does_not_reset_visit(db):
"""Calling find_or_create on an existing artist returns it as-is —
must NOT touch the visit row's timestamp."""
a = await _seed_artist(db, "Frank-Existing")
await _seed_visit(db, a.id, _LONG_AGO)
await db.commit()
artist, created = await ArtistService(db).find_or_create("Frank-Existing")
assert created is False
assert artist.id == a.id
visit_at = (await db.execute(
select(ArtistVisit.last_viewed_at).where(ArtistVisit.artist_id == a.id)
)).scalar_one()
assert visit_at == _LONG_AGO

Some files were not shown because too many files have changed in this diff Show More