Compare commits

...

111 Commits

Author SHA1 Message Date
bvandeusen 37e8b796a1 Merge pull request 'v26.05.27.0: PostCard redesign + IR-style tag suffix + drop meta/rating + extension v1.0.4 CSP fix' (#28) from dev into main 2026-05-27 11:31:18 -04:00
bvandeusen 8675f105ad fix(tests): test_api_tags prefix tests use character: not artist: (KNOWN_KINDS dropped artist)
The two prefix-parsing tests were pinned to `artist:Eric`, but `artist`
was removed from KNOWN_KINDS in commit 4cad07a (provenance is a separate
axis from tags). The parser now keeps `artist:` literal, so the assertion
`body["name"] == "Eric"` failed.

Repointed to `character:Saber` (still in KNOWN_KINDS). Also updated the
stale `artist:` docstring example in parse_kind_prefix to `fandom:`.

Caught by [[reference-grep-pinned-tests-in-plans]] — should have grep'd
tests/ for `artist:` when shrinking KNOWN_KINDS. Banking the miss.
2026-05-27 11:09:04 -04:00
bvandeusen 74dac6b960 fix(extension+migration): MV3 CSP opt-out from upgrade-insecure-requests (v1.0.4) + alembic 0023 drops the ck_tag_fandom_requires_character check before the type swap
extension/manifest.json: add content_security_policy.extension_pages = "script-src 'self'; object-src 'self';" — explicitly omits the upgrade-insecure-requests directive that MV3 inherits by default. Without this, every fetch(http://curator.../...) silently upgrades to https:// at the browser layer (Sec-Fetch-Site=same-origin, NS_ERROR_GENERATE_FAILURE), regardless of about:config. Bump XPI version 1.0.3 → 1.0.4 so a fresh signed build replaces the cached one. Operator-troubleshot 2026-05-26 via Inspect-the-extension dev tools showing the silent scheme upgrade.

alembic 0023: drop ck_tag_fandom_requires_character before the tag_kind type swap and recreate after. Postgres can't resolve `kind = 'character'` across the rename (column on tag_kind_old, literal binds to new tag_kind → "operator does not exist"). Same dance on downgrade. Banked under reference_tag_kind_enum_swap_check_drop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:57:38 -04:00
bvandeusen 9e19c081b0 fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:23:12 -04:00
bvandeusen 3838f04c16 feat(tag-kinds): drop meta + rating entirely — alembic 0023 deletes existing meta/rating tags (CASCADE clears related image_tag / alias / allowlist / suggestion_rejection / reference_embedding / series_page rows) then recreates the tag_kind ENUM without those values. Python TagKind enum trimmed; KIND_OPTIONS + KIND_COLOR + KIND_ICONS maps + TagsView KINDS array all updated. Operator confirmed they have no use for the data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:07:31 -04:00
bvandeusen 42b1340324 fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:00:12 -04:00
bvandeusen 3b1e2f1ceb feat(tag-input): IR-style kind:name suffix — drop the kind dropdown from TagAutocomplete; client-side parser mirrors backend's parse_kind_prefix (KNOWN_KINDS = artist/character/fandom/series/meta/rating); autocomplete searches across all kinds and shows kind chip in results; Create label uses parsed kind; character flow still goes through FandomPicker
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:54:11 -04:00
bvandeusen 8cdf0af0e1 feat(tags-api): IR-style kind:name parsing at POST /api/tags — when caller doesn't supply explicit kind, parse_kind_prefix runs on the name (artist:Eric → kind=artist, name='Eric'); explicit kind always wins for backward-compat; falls back to general when no recognized prefix is present. Updates the old "missing required" test that assumed kind was mandatory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:53:27 -04:00
bvandeusen ccee344099 feat(tag-prefix): parse_kind_prefix util — IR-style \kind:name\ parser at the input boundary; KNOWN_KINDS = artist/character/fandom/series/meta/rating (excludes default \general\ and system-managed archive/post)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:49 -04:00
bvandeusen 0316f92e8b feat(artist-posts-tab): bump max-width 900 → 1600 so the new wide-layout PostCard has room and the artist Posts feed doesn't leave most of an ultra-wide screen empty
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:21 -04:00
bvandeusen 6df74683b3 feat(post-card): responsive redesign — container-query split (stack <800px / side-by-side ≥800px), hero + thumb rail, +N overflow chip, line-clamp body (3 narrow / 5 wide), title/desc fallbacks for sparse data, click→postModal.open
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:10 -04:00
bvandeusen 243e536225 feat(app): mount PostModal at app root next to ImageViewer — single instance driven by usePostModalStore so PostCard can open from anywhere
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:50:34 -04:00
bvandeusen 2f16699971 feat(post-modal): PostModal — full Patreon-style v-dialog (header + image grid + sanitized body + attachments); reads from usePostModalStore
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:59 -04:00
bvandeusen a5cb684d34 feat(post-modal): PostImageGrid — fixed-cell grid (auto-fill 220px+, 4:3 aspect-cover) inside PostModal; click opens ImageViewer scoped to the post's images via modalStore.open(id, { postImageIds })
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:20 -04:00
bvandeusen 965a953b2e feat(post-card): PostEmptyThumbs — dashed-border placeholder shown in PostCard's hero slot when post has zero linked images
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:49:02 -04:00
bvandeusen 90c176b195 feat(postmodal-store): Pinia store driving the app-level PostModal — open(post) fetches full detail via posts store; close() clears
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:49 -04:00
bvandeusen b8d89b9f2a feat(modal-store): post-scoped cycle — open(id, { postImageIds }) pins prev/next to the array; canPrev/canNext + goPrev/goNext check the array index instead of current.value.neighbors when set. Gallery-context callers unchanged (default args clear scope)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:48:30 -04:00
bvandeusen 07344e0843 feat(util): htmlSanitize — whitelist-based DOM scrubber for PostModal's description v-html (Patreon ships HTML; sanitize before render)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:36 -04:00
bvandeusen 42c33e44f9 feat(post-api): get_post returns uncapped thumbnails — PostModal masonry needs full image list; feed query unchanged (still capped at 6 for previews). _thumbnails_for gains a limit kwarg; get_post passes limit=None.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:47:18 -04:00
bvandeusen 4e82208926 Merge pull request 'v26.05.26.5 — extension CORS unblock + UI gap closes + CI workflow cleanup' (#27) from dev into main 2026-05-26 20:15:07 -04:00
bvandeusen 85b640f32e fix(views): close the 24-32px gap below TopNav across all views — every v-container had py-6 (or py-8 on PlaceholderView) which pushed the first content item well below where the TopNav's fade-to-transparent gradient bottoms out. Switch to pt-2 pb-6 (8px top, 24px bottom) so content sits comfortably right below the nav, matches the ArtistHeader's 'continuous with TopNav' feel. PlaceholderView uses pt-3 pb-8 keeping its larger bottom padding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:47:23 -04:00
bvandeusen c7001f4aed fix(extension): CORS preflight for moz-extension:// + chrome-extension:// origins — operator-flagged 2026-05-26 that the extension's Test connection returned NetworkError because /api/credentials POSTs with X-Extension-Key trigger a browser preflight OPTIONS that hit a 405 (no OPTIONS method registered) with no Access-Control-Allow-* headers. Adds two app-level hooks: before_request short-circuits OPTIONS from extension origins with 204, after_request stamps the necessary ACL headers on responses to extension-origin requests. Whitelist is intentionally narrow (extension schemes only) so normal browser usage doesn't get permissive CORS. Five integration tests pin the contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:43:31 -04:00
bvandeusen f827612930 fix(artist-header): close gap below TopNav (top:64px → 48px to match TopNav's actual ~48px height) + center the tab strip via 1fr|auto|1fr layout with a right-side spacer cell
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:36:19 -04:00
bvandeusen 3f0153cba5 ci(workflows): dedupe + versioned image tags
ci.yml: drop pull_request: trigger — push: branches: [dev, main] already covers it; pull_request was duplicating ci.yml runs on every dev push with an open PR. (No fork PRs in this repo.)

build.yml: drop dev from push triggers — operator doesn't use the :dev image. Add tags: ['v*'] trigger + tag-push branch in the Determine-tag logic so cutting a release tag publishes an immutable :v26.05.26.X image (rollback story) without re-publishing :latest. Extend the XPI-download step to fire on tag pushes too so the versioned image carries the signed extension.

Net per hotfix cycle: 5 runs → 3 (no tag) / 4 (with tag).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:26:56 -04:00
bvandeusen 52fff00353 Merge pull request 'v26.05.26.4 — hotfix: migration 0022 pre-DELETE colliding ImageProvenance before UPDATE' (#26) from dev into main 2026-05-26 18:06:20 -04:00
bvandeusen f3e8f30a8f fix(migration-0022): pre-DELETE colliding image_provenance rows before the UPDATE post_id — same row-by-row UNIQUE pattern as the post-collision case, just one level deeper. When image X has provenance under both keep and drop, UPDATE drop→keep would fire uq_image_provenance_image_post on the row that'd collide with the existing (X, keep). Pre-delete those rows (their info is already represented by the keep-side provenance) before the UPDATE moves the rest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:05:49 -04:00
bvandeusen eee107766e fix(migration-0022): rename unused _epid loop var (ruff B007)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:03:11 -04:00
bvandeusen c14338cbce Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main 2026-05-26 17:52:59 -04:00
bvandeusen 7a64730bd2 fix(migration-0022): pre-merge ALL duplicate-external_post_id Posts across the (canonical+others) group, not just canonical-vs-others — operator's v26.05.26.2 deploy still tripped uq_post_source_external_id because two non-canonical Sources both had Posts with epid=6166997. Bulk UPDATE moved the first cleanly then collided on the second. New pre-merge groups all Posts in the (artist, platform) by external_post_id; for any group with count>1, picks the keep (prefer one under canonical; else lowest id) and merges the rest before the bulk reparent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:52:29 -04:00
bvandeusen 1803a09306 ci(workflow): remove the 4 Cache pip wheels steps entirely — act_runner's cache backend has been broken for 11+ days and the cached path (~/.cache/pip) wasn't even the primary install tool's cache anyway (uv uses ~/.cache/uv). Net cost ~30s/job of wheel downloads. Long-term: mount ~/.cache/uv as a docker volume at the runner level (skips actions/cache entirely) or fix the runner-side cache backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:54:05 -04:00
bvandeusen 8c36dd28b0 Merge pull request 'v26.05.26.2 — hotfix: alembic 0022 Post-collision pre-merge + ci.yml cache continue-on-error' (#24) from dev into main 2026-05-26 16:50:43 -04:00
bvandeusen 0f7cd3cb76 fix(migration-0022): pre-merge colliding Posts before the bulk reparent — Postgres fires uq_post_source_external_id row-by-row during UPDATE, so the post-reparent merge-collisions step never ran (operator's v26.05.26.1 deploy hit it: 'duplicate key (source_id, external_post_id)=(42, 6166997)'). Detect (keep, drop) Post pairs whose external_post_id already exists under canonical, merge the drop into keep, then bulk-reparent the rest cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:48:49 -04:00
bvandeusen 7b0dd4182c ci(workflow): continue-on-error on Cache pip wheels — act_runner's cache backend has been broken since 2026-05-15 and now hard-fails ('Cannot find module .../dist/restore/index.js') instead of warning. Install step handles cold caches natively; ~30s wheel-download cost per job until the runner-side cache backend is fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:35:26 -04:00
bvandeusen 88cfb3dd02 Merge pull request 'v26.05.26.1 — thumb backfill, modal redesign, recovery sweep race-safety, artist view redesign, extension fixes' (#23) from dev into main 2026-05-26 16:32:00 -04:00
bvandeusen fb41b90110 fix(extension): _find_or_create_artist + _find_or_create_source race-safe via savepoint + IntegrityError recovery — same pattern as importer's helpers. Two concurrent quick-add-source calls on the same artist/url would have 500'd on uq_artist_slug / uq_source_artist_platform_url; now the second one rolls the savepoint back and returns the existing row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:27:51 -04:00
bvandeusen 7d84990f6d feat(artist-view): ArtistView rewrite — sticky frosted ArtistHeader (name + stats + tabs) replaces the in-body h1; three lazy tabs (Posts default, Gallery fallback, Management); ?tab= URL state; cross-artist store reset; document.title set on slug change
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:01:14 -04:00
bvandeusen ca55d92c68 feat(artist-view): ArtistManagementTab — Overview chips (Subscription badge + subscription count) + Frequent tags + Activity sparkline + Subscriptions table + Danger zone. 'View posts' chip and 'Credential health · FC-3b' placeholder chip dropped; 'Sources' section renamed to 'Subscriptions'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:47 -04:00
bvandeusen cce014be3a feat(artist-view): ArtistGalleryTab — MasonryGrid wired to the artist store's existing loadMoreImages; modal-open preserves ?tab=. No global gallery-store coupling (avoids cross-pollution into /gallery)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:14 -04:00
bvandeusen c07effb593 feat(artist-view): ArtistPostsTab — PostCard infinite-scroll list, artist_id pinned, no filter bar; mirrors PostsView mechanics
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 16:00:02 -04:00
bvandeusen a36f72b383 feat(artist-view): ArtistHeader — sticky frosted bar (top:64px) matching TopNav recipe, hosts name + image/post stats + tab strip
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:59:46 -04:00
bvandeusen 2e8d7c960c feat(artist): post_count on the artist overview response — drives the Posts/Gallery default-tab fallback in the upcoming ArtistView redesign
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:59:28 -04:00
bvandeusen 992f38ec20 fix(test): drop unused Post binding in test_importer_provenance_race (ruff F841)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:39:57 -04:00
bvandeusen 0bc5767a2b fix(importer): Source = one per (artist, platform), not one per post — filesystem importer's sidecar paths now reuse the artist's existing subscription Source (or create one synthetic anchor with enabled=False) instead of fabricating a new Source per post URL. Alembic 0022 consolidates existing per-post Sources to canonical (prefers campaign URL; falls back to sidecar:<platform>:<slug>) and re-parents Posts + ImageProvenance, merging Post collisions.
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 14:19:29 -04:00
bvandeusen 397021dcbd fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 13:47:55 -04:00
bvandeusen b0bfbc585a fix(import-admin): retry-failed + clear-stuck — same UPDATE…WHERE pattern as the maintenance sweep, so neither endpoint can hit psycopg's 65535-parameter ceiling once accumulated row counts exceed ~65k
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:47:40 -04:00
bvandeusen 110c1c0e51 fix(maintenance): recover_interrupted_tasks — fold SELECT into UPDATE…WHERE…RETURNING so the IN-list no longer blows past psycopg's 65535-parameter ceiling (operator-hit 2026-05-26 after deep scan orphan pile)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:46:06 -04:00
bvandeusen 6de84d0d60 feat(ui): ErrorDetailModal — context panel (task/queue/target/duration/started/retries/worker/celery-id/args) + contrast fix (background-token bg vs surface-variant pale-on-pale) + copyText helper for Copy button
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:16:37 -04:00
bvandeusen 4e1f208a9f fix(ui-copy): copyText utility with execCommand fallback — navigator.clipboard is gated by Secure Context (HTTPS-only) and is undefined on plain-HTTP self-hosted deployments. Apply to ExtensionKeyBar + BrowserExtensionCard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 11:16:32 -04:00
bvandeusen 06913eba8e feat(thumb-backfill): MaintenancePanel — wire ThumbnailBackfillCard into grid, broaden intro to cover non-ML backfills
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:37:09 -04:00
bvandeusen b7f693b15e feat(thumb-backfill): ThumbnailBackfillCard — 'Run backfill now' button, mirrors MLBackfillCard pattern
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:55 -04:00
bvandeusen ecd0199799 feat(thumb-backfill): Pinia store — triggerBackfill() POSTs /api/thumbnails/backfill
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:45 -04:00
bvandeusen 983da9e5b1 feat(thumb-backfill): /api/thumbnails/backfill endpoint — POST → 202 + celery_task_id
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:38 -04:00
bvandeusen a41eddae3f feat(thumb-backfill): backfill_thumbnails planner task — keyset-paginates ImageRecord, NULLs bad thumb paths, enqueues generate_thumbnail for NULL/missing/corrupt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:36:21 -04:00
bvandeusen 7aa7f5a3d6 feat(thumb-backfill): _thumb_is_valid helper — JPEG/PNG magic-byte check on the on-disk thumbnail file
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 10:35:06 -04:00
bvandeusen 5d4f223b71 Merge pull request 'Release v26.05.25.7 — FC-Cleanup tab + UniqueViolation fix + error modal + extension install fix' (#22) from dev into main 2026-05-26 08:26:46 -04:00
bvandeusen 2505b197ae feat(fc-cleanup): Pinia store + 3 cards + CleanupView + SettingsView tab + TagMaintenanceCard moved from Maintenance + ruff lint fixes — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:16:46 -04:00
bvandeusen 0d0b236ac3 feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:13:53 -04:00
bvandeusen a06ada4c9b fix(ext-ui): direct :href install button (Firefox needs anchor click, not programmatic navigation) + manifest version detection ignores -latest.xpi alias — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 08:07:22 -04:00
bvandeusen ebd985990c feat(ui): ErrorDetailModal — click error → flat-text modal with copy button (replaces unusable :title tooltip for multi-line SQLAlchemy tracebacks) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:53:14 -04:00
bvandeusen 4da8d1d774 fix(importer): race-safe savepoint-based find-or-create for Source + Post (uq_source_artist_platform_url UniqueViolation operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 07:51:12 -04:00
bvandeusen 05090c6e85 Merge pull request 'Release v26.05.25.7 — animated-WebP worker fix + FC-Cleanup backend' (#21) from dev into main 2026-05-26 01:48:13 -04:00
bvandeusen 2d4bfa4375 fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:32:50 -04:00
bvandeusen 6ed2021ad6 feat(fc-cleanup): scan_library_for_rule Celery task + maintenance-queue registration — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:21:37 -04:00
bvandeusen 4f2ceaaf31 feat(fc-cleanup): audit lifecycle service functions (start/apply/cancel) + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:41 -04:00
bvandeusen 8a5b337a53 feat(fc-cleanup): min-dimension service functions + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:20:04 -04:00
bvandeusen 900d878d27 feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:38 -04:00
bvandeusen fd80d40a34 feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:18:07 -04:00
bvandeusen 929d3fc092 feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 01:17:42 -04:00
bvandeusen c0c9e56fb9 fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:45:08 -04:00
bvandeusen 3a577d5ade Merge pull request 'fix(ext-ci): use browser_download_url + curl -f + ZIP magic check (XPI silently corrupt)' (#20) from dev into main 2026-05-26 00:43:02 -04:00
bvandeusen 06a2f60c08 fix(ext-ci): use browser_download_url not /releases/assets/<id> + add -f to curl + magic-byte sanity check (operator-flagged 2026-05-26: prior build silently wrote '404 page not found' into the XPI file, Firefox rejected as corrupt) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:42:35 -04:00
bvandeusen 0978fbac66 fix(sidecar): strip gallery-dl 'NN_' numbering prefix when locating sidecars — fixes 'deep scan refresh count high but 0 Posts created' (operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-26 00:27:43 -04:00
bvandeusen f4fe02e346 Merge pull request 'fix(ext-ci): drop actions/upload-artifact (Forgejo doesn't support v4+ GHES)' (#19) from dev into main 2026-05-25 23:33:40 -04:00
bvandeusen efb142239d fix(ext-ci): drop actions/upload-artifact (Forgejo Actions doesn't support v4+ GHES) — build-web reads XPI directly from the ext-<version> Forgejo release asset — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:32:58 -04:00
bvandeusen e766197d99 Merge pull request 'fix(ext-ci): jq→python + bump ext to 1.0.3 + rollback-on-upload-failure' (#18) from dev into main 2026-05-25 23:14:51 -04:00
bvandeusen 5587a76606 fix(ext-ci): replace jq with python3 (jq not in ci-python image) + bump ext 1.0.2→1.0.3 (escape AMO 'version already exists' from prior partial-failure run) + add rollback to prevent empty cache-release tombstones — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 23:14:11 -04:00
bvandeusen 3872e1dda9 Merge pull request 'fix(ext-ci): web-ext v8 .cjs config workaround' (#17) from dev into main 2026-05-25 22:49:14 -04:00
bvandeusen 17e19081a2 fix(ext-ci): drop web-ext-config.cjs (v8 mis-parses .cjs configs as if module.exports were a config option) — inline ignore-files on CLI + --no-config-discovery — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:46:58 -04:00
bvandeusen 9814f3dbaf Merge pull request 'Release v26.05.25.5 — Extension publish refactor, deep-scan IR-parity, archive-import perf, artist Settings tab' (#16) from dev into main 2026-05-25 22:44:59 -04:00
bvandeusen 770bcf3aa6 feat(artist): tab split (Overview/Settings) so DangerZone is reachable without exhausting the infinite-scroll image grid — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:32:07 -04:00
bvandeusen 52d7905c43 perf(importer): cache phash candidates on Importer to fix archive-import soft-timeout (was O(M×N) per-member SELECTs) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:25:32 -04:00
bvandeusen e6ededbe8e feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 22:16:26 -04:00
bvandeusen c06cbc0abe feat(ci): inline extension sign into build.yml + Forgejo Release Assets as XPI cache (v26.05.25.5) — bump ext to 1.0.2 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 21:30:11 -04:00
bvandeusen b214460fdb Merge pull request 'Release v26.05.25.4 — importer ext sanitize fix, CI shard split, BrowserExtensionCard on Overview' (#15) from dev into main 2026-05-25 21:11:50 -04:00
bvandeusen ac39509a74 fix(ci): rename shard jobs to no-separator names (intapi/intimp/intcore) + add diagnostic docker ps dump so next bounce surfaces the real act_runner naming convention — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:59:17 -04:00
bvandeusen 3531f373ee feat(settings): move BrowserExtensionCard from Maintenance to Overview tab — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen 36cc0622cb fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:33:44 -04:00
bvandeusen e50f92d900 perf(ci): shard integration suite into 3 parallel jobs (int_api, int_imp, int_core) — newly feasible after act_runner capacity 2→6 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen ba8d9b112d fix(ext-ci): add diagnostic tracing to commit step to surface why run #309 reported success without producing the XPI side-commit — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 20:05:15 -04:00
bvandeusen c451061ca5 fix(ext-ci): self-retrigger workflow on its own edits (path filter includes workflow file) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:24:31 -04:00
bvandeusen ac55d0e8d8 Merge pull request 'fix(ext-ci): match AMO-renamed signed XPI' (#14) from dev into main 2026-05-25 18:22:50 -04:00
bvandeusen 47d760550d fix(ext-ci): glob AMO-renamed signed XPI + canonicalize to fabledcurator-<version>.xpi — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 18:22:01 -04:00
bvandeusen 89a89e0ded Merge pull request 'Release v26.05.25.3 — ML embedder SigLIP fix, import-UX, extension publish' (#13) from dev into main 2026-05-25 17:56:50 -04:00
bvandeusen dc3bce7fc1 chore(ext): bump to 1.0.1 to trigger initial sign-and-publish — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen f657582f30 feat(import-ui): deep scan button, sticky settings tabs, tasks-above-filters, fix Scanning-undefined source_path — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:37:10 -04:00
bvandeusen 111b952535 fix(ml): load SigLIP image-only processor to avoid SentencePiece dep — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> 2026-05-25 17:31:06 -04:00
bvandeusen 4e9aac2c05 Merge pull request 'v26.05.25.2: supersede + sidecar enrichment, scan toast feedback, CI uv + pip cache + durations' (#12) from dev into main 2026-05-25 14:30:25 -04:00
bvandeusen a0470b5f60 feat(importer): _supersede() now applies the new (larger) file's sidecar — operator wanted to scan GS download dir to supersede smaller IR-migrated images AND wire up gallery-dl Post metadata, but supersede was file-only and silently dropped the sidecar.
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.

Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:51:51 -04:00
bvandeusen b0bb7ae6cc ci: pip wheel cache (actions/cache on requirements.txt hash) + uv-when-available — ~2 min saved on warm runs, no risk
uv falls back to pip install on runners without uv binary, so this
change is forward-compatible with the current ci-python image. When
the runner image gets uv pre-installed in a future bump, the warm
install path drops from ~2 min to ~10 seconds.

pytest-xdist parallelization is OUT OF SCOPE for this commit:
tests/conftest.py uses a TRUNCATE ALL TABLES RESTART IDENTITY CASCADE
fixture after every integration test against a single shared
database; xdist workers running in parallel would nuke each other's
mid-test state. A future refactor to per-worker databases or
per-worker schema isolation is the prerequisite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:47:27 -04:00
bvandeusen 1bbe478fd0 ci: report slowest 25 integration tests via pytest --durations=25 — instrumentation pass before deciding parallelization vs targeted slow-test fixes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:09:25 -04:00
bvandeusen 5666fd5ca5 fix(ui): scan trigger immediate-feedback toast + delayed status re-poll — operator-flagged 'click does nothing' was actually scan_directory's skip-set finalizing the batch in <100ms when every file already had an ImportTask row, before refreshStatus could ever see the active state. Now the click always produces visible feedback (immediate 'Scan triggered' + 2s 'no new files' if it quick-finalizes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:06:11 -04:00
bvandeusen 2879ac6f2b Merge pull request 'v26.05.25.1: maintenance sweep + Camie v2 + corrupt-file handling + post-date gallery + clear-stuck escape hatch' (#11) from dev into main 2026-05-25 12:57:46 -04:00
bvandeusen 3a359f6c5e fix(ui): Quick scan button always visible (disabled when active batch present) + inline Clear stuck action — operator was clicking a spinner area thinking it was the button because activeBatch hid the Quick scan button entirely
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:43:37 -04:00
bvandeusen b6a917ac81 feat(import): /api/import/clear-stuck endpoint + Clear stuck UI button — escape hatch for the autoretry-loop case the automatic sweep can't break
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.

/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
  marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
  underlying cause is resolved

UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.

Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:37:07 -04:00
bvandeusen c361032554 feat(gallery): sort/group/jump by COALESCE(post.post_date, image_record.created_at) — surface migrated content at its original publish date, not FC scan date
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.

Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
  ImageProvenance (only if currently NULL — preserves the canonical
  download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
  COALESCE(post.post_date, image_record.created_at), used in:
    * scroll() ORDER BY + cursor WHERE clauses
    * timeline() year/month group-by
    * jump_cursor() year/month filter
    * _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
  works for images without a post (NULL on the Post side, fall back
  to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
  /scroll exposes both alongside the existing created_at so the UI
  can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.

Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.

To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:30:18 -04:00
bvandeusen 9f54efdedf fix(migrators): tag_apply phase 4 now covers deviantart + pixiv (was silently dropping IR PostMetadata from those platforms)
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.

Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.

To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:25:45 -04:00
bvandeusen 6b1bb87647 fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 11:12:55 -04:00
bvandeusen 68cffce322 fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.

Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:34:36 -04:00
bvandeusen 52445eb501 fix(ui): double directory card width (220px -> 440px) + bound preview slot height (min 150 / max 220 / overflow hidden / explicit display+object-position) so tall source images can't escape on browsers that don't compute aspect-ratio
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:32:05 -04:00
bvandeusen 3b3e7565fb fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output)
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.

Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:25:30 -04:00
bvandeusen 9d5abb09f6 fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed
scan_directory creates ImportTask rows with status='pending' (commit) then
in a second pass transitions to 'queued' + .delay() (commit). Crashes in
that window leave rows orphaned with no recovery path. Operator hit 5490
such rows 2026-05-25; the existing sweep only handled 'processing'.
Flipping to 'failed' (not re-enqueue) lets the operator drain via the
existing /api/import/retry-failed endpoint at their own pace.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 02:00:29 -04:00
120 changed files with 7272 additions and 803 deletions
+217 -5
View File
@@ -2,26 +2,232 @@ name: Build images
on:
push:
branches: [dev, main]
# `:dev` builds dropped 2026-05-26 — operator tests from `:latest` after
# merge-to-main, not from the dev branch image. Saves one full docker
# build per dev push.
branches: [main]
# Tag-push triggers an immutable per-version image build (e.g.
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
# negligible per tag. Doesn't overlap with the push-to-main build (that
# one publishes `:main` + `:latest`; the tag-push build publishes only
# `:<tag>`).
tags: ['v*']
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com)
# - write:release (for future release-cutting workflows)
# - write:release (for ext-<version> release asset cache)
# - write:issue (for future issue-management automation)
# The injected GITHUB_TOKEN cannot be used — it lacks write:package.
jobs:
build-web:
# Sign-or-fetch-from-cache: signs the extension via AMO if no ext-<version>
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
# Result is uploaded as an Actions artifact for build-web to consume.
#
# Why this lives in build.yml (not a separate workflow): the merge-commit's
# docker image tagged `:latest` MUST carry the XPI. A separate sign workflow
# racing build.yml leaves `:latest` without the XPI for ~5min (until the
# commit-back triggers another build). Inline ordering eliminates the race.
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
# branch to manage). AMO blocks re-signing the same version (returns 409),
# so signing is intentionally one-shot per version bump.
sign-extension:
if: github.ref == 'refs/heads/main'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Resolve extension version
id: extver
run: |
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Resolved extension version: $VERSION"
- name: Check Forgejo release-asset cache
id: cache
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eu
VERSION=${{ steps.extver.outputs.version }}
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
echo "Tag lookup HTTP status: $STATUS"
# JSON parsing via python (ci-python:3.14 has stdlib json; jq is
# not in the image and adding it per ci-requirements.md is not
# warranted for a single consumer — operator-flagged 2026-05-26
# after a sign job failed with `jq: not found`).
if [ "$STATUS" = "200" ]; then
ASSET_ID=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['id'] if xpis else '')")
if [ -n "$ASSET_ID" ]; then
echo "cached=true" >> "$GITHUB_OUTPUT"
echo "asset_id=$ASSET_ID" >> "$GITHUB_OUTPUT"
echo "Cached XPI exists at ext-$VERSION (asset id $ASSET_ID); skipping AMO sign"
else
echo "cached=false" >> "$GITHUB_OUTPUT"
echo "Release ext-$VERSION exists but has no .xpi asset; will re-sign + re-upload"
fi
else
echo "cached=false" >> "$GITHUB_OUTPUT"
echo "No release named ext-$VERSION; will sign via AMO and upload"
fi
# No "download cached XPI in sign-extension" step: build-web
# fetches directly from the Forgejo ext-<version> release asset
# (removed 2026-05-26 alongside the actions/upload-artifact
# removal — sign-extension's job is just to ensure the cache
# exists on Forgejo; the build-web side reads it independently).
- name: Sign via AMO (cache miss)
if: steps.cache.outputs.cached != 'true'
run: |
cd extension && npm install --no-save --no-audit --no-fund && npm run sign
env:
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
- name: Upload signed XPI to ext-<version> release (cache miss)
if: steps.cache.outputs.cached != 'true'
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=${{ steps.extver.outputs.version }}
# AMO renames signed XPIs with its internal addon-id-safe-string;
# canonicalize to fabledcurator-<version>.xpi so the FC server's
# whitelist (backend/app/frontend.py expects 'fabledcurator-*.xpi')
# keeps working.
SIGNED=$(ls extension/web-ext-artifacts/*.xpi | head -1)
XPI="extension/web-ext-artifacts/fabledcurator-$VERSION.xpi"
cp "$SIGNED" "$XPI"
# Find-or-create the ext-<version> release. Track whether WE
# created it so an upload failure below can roll back (don't
# leave an empty release tombstone that the next run's
# cache-check mistakes for a partial-failure state).
STATUS=$(curl -s -o release.json -w "%{http_code}" \
-H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
if [ "$STATUS" = "200" ]; then
CREATED_BY_US=false
else
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"main\"}" \
-o release.json \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
CREATED_BY_US=true
fi
RELEASE_ID=$(python3 -c "import json; print(json.load(open('release.json'))['id'])")
test -n "$RELEASE_ID"
# Rollback-on-failure: if the asset upload fails AND we just
# created the release in this run, delete it. Prevents an empty
# ext-<version> release from poisoning the next workflow run
# (operator-flagged 2026-05-26 — without rollback the next run
# saw 'release exists, no asset → cache miss → sign' which AMO
# then rejected with 409 'Version already exists').
rollback_if_we_created() {
if [ "$CREATED_BY_US" = "true" ]; then
echo "Rolling back: deleting just-created release $RELEASE_ID"
curl -s -X DELETE -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID" || true
curl -s -X DELETE -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/tags/ext-$VERSION" || true
fi
}
trap 'rollback_if_we_created' EXIT
HTTP_CODE=$(curl -s -X POST -H "Authorization: token $TOKEN" \
-F "attachment=@$XPI" \
-o /dev/null -w "%{http_code}" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID/assets?name=fabledcurator-$VERSION.xpi")
if [ "$HTTP_CODE" != "201" ] && [ "$HTTP_CODE" != "200" ]; then
echo "Asset upload failed with HTTP $HTTP_CODE"
exit 1
fi
# Upload succeeded — clear the rollback trap.
trap - EXIT
echo "Uploaded fabledcurator-$VERSION.xpi to ext-$VERSION release"
# No actions/upload-artifact step: Forgejo Actions (and our
# act_runner) doesn't support upload-artifact@v4+ (GHES limitation
# surfaced 2026-05-26). Instead build-web reads the signed XPI
# straight from the ext-<version> Forgejo release we just uploaded
# to. Same source of truth; no double-store.
build-web:
needs: [sign-extension]
# sign-extension is main-only; on dev it's skipped, build-web still runs.
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Download signed XPI from Forgejo release asset (main + tags)
# Fires on main-push AND on tag-push. Tag-push builds re-package the
# same source code as the preceding main-push build but with an
# immutable version tag — they need the XPI too, otherwise the
# versioned image ships without the signed extension.
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
run: |
set -eux
VERSION=$(grep -E '"version"' extension/package.json | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
# Look up the ext-<version> release; extract the .xpi asset's
# browser_download_url (Forgejo's /releases/assets/<id> endpoint
# returns ASSET METADATA, not the binary blob — operator-flagged
# 2026-05-26: my prior code curl'd the metadata endpoint without
# -f and wrote the resulting 404-page-not-found text into
# fabledcurator-*.xpi, which Firefox then rejected as "corrupt").
# browser_download_url is the canonical binary endpoint and is
# also publicly accessible (no token needed) but we pass the
# token anyway for symmetry with private-repo support.
curl -sf -H "Authorization: token $TOKEN" \
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" \
-o release.json
DOWNLOAD_URL=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['browser_download_url'])")
test -n "$DOWNLOAD_URL"
echo "Downloading XPI from: $DOWNLOAD_URL"
mkdir -p frontend/public/extension
DEST="frontend/public/extension/fabledcurator-$VERSION.xpi"
# -f = fail on HTTP error (prevents silent corruption like the
# 2026-05-26 incident); -L = follow redirects.
curl -sfL -H "Authorization: token $TOKEN" -o "$DEST" "$DOWNLOAD_URL"
# Sanity check: the binary should start with the ZIP magic (PK\x03\x04).
# If it's anything else, the next docker build will ship a corrupt XPI.
MAGIC=$(head -c 2 "$DEST" | od -An -c | tr -d ' \n')
if [ "$MAGIC" != "PK" ]; then
echo "ERROR: downloaded XPI does not start with ZIP magic 'PK' (got '$MAGIC')"
echo "File contents preview:"
head -c 200 "$DEST"
exit 1
fi
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
ls -la frontend/public/extension/
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Three trigger shapes:
# refs/tags/v… → tag-push: publish ONLY the immutable version
# tag (e.g. :v26.05.26.5). Don't touch :latest;
# that already got published by the main-push
# build for the merge commit.
# refs/heads/main → push to main (incl. PR merge commits):
# publish :main + :latest (floating).
# anything else → safety net; shouldn't fire given the `on:`
# config above (dev was dropped). Tag :dev to
# surface the unexpected run in the registry.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
@@ -52,7 +258,13 @@ jobs:
- name: Determine tag
id: tag
run: |
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Mirrors build-web's three-shape logic (tag-push / main-push /
# safety-net dev). The -ml image follows the same release cadence
# as the web image.
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
+183 -24
View File
@@ -8,8 +8,10 @@ name: CI
on:
push:
branches: [dev, main]
pull_request:
branches: [main]
# pull_request trigger intentionally absent — with branches: [dev, main]
# above, every PR commit already fires CI via the push event on dev. Adding
# pull_request would duplicate runs on dev→main PRs. FC has no fork PRs
# (single-operator Forgejo repo) so push coverage is complete.
jobs:
backend-lint-and-test:
@@ -24,11 +26,30 @@ jobs:
steps:
- uses: actions/checkout@v4
# Cache step removed 2026-05-26: act_runner's cache backend has been
# broken on this homelab runner since 2026-05-15 (first as request-
# timeout warnings, then as hard "Cannot find module .../dist/restore/
# index.js" failures that tank the whole job). The cache step targeted
# ~/.cache/pip but the install below uses `uv pip install` primarily,
# whose own cache lives at ~/.cache/uv — so the cache step's real
# benefit was marginal even when working. Cost of removal: ~30s of
# wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
# docker volume at the runner level (skips actions/cache entirely),
# or fix the runner-side cache backend (clear /var/run/act/actions/*,
# pin act_runner version, etc.).
- name: Install Python deps
# ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
# Dockerfile's RUFF_VERSION). Per FabledRulebook ci-runners.md, toolchain
# versions live on the runner image, not here.
run: pip install -r requirements.txt pytest pytest-asyncio
# uv: 5-10x faster wheel resolve than pip for cold caches.
# Falls back to pip install on uv-missing runners (older images).
run: |
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
- name: Ruff lint
run: ruff check backend/ tests/ alembic/
@@ -57,17 +78,28 @@ jobs:
- run: npm run test:unit
- run: npm run build
integration:
# This act_runner (swarm-runner v0.6.1) puts service containers on the
# default bridge with NO service-name DNS, and publishing fixed host
# ports collides with the operator's running docker-compose dev stack on
# the same shared daemon. Workaround: publish NO host ports, and reach
# each service by its bridge IP — discovered at runtime via the mounted
# docker socket (the ci-python image ships /usr/bin/docker). Default-bridge
# containers can talk by IP (only embedded DNS is missing), so IP
# addressing is reliable here. Everything runs in ONE step so resolved
# values don't depend on cross-step env passing. Pattern documented in
# FabledRulebook/forgejo.md "CI philosophy".
# Integration suite split into THREE parallel shards (2026-05-25, runner
# capacity bumped 2→6). Each shard gets its own Postgres + Redis service
# set and runs alembic + a disjoint subset of integration tests. Shards
# share no DB state, so the autouse TRUNCATE fixture in tests/conftest.py
# stays single-threaded per shard but multiple shards run in parallel
# wall-clock. Approximate split — rebalance once --durations=15 output
# reveals which shard is the long pole.
#
# Each shard's docker-ps filter uses its own unique job name to scope
# service-container resolution. act_runner appears to strip underscores
# from job names when building container labels — `int_api` yielded
# zero matches on 2026-05-25 — so shards use no-separator names
# (`intapi`, `intimp`, `intcore`) instead. Each step prints
# `docker ps -a` first so a future naming-convention shift surfaces in
# the log without another guess-and-push cycle.
#
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT
# done — per ci-requirements.md, FC is the only Python consumer of that
# image and the CI-Runner project's "add deps to image when used by >1
# project" rule keeps the install per-job.
intapi:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
@@ -98,15 +130,14 @@ jobs:
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Integration suite (resolve service IPs, migrate, test)
- name: API integration shard (resolve service IPs, migrate, test)
run: |
set -eux
# Scope to THIS job's service containers (act_runner names them
# ...JOB-integration...); the operator's compose stack uses the
# same images but different names, so it won't match.
PG=$(docker ps --filter "name=JOB-integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=JOB-integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intapi" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intapi" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
@@ -114,11 +145,139 @@ jobs:
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
# Wait for Postgres to accept TCP (bash /dev/tcp; no extra tools).
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
pip install -r requirements.txt pytest pytest-asyncio
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration
pytest tests/test_api_*.py -v -m integration --durations=15
intimp:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Importer integration shard (resolve service IPs, migrate, test)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intimp" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intimp" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/test_importer*.py tests/test_import_*.py tests/test_migration_*.py tests/test_phash_*.py tests/test_sidecar_*.py tests/test_scan_*.py tests/test_archive_extractor.py tests/test_backfill_phash.py -v -m integration --durations=15
intcore:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
run: |
set -eux
echo "=== container landscape (diagnostic for filter scoping) ==="
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
echo "=== end landscape ==="
PG=$(docker ps --filter "name=intcore" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
RD=$(docker ps --filter "name=intcore" --filter "ancestor=redis:7-alpine" -q | head -n1)
test -n "$PG" && test -n "$RD"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
test -n "$PG_IP" && test -n "$RD_IP"
export DB_HOST="$PG_IP"
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt pytest pytest-asyncio
else
pip install -r requirements.txt pytest pytest-asyncio
fi
alembic upgrade head
pytest tests/ -v -m integration --durations=15 \
--ignore-glob='tests/test_api_*.py' \
--ignore-glob='tests/test_importer*.py' \
--ignore-glob='tests/test_import_*.py' \
--ignore-glob='tests/test_migration_*.py' \
--ignore-glob='tests/test_phash_*.py' \
--ignore-glob='tests/test_sidecar_*.py' \
--ignore-glob='tests/test_scan_*.py' \
--ignore-glob='tests/test_archive_extractor.py' \
--ignore-glob='tests/test_backfill_phash.py'
+12 -42
View File
@@ -1,8 +1,19 @@
name: extension
# Lint-only workflow. The sign-and-publish dance moved into build.yml's
# `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
# because sign-extension runs as a build-web dependency in the SAME workflow,
# eliminating the prior race between build.yml and a separate extension.yml.
# Signed XPIs are cached in Forgejo Release Assets named `ext-<version>`.
on:
push:
branches: [dev, main]
paths: ['extension/**']
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
workflow_dispatch:
jobs:
@@ -16,44 +27,3 @@ jobs:
run: cd extension && npm install --no-save --no-audit --no-fund
- name: Lint
run: cd extension && npm run lint
sign-and-publish:
needs: lint
if: github.ref == 'refs/heads/main'
runs-on: python-ci
container:
image: node:22-bookworm-slim
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.RELEASE_TOKEN }}
- name: Install web-ext + git
run: |
apt-get update && apt-get install -y --no-install-recommends git ca-certificates
cd extension && npm install --no-save --no-audit --no-fund
- name: Sign XPI
run: cd extension && npm run sign
env:
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
- name: Commit signed XPI to frontend/public/extension/
run: |
set -e
XPI=$(ls extension/web-ext-artifacts/fabledcurator-*.xpi | head -1)
if [ -z "$XPI" ]; then
echo "No XPI produced by web-ext sign — exiting"
exit 1
fi
mkdir -p frontend/public/extension
cp "$XPI" frontend/public/extension/
# Also copy as -latest.xpi so the FC server can serve a stable URL.
cp "$XPI" "frontend/public/extension/fabledcurator-latest.xpi"
git config user.name "FC extension CI"
git config user.email "noreply@fabledsword.com"
git add frontend/public/extension/
if git diff --cached --quiet; then
echo "No changes to commit"
else
git commit -m "ext: publish signed XPI $(basename $XPI)"
git push origin HEAD:main
fi
@@ -0,0 +1,38 @@
"""import_batch.refreshed counter for deep-scan sidecar re-application
Revision ID: 0019
Revises: 0018
Create Date: 2026-05-25
Adds a `refreshed` counter to `import_batch`, mirroring the existing
`imported`/`skipped`/`failed`/`attachments` columns. Deep scan now
re-applies sidecar metadata to already-imported files (the IR feature
that didn't make the FC port the first time); a "refreshed" outcome
increments this counter so the UI can surface "X new, Y refreshed"
instead of the misleading "Scan complete — no new files" message.
server_default=0 backfills existing rows in place — no UPDATE needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0019"
down_revision: Union[str, None] = "0018"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_batch",
sa.Column(
"refreshed", sa.Integer(),
nullable=False, server_default=sa.text("0"),
),
)
def downgrade() -> None:
op.drop_column("import_batch", "refreshed")
@@ -0,0 +1,65 @@
"""fc-cleanup: library_audit_run table for async transparency/single_color audits
Revision ID: 0020
Revises: 0019
Create Date: 2026-05-26
The table backs the async audit lifecycle: rule + params snapshot, status
state machine ('running''ready''applied'/'cancelled'/'error'), and
the matched_ids JSONB array that the apply step deletes. Capped at 50k IDs
per row by the scan task (oversize = rule too aggressive, operator narrows
before re-running).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0020"
down_revision: Union[str, None] = "0019"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"library_audit_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rule", sa.String(32), nullable=False),
sa.Column("params", postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column(
"status", sa.String(16),
nullable=False, server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"scanned_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_ids", postgresql.JSONB(astext_type=sa.Text()),
nullable=False, server_default=sa.text("'[]'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
)
op.create_index(
"ix_library_audit_run_rule", "library_audit_run", ["rule"],
)
op.create_index(
"ix_library_audit_run_status", "library_audit_run", ["status"],
)
def downgrade() -> None:
op.drop_index("ix_library_audit_run_status", table_name="library_audit_run")
op.drop_index("ix_library_audit_run_rule", table_name="library_audit_run")
op.drop_table("library_audit_run")
@@ -0,0 +1,54 @@
"""provenance-race: dedupe + UNIQUE(image_record_id, post_id) on image_provenance
Revision ID: 0021
Revises: 0020
Create Date: 2026-05-26
Closes the race in Importer._apply_sidecar's existence-check + INSERT pattern.
Two workers writing for the same (image, post) pair both saw no existing row
and both inserted, leaving duplicates that then broke .scalar_one_or_none()
on every subsequent deep-scan rederive against those images
(MultipleResultsFound). Most plausibly seeded when the 5-min recovery sweep
re-enqueued a still-running long-import task and the second worker collided
with the first inside _apply_sidecar.
Migration steps:
1. DELETE all but min(id) per (image_record_id, post_id) pair. Operator's
DB had 2 affected pairs at write-time; harmless no-op if zero.
2. Add UNIQUE constraint so the importer's new savepoint+IntegrityError
recovery path can trip on collision and re-select, mirroring
uq_source_artist_platform_url and uq_post_source_external_id.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0021"
down_revision: Union[str, None] = "0020"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"""
DELETE FROM image_provenance ip1
USING image_provenance ip2
WHERE ip1.image_record_id = ip2.image_record_id
AND ip1.post_id = ip2.post_id
AND ip1.id > ip2.id
"""
)
op.create_unique_constraint(
"uq_image_provenance_image_post",
"image_provenance",
["image_record_id", "post_id"],
)
def downgrade() -> None:
op.drop_constraint(
"uq_image_provenance_image_post",
"image_provenance",
type_="unique",
)
@@ -0,0 +1,223 @@
"""source-collapse: one Source per (artist, platform) — consolidate junk per-post Sources
Revision ID: 0022
Revises: 0021
Create Date: 2026-05-26
Closes the operator-flagged 2026-05-26 issue where the filesystem importer
called _find_or_create_source(url=sd.post_url), creating one Source row per
imported post URL. Operator's Atole artist had 406 Source rows where there
should have been 1 (the /cw/Atole subscription Source).
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). Posts hang off it. The
filesystem importer was misusing Source as a per-post key.
Migration steps per (artist_id, platform) group with >1 Source:
1. Pick canonical — prefer a URL NOT matching '/posts/<id>$' (real
campaign URL like /cw/Atole); else min(id).
2. PRE-merge any Posts under non-canonical sources whose
external_post_id ALREADY exists under the canonical source. (Same
gallery-dl post imported via two different sidecar paths can plant
two Post rows with identical external_post_id under different
Sources for the same artist.) Repoint ImageProvenance +
ImageRecord.primary_post_id to the canonical-side Post, dedupe
ImageProvenance against alembic 0021's uq, then delete the
non-canonical-side Post. This MUST happen before step 3 — Postgres
fires uq_post_source_external_id row-by-row during the bulk UPDATE
and the merge-after-reparent ordering 500s on first collision
(operator-hit during v26.05.26.1 deploy, 2026-05-26).
3. Reparent remaining Posts onto canonical (no collisions possible now).
4. Reparent ImageProvenance.source_id off the non-canonical sources.
5. Delete the orphan Source rows.
6. If the canonical Source's URL still looks like a per-post URL (no
campaign URL existed among candidates), rewrite it to
'sidecar:<platform>:<artist_slug>' so the artist detail page shows
something readable.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0022"
down_revision: Union[str, None] = "0021"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_POST_URL_RE = r"/posts/[^/]+$"
def upgrade() -> None:
conn = op.get_bind()
# Find (artist_id, platform) groups with > 1 Source row.
groups = conn.execute(text("""
SELECT artist_id, platform
FROM source
GROUP BY artist_id, platform
HAVING COUNT(*) > 1
""")).fetchall()
for artist_id, platform in groups:
rows = conn.execute(
text("""
SELECT id, url FROM source
WHERE artist_id = :a AND platform = :p
ORDER BY id ASC
"""),
{"a": artist_id, "p": platform},
).fetchall()
# Canonical: first row whose URL doesn't look like a per-post URL;
# else min(id).
canonical_id = None
for sid, url in rows:
if not _matches_post_url(url):
canonical_id = sid
break
if canonical_id is None:
canonical_id = rows[0][0]
other_ids = [sid for sid, _ in rows if sid != canonical_id]
if not other_ids:
continue
# STEP 2: PRE-merge ALL Posts with duplicate external_post_id
# across the entire (canonical + others) group, BEFORE the bulk
# reparent. Two cases must both be handled:
# (A) canonical has Post X with epid=N; an "other" source has
# Post Y with epid=N → after bulk UPDATE, (canonical, N)
# collides with itself.
# (B) two different "other" sources each have a Post with
# epid=N; canonical has none → after bulk UPDATE, both
# are repointed to (canonical, N) and the second collides.
# The earlier version of this migration only handled (A); the
# operator's deploy 2026-05-26 tripped (B) at line 139.
# Fix: group ALL Posts in the (artist, platform) by epid; for
# any group with count>1, pick the keep (prefer one already
# under canonical; else lowest id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:others)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "others": other_ids},
).fetchall()
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
# Prefer a Post already under canonical as the keep.
canonical_posts = [p for p in posts if p[1] == canonical_id]
if canonical_posts:
keep_id = canonical_posts[0][0]
else:
keep_id = posts[0][0] # already sorted by id ASC
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id ALREADY has a provenance under keep —
# the UPDATE below would otherwise repoint them and
# trip uq_image_provenance_image_post (alembic 0021)
# row-by-row before any after-the-fact dedupe could
# run. Operator's v26.05.26.3 deploy 2026-05-26 tripped
# this at line 123.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
# Now safe to repoint the survivors.
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP 3: Bulk reparent the remaining Posts off the other
# Sources. After step 2, no collisions on
# (canonical, external_post_id) are possible.
conn.execute(
text("""
UPDATE post SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 4: Reparent ImageProvenance.source_id (denormalized FK).
# No UNIQUE on source_id; safe bulk update.
conn.execute(
text("""
UPDATE image_provenance SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 5: Drop the orphan Sources.
conn.execute(
text("DELETE FROM source WHERE id = ANY(:others)"),
{"others": other_ids},
)
# If the canonical's URL still looks per-post (no campaign URL
# existed among the candidates), rewrite to a synthetic anchor so
# the artist detail page renders something readable.
canonical_url = conn.execute(
text("SELECT url FROM source WHERE id = :id"),
{"id": canonical_id},
).scalar_one()
if _matches_post_url(canonical_url):
slug = conn.execute(
text("SELECT slug FROM artist WHERE id = :id"),
{"id": artist_id},
).scalar_one()
conn.execute(
text("""
UPDATE source
SET url = :new_url, enabled = false
WHERE id = :id
"""),
{
"id": canonical_id,
"new_url": f"sidecar:{platform}:{slug}",
},
)
def downgrade() -> None:
# Lossy migration — orphan Sources deleted, Posts reparented, Posts
# merged. No safe downgrade. If you need to roll back the schema
# invariant, fork from 0021 and re-run filesystem imports.
pass
def _matches_post_url(url: str) -> bool:
"""True if url ends with /posts/<token> (gallery-dl-style per-post URL)."""
import re
return bool(re.search(_POST_URL_RE, url or ""))
@@ -0,0 +1,99 @@
"""drop meta + rating tag kinds — operator-retired 2026-05-26
Revision ID: 0023
Revises: 0022
Create Date: 2026-05-26
Operator decided meta + rating aren't valid tag kinds for FC. Per-row
behavior: DELETE existing rows (operator chose "clean break" over
"convert to general"). All cascading FKs (image_tag, tag_alias,
tag_allowlist, tag_reference_embedding, tag_suggestion_rejection,
series_page) use ondelete="CASCADE" so a single DELETE on tag cleans
the related rows in one go.
After the data cleanup, recreate the tag_kind ENUM without 'meta' /
'rating' (Postgres has no `ALTER TYPE ... DROP VALUE`; standard
rename-create-cast-drop dance). The server default 'general' is
dropped before the type swap and restored after.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0023"
down_revision: Union[str, None] = "0022"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Delete tags of the retired kinds. CASCADE handles related tables.
op.execute("DELETE FROM tag WHERE kind IN ('meta', 'rating')")
# 2. Drop the CHECK constraint that references the enum's literal
# values. Postgres can't resolve `kind = 'character'` across the
# type swap below — the literal would bind to the new tag_kind
# but the column is on tag_kind_old, producing
# "operator does not exist: tag_kind = tag_kind_old".
# (Operator-hit during the v26.05.26.5 deploy attempt; ck was
# originally added by alembic 0002.) Recreated post-swap.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
# 3. Drop the server default — ALTER COLUMN TYPE can't carry it
# across the type swap below.
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
# 4. Recreate the tag_kind enum without meta/rating.
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
# 5. Restore the server default.
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
# 6. Restore the CHECK constraint (now bound to the new tag_kind).
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
def downgrade() -> None:
# Add the values back to the enum so old code can boot. The deleted
# tag rows are gone permanently — no safe restore.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post', 'meta', 'rating'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
+38 -1
View File
@@ -3,13 +3,23 @@
import logging
from pathlib import Path
from quart import Quart
from quart import Quart, request
from .api import all_blueprints
from .config import get_config
from .frontend import frontend_bp
from .services.credential_crypto import CredentialCrypto
# Browser-extension origins. The FabledCurator extension fetches from
# moz-extension://<uuid>/ on Firefox and chrome-extension://<uuid>/ on
# Chromium-based browsers. Operator-flagged 2026-05-26: extension's
# 'Test connection' returned `NetworkError` because the X-Extension-Key
# header on /api/credentials triggers a CORS preflight that our routes
# don't handle. Whitelisting only these two schemes (not opening CORS
# up generally) lets the extension talk to a plain-HTTP self-hosted FC
# without weakening the no-CORS posture for normal browser usage.
_EXTENSION_ORIGIN_SCHEMES = ("moz-extension://", "chrome-extension://")
_CREDENTIAL_KEY_PATH = Path("/images/secrets/credential_key.b64")
@@ -35,6 +45,33 @@ def create_app() -> Quart:
# Registered last so /api/* routes win over the SPA catch-all.
app.register_blueprint(frontend_bp)
@app.before_request
async def _extension_cors_preflight():
# Short-circuit OPTIONS preflight from the browser extension with a
# 204 + CORS headers (the after_request hook below adds them).
# Without this, OPTIONS lands on routes that only declared POST/GET
# methods and 405s before the after_request gets a chance.
if request.method != "OPTIONS":
return None
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
return "", 204
return None
@app.after_request
async def _extension_cors_headers(response):
origin = request.headers.get("Origin", "")
if any(origin.startswith(s) for s in _EXTENSION_ORIGIN_SCHEMES):
response.headers["Access-Control-Allow-Origin"] = origin
response.headers["Access-Control-Allow-Methods"] = (
"GET, POST, PATCH, DELETE, OPTIONS"
)
response.headers["Access-Control-Allow-Headers"] = (
"Content-Type, X-Extension-Key"
)
response.headers["Access-Control-Max-Age"] = "86400"
return response
@app.after_serving
async def _dispose_db_engine() -> None:
from .extensions import dispose_engine
+4
View File
@@ -20,6 +20,7 @@ def all_blueprints() -> list[Blueprint]:
from .artist import artist_bp
from .artists import artists_bp
from .attachments import attachments_bp
from .cleanup import cleanup_bp
from .credentials import credentials_bp
from .downloads import downloads_bp
from .extension import extension_bp
@@ -37,6 +38,7 @@ def all_blueprints() -> list[Blueprint]:
from .system_activity import system_activity_bp
from .system_backup import system_backup_bp
from .tags import tags_bp
from .thumbnails import thumbnails_bp
return [
api_bp,
attachments_bp,
@@ -50,12 +52,14 @@ def all_blueprints() -> list[Blueprint]:
system_activity_bp,
system_backup_bp,
admin_bp,
cleanup_bp,
import_admin_bp,
migrate_bp,
suggestions_bp,
allowlist_bp,
aliases_bp,
ml_admin_bp,
thumbnails_bp,
sources_bp,
platforms_bp,
posts_bp,
+193
View File
@@ -0,0 +1,193 @@
"""FC-Cleanup: /api/cleanup/* — retroactive enforcement of import filters.
Endpoints:
POST /min-dimension/preview synchronous SQL audit
POST /min-dimension/delete synchronous SQL delete (Tier-C token)
POST /audit async transparency / single_color start
GET /audit list recent audit_run rows
GET /audit/<id> single audit_run row
POST /audit/<id>/apply apply matched_ids deletes (Tier-C token)
POST /audit/<id>/cancel flip running audit to cancelled
Unused-tags retroactive prune intentionally NOT in this namespace —
TagMaintenanceCard (Maintenance tab → moved to Cleanup tab in v26.05.25.7)
uses the existing /api/admin/tags/prune-unused endpoint via the admin
store. No duplicate route here.
Confirm-token format matches modal/DestructiveConfirmModal.vue convention:
`delete-min-dim-<sha8(w,h)>` for min-dim delete
`delete-audit-<id>` for audit apply
(Modal hardcodes action ∈ {'restore', 'delete'}; "apply audit" is semantically a delete of the matched images, so we use `delete-audit-<id>`.)
"""
from __future__ import annotations
import hashlib
from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import LibraryAuditRun
from ..services import cleanup_service
cleanup_bp = Blueprint("cleanup", __name__, url_prefix="/api/cleanup")
IMAGES_ROOT = Path("/images")
def _bad(error: str, *, status: int = 400, **extra):
body = {"error": error}
body.update(extra)
return jsonify(body), status
def _min_dim_token(min_w: int, min_h: int) -> str:
# SHA-256 (not MD5) — Web Crypto's subtle.digest rejects MD5; both
# sides use SHA-256 truncated to 8 hex chars.
canon = f"{min_w}x{min_h}"
return f"delete-min-dim-{hashlib.sha256(canon.encode()).hexdigest()[:8]}"
def _serialize_audit_run(audit: LibraryAuditRun) -> dict:
return {
"id": audit.id,
"rule": audit.rule,
"params": audit.params,
"status": audit.status,
"started_at": audit.started_at.isoformat() if audit.started_at else None,
"finished_at": audit.finished_at.isoformat() if audit.finished_at else None,
"scanned_count": audit.scanned_count,
"matched_count": audit.matched_count,
"matched_ids": audit.matched_ids,
"error": audit.error,
}
@cleanup_bp.route("/min-dimension/preview", methods=["POST"])
async def min_dim_preview():
body = await request.get_json(silent=True) or {}
try:
min_w = int(body.get("min_width", 0))
min_h = int(body.get("min_height", 0))
except (TypeError, ValueError):
return _bad("invalid_dimensions")
if min_w < 0 or min_h < 0:
return _bad("invalid_dimensions")
async with get_session() as session:
projection = await session.run_sync(
lambda s: cleanup_service.project_min_dimension_violations(
s, min_width=min_w, min_height=min_h,
)
)
return jsonify(projection)
@cleanup_bp.route("/min-dimension/delete", methods=["POST"])
async def min_dim_delete():
body = await request.get_json(silent=True) or {}
try:
min_w = int(body.get("min_width", 0))
min_h = int(body.get("min_height", 0))
except (TypeError, ValueError):
return _bad("invalid_dimensions")
if min_w < 0 or min_h < 0:
return _bad("invalid_dimensions")
supplied = body.get("confirm", "")
expected = _min_dim_token(min_w, min_h)
if supplied != expected:
return _bad("confirm_mismatch", expected=expected)
async with get_session() as session:
deleted = await session.run_sync(
lambda s: cleanup_service.delete_min_dimension_violations(
s, min_width=min_w, min_height=min_h, images_root=IMAGES_ROOT,
)
)
await session.commit()
return jsonify({"deleted": deleted})
@cleanup_bp.route("/audit", methods=["POST"])
async def audit_create():
body = await request.get_json(silent=True) or {}
rule = body.get("rule")
params = body.get("params") or {}
if rule not in ("transparency", "single_color"):
return _bad("invalid_rule")
if not isinstance(params, dict):
return _bad("invalid_params")
async with get_session() as session:
try:
audit_id = await session.run_sync(
lambda s: cleanup_service.start_audit_run(
s, rule=rule, params=params,
)
)
except cleanup_service.AuditAlreadyRunning as running_id:
return _bad(
"audit_already_running", status=409,
running_id=int(str(running_id)),
)
except ValueError as exc:
return _bad(str(exc))
await session.commit()
return jsonify({"audit_id": audit_id, "status": "running"}), 202
@cleanup_bp.route("/audit/<int:audit_id>", methods=["GET"])
async def audit_get(audit_id: int):
async with get_session() as session:
audit = (await session.execute(
select(LibraryAuditRun).where(LibraryAuditRun.id == audit_id)
)).scalar_one_or_none()
if audit is None:
return _bad("not_found", status=404)
return jsonify(_serialize_audit_run(audit))
@cleanup_bp.route("/audit", methods=["GET"])
async def audit_history():
try:
limit = min(int(request.args.get("limit", "20")), 100)
except ValueError:
return _bad("invalid_limit")
async with get_session() as session:
rows = (await session.execute(
select(LibraryAuditRun)
.order_by(LibraryAuditRun.id.desc())
.limit(limit)
)).scalars().all()
return jsonify({"runs": [_serialize_audit_run(r) for r in rows]})
@cleanup_bp.route("/audit/<int:audit_id>/apply", methods=["POST"])
async def audit_apply(audit_id: int):
body = await request.get_json(silent=True) or {}
confirm = body.get("confirm", "")
async with get_session() as session:
try:
deleted = await session.run_sync(
lambda s: cleanup_service.apply_audit_run(
s, audit_id=audit_id, confirm_token=confirm,
images_root=IMAGES_ROOT,
)
)
except cleanup_service.AuditNotReady as exc:
return _bad("audit_not_ready", current_status=str(exc))
except cleanup_service.ConfirmTokenMismatch as exc:
return _bad("confirm_mismatch", expected=str(exc))
except ValueError as exc:
return _bad("not_found", status=404, detail=str(exc))
await session.commit()
return jsonify({"deleted": deleted})
@cleanup_bp.route("/audit/<int:audit_id>/cancel", methods=["POST"])
async def audit_cancel(audit_id: int):
async with get_session() as session:
await session.run_sync(
lambda s: cleanup_service.cancel_audit_run(s, audit_id=audit_id)
)
await session.commit()
return jsonify({"cancelled": True})
+13 -3
View File
@@ -93,10 +93,20 @@ def _read_manifest_sync() -> dict | None:
asyncio.to_thread (ASYNC240: no pathlib I/O in async functions)."""
if not XPI_DIR.is_dir():
return None
xpis = sorted(XPI_DIR.glob("fabledcurator-*.xpi"), key=lambda p: p.stat().st_mtime)
if not xpis:
# Exclude the `fabledcurator-latest.xpi` alias when picking the file to
# extract a version from — it's a copy of the latest versioned XPI,
# written at the same mtime by build.yml, and would otherwise tie or
# win the sort (operator-flagged 2026-05-26: UI displayed "v latest"
# because `_extract_version("fabledcurator-latest.xpi")` returns
# the literal "latest"). The alias still serves as `latest_url`.
versioned = [
p for p in XPI_DIR.glob("fabledcurator-*.xpi")
if p.name != "fabledcurator-latest.xpi"
]
if not versioned:
return None
latest = xpis[-1]
versioned.sort(key=lambda p: p.stat().st_mtime)
latest = versioned[-1]
return {
"installed": True,
"version": _extract_version(latest.name),
+2
View File
@@ -42,6 +42,8 @@ async def scroll():
"width": i.width,
"height": i.height,
"created_at": i.created_at.isoformat(),
"posted_at": i.posted_at.isoformat() if i.posted_at else None,
"effective_date": i.effective_date.isoformat(),
"thumbnail_url": i.thumbnail_url,
"artist": i.artist,
}
+91 -8
View File
@@ -47,10 +47,13 @@ async def status():
if active:
payload["active_batch"] = {
"id": active.id,
"source_path": active.source_path,
"scan_mode": active.scan_mode,
"total_files": active.total_files,
"imported": active.imported,
"skipped": active.skipped,
"failed": active.failed,
"refreshed": active.refreshed,
"started_at": active.started_at.isoformat(),
}
return jsonify(payload)
@@ -100,17 +103,22 @@ async def list_tasks():
@import_admin_bp.route("/retry-failed", methods=["POST"])
async def retry_failed():
# Fold SELECT into UPDATE…WHERE…RETURNING — the prior SELECT-then-
# UPDATE-WHERE-id-IN pattern blew past psycopg's 65535-parameter
# ceiling once failed_ids exceeded ~65k rows.
async with get_session() as session:
failed_ids = (
await session.execute(select(ImportTask.id).where(ImportTask.status == "failed"))
).scalars().all()
result = await session.execute(
update(ImportTask)
.where(ImportTask.status == "failed")
.values(
status="queued", error=None,
started_at=None, finished_at=None,
)
.returning(ImportTask.id)
)
failed_ids = [row[0] for row in result.all()]
if not failed_ids:
return jsonify({"retried": 0})
await session.execute(
update(ImportTask)
.where(ImportTask.id.in_(failed_ids))
.values(status="queued", error=None, started_at=None, finished_at=None)
)
await session.commit()
from ..tasks.import_file import import_media_file
@@ -120,6 +128,81 @@ async def retry_failed():
return jsonify({"retried": len(failed_ids)})
@import_admin_bp.route("/clear-stuck", methods=["POST"])
async def clear_stuck():
"""Force any non-terminal ImportTask (status in pending/queued/
processing) to 'failed' AND finalize any ImportBatch that ends up
with no active children. Escape hatch for the operator when the
automatic recover_interrupted_tasks sweep keeps re-queueing the
same stuck row forever (e.g., underlying file is genuinely broken
and the import keeps OSError-looping at PIL load).
Idempotent + non-destructive: rows survive as 'failed' so the
Retry-Failed button can re-attempt them once whatever was broken
is fixed. Banked 2026-05-25 — operator hit 3 large PNGs that
autoretry-looped for 2 days after a corrupt-data PIL OSError.
"""
async with get_session() as session:
# Fold SELECT into UPDATE…WHERE — see /retry-failed for the
# 65535-parameter ceiling rationale. rowcount is enough here
# because we don't need the ids afterward (no .delay()).
clear_result = await session.execute(
update(ImportTask)
.where(
ImportTask.status.in_(["pending", "queued", "processing"])
)
.values(
status="failed",
finished_at=datetime.now(UTC),
error=(
"manually cleared via /api/import/clear-stuck "
"— stuck in non-terminal state; retry once "
"underlying cause (corrupt file, missing model, "
"etc.) is resolved"
),
)
)
tasks_failed = clear_result.rowcount or 0
# Finalize any 'running' ImportBatch that no longer has any
# active children. The "Scanning..." banner is driven by
# /api/import/status finding a running batch; left untouched,
# it would persist forever after the stuck-task clear.
running_batches = (
await session.execute(
select(ImportBatch.id).where(ImportBatch.status == "running")
)
).scalars().all()
finalized_batches = 0
for batch_id in running_batches:
still_active = (
await session.execute(
select(ImportTask.id)
.where(ImportTask.batch_id == batch_id)
.where(ImportTask.status.in_(
["pending", "queued", "processing"]
))
.limit(1)
)
).scalar_one_or_none()
if still_active is None:
await session.execute(
update(ImportBatch)
.where(ImportBatch.id == batch_id)
.values(
status="complete",
finished_at=datetime.now(UTC),
)
)
finalized_batches += 1
await session.commit()
return jsonify({
"tasks_failed": tasks_failed,
"batches_finalized": finalized_batches,
})
@import_admin_bp.route("/clear-completed", methods=["POST"])
async def clear_completed():
body = await request.get_json(silent=True) or {}
+32 -5
View File
@@ -15,6 +15,7 @@ from ..services.tag_service import (
TagService,
TagValidationError,
)
from ..utils.tag_prefix import parse_kind_prefix
tags_bp = Blueprint("tags", __name__, url_prefix="/api")
@@ -105,13 +106,39 @@ async def directory():
@tags_bp.route("/tags", methods=["POST"])
async def create_tag():
"""Create a tag. Two input shapes accepted:
1. Explicit: {name, kind, fandom_id?} — caller already split, kind wins.
2. IR-suffix: {name} where name = "kind:Name" (e.g. "artist:Eric").
The server runs parse_kind_prefix(name) to derive kind; the colon
and prefix are stripped from the stored tag name. If no recognized
prefix is present, the kind defaults to `general`.
Explicit kind ALWAYS wins (backward-compat for existing callers).
"""
body = await request.get_json()
if not body or "name" not in body or "kind" not in body:
return jsonify({"error": "name and kind required"}), 400
if not body or "name" not in body:
return jsonify({"error": "name required"}), 400
name = body["name"]
kind = _coerce_kind(body["kind"])
if kind is None:
return jsonify({"error": f"invalid kind {body['kind']!r}"}), 400
explicit_kind_raw = body.get("kind")
if explicit_kind_raw is not None:
# Caller provided kind — honor it; don't re-parse.
kind = _coerce_kind(explicit_kind_raw)
if kind is None:
return jsonify({"error": f"invalid kind {explicit_kind_raw!r}"}), 400
else:
# IR-style: parse "kind:Name" from the raw name.
parsed_kind, parsed_name = parse_kind_prefix(name)
if parsed_kind is not None:
name = parsed_name
kind = _coerce_kind(parsed_kind)
# parse_kind_prefix only returns kinds from KNOWN_KINDS which
# are all valid TagKind members, so _coerce_kind can't return
# None here — but defensive.
if kind is None:
return jsonify({"error": f"invalid kind {parsed_kind!r}"}), 400
else:
kind = TagKind.general
fandom_id = body.get("fandom_id")
async with get_session() as session:
+13
View File
@@ -0,0 +1,13 @@
"""Thumbnail admin API: backfill trigger."""
from quart import Blueprint, jsonify
thumbnails_bp = Blueprint("thumbnails", __name__, url_prefix="/api/thumbnails")
@thumbnails_bp.route("/backfill", methods=["POST"])
async def trigger_backfill():
from ..tasks.thumbnail import backfill_thumbnails
r = backfill_thumbnails.delay()
return jsonify({"celery_task_id": r.id}), 202
+2
View File
@@ -33,6 +33,7 @@ def make_celery() -> Celery:
"backend.app.tasks.download",
"backend.app.tasks.backup",
"backend.app.tasks.admin",
"backend.app.tasks.library_audit",
],
)
app.conf.update(
@@ -47,6 +48,7 @@ def make_celery() -> Celery:
"backend.app.tasks.migration.*": {"queue": "maintenance"},
"backend.app.tasks.backup.*": {"queue": "maintenance"},
"backend.app.tasks.admin.*": {"queue": "maintenance"},
"backend.app.tasks.library_audit.*": {"queue": "maintenance"},
},
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True,
+2
View File
@@ -11,6 +11,7 @@ from .image_record import ImageRecord
from .import_batch import ImportBatch
from .import_settings import ImportSettings
from .import_task import ImportTask
from .library_audit_run import LibraryAuditRun
from .migration_run import MigrationRun
from .ml_settings import MLSettings
from .post import Post
@@ -43,6 +44,7 @@ __all__ = [
"ImportBatch",
"ImportTask",
"ImportSettings",
"LibraryAuditRun",
"MLSettings",
"MigrationRun",
"TagAlias",
+15 -5
View File
@@ -1,14 +1,18 @@
"""ImageProvenance — links an ImageRecord to a Post.
Many-to-one (one image, many provenance rows) enables the enrich-on-duplicate
rule (spec §3): when a downloaded image is a pHash dupe of an existing
record, we append a new provenance row to the existing record rather than
dropping the metadata.
One image can have many provenance rows — different posts each contribute
metadata (enrich-on-duplicate rule, spec §3: a downloaded image that is a
pHash dupe of an existing record gets a NEW provenance row for the new post
appended, rather than the metadata being dropped). But the (image, post)
pair is unique — alembic 0021 enforces uq_image_provenance_image_post
after operator-flagged 2026-05-26 saw _apply_sidecar's existence-check +
INSERT race plant duplicates that then broke .scalar_one_or_none() on
every later deep-scan rederive (MultipleResultsFound).
"""
from datetime import datetime
from sqlalchemy import JSON, DateTime, ForeignKey, Integer, func
from sqlalchemy import JSON, DateTime, ForeignKey, Integer, UniqueConstraint, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
@@ -16,6 +20,12 @@ from .base import Base
class ImageProvenance(Base):
__tablename__ = "image_provenance"
__table_args__ = (
UniqueConstraint(
"image_record_id", "post_id",
name="uq_image_provenance_image_post",
),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
image_record_id: Mapped[int] = mapped_column(
+4
View File
@@ -26,6 +26,10 @@ class ImportBatch(Base):
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# Deep-scan only: count of already-imported files whose sidecar metadata
# got re-applied this run (post/source/provenance upsert). Stays 0 on
# quick-scan batches. See `Importer.import_one(deep_scan=True)`.
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True)
# running | complete | cancelled
+37
View File
@@ -0,0 +1,37 @@
"""LibraryAuditRun — async transparency / single_color audit lifecycle.
State machine: running → ready → applied / cancelled / error.
matched_ids JSONB is appended-to by scan_library_for_rule; apply_audit_run
reads it and routes through cleanup_service.delete_images.
"""
from datetime import datetime
from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class LibraryAuditRun(Base):
__tablename__ = "library_audit_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
rule: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True,
)
# running | ready | applied | cancelled | error
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(),
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
+4 -2
View File
@@ -35,8 +35,10 @@ class TagKind(StrEnum):
series = "series"
archive = "archive"
post = "post"
meta = "meta"
rating = "rating"
# `meta` and `rating` retired by operator 2026-05-26 (alembic 0023).
# `artist` retired in FC-2d-vii-c — artists are first-class entities
# via Artist/Source rows now, not tags — but the enum value stays
# to keep historic tag rows queryable.
image_tag = Table(
+21 -2
View File
@@ -25,12 +25,31 @@ def _snapshot(repo_id: str, dest: Path, allow_patterns: list[str] | None) -> Non
def ensure_camie() -> None:
"""Fetch Camie v2 weights + metadata.
v2 layout (HuggingFace Camais03/camie-tagger-v2): the ONNX file is
named camie-tagger-v2.onnx (not model.onnx) and tags ship inside
camie-tagger-v2-metadata.json (not selected_tags.csv). Both at root.
The repo also contains app/, game/, training/, images/ subdirs full
of setup/demo files we don't need — allow_patterns scopes the fetch
to just the inference essentials (~790 MB instead of ~2 GB).
"""
dest = MODEL_ROOT / "camie"
if (dest / "model.onnx").is_file() and (dest / "selected_tags.csv").is_file():
model_file = dest / "camie-tagger-v2.onnx"
meta_file = dest / "camie-tagger-v2-metadata.json"
if model_file.is_file() and meta_file.is_file():
print(f"[download_models] Camie present at {dest}")
return
print(f"[download_models] Fetching {CAMIE_REPO} -> {dest}")
_snapshot(CAMIE_REPO, dest, ["model.onnx", "selected_tags.csv", "*.json"])
_snapshot(
CAMIE_REPO, dest,
[
"camie-tagger-v2.onnx",
"camie-tagger-v2-metadata.json",
"config.json",
"config.yaml",
],
)
def ensure_siglip() -> None:
+10
View File
@@ -111,12 +111,22 @@ class ArtistService:
)
).all()
post_count = (
await self.session.execute(
select(func.count(func.distinct(Post.id)))
.select_from(Post)
.join(Source, Source.id == Post.source_id)
.where(Source.artist_id == aid)
)
).scalar_one()
return {
"id": artist.id,
"name": artist.name,
"slug": artist.slug,
"is_subscription": bool(artist.is_subscription),
"image_count": int(image_count),
"post_count": int(post_count),
"date_range": {
"min": dmin.isoformat() if dmin else None,
"max": dmax.isoformat() if dmax else None,
+6
View File
@@ -0,0 +1,6 @@
"""Audit rule modules. Each module exposes evaluate(pil_image, **params) -> bool.
The retroactive library-cleanup tab and (future) import-time filter logic
both consume these. Importers should NOT inline rule logic going forward;
add the rule here and call from both sides.
"""
@@ -0,0 +1,53 @@
"""Single-color audit: matches images where one color dominates beyond
the threshold (within the given Euclidean RGB tolerance). The first
canonical implementation — the import-side filter (SkipReason.single_color)
was never wired; FC-Cleanup's audit module is the source of truth and a
future spec can adopt it on the import path too.
"""
from PIL import Image
_THUMB_SIZE = (64, 64)
def evaluate(
pil_image,
*,
threshold: float,
tolerance: int,
) -> bool:
"""True iff the fraction of pixels within `tolerance` (Euclidean RGB
distance) of the dominant color exceeds `threshold`.
Downsamples to 64x64 for speed (~4ms regardless of source size).
Alpha channels are stripped; only RGB is considered. Animated images
use frame 0 (PIL's default after Image.open without seek).
"""
im = pil_image
if im.mode == "RGBA":
im = im.convert("RGB")
elif im.mode not in ("RGB", "L"):
im = im.convert("RGB")
if im.size != _THUMB_SIZE:
im = im.resize(_THUMB_SIZE, Image.Resampling.BILINEAR)
pixels = list(im.getdata())
if not pixels:
return False
# Normalize L-mode pixels to RGB tuples for distance math.
if isinstance(pixels[0], int):
pixels = [(p, p, p) for p in pixels]
# Dominant color = mean RGB.
n = len(pixels)
sum_r = sum(p[0] for p in pixels)
sum_g = sum(p[1] for p in pixels)
sum_b = sum(p[2] for p in pixels)
dom = (sum_r / n, sum_g / n, sum_b / n)
tol_sq = tolerance * tolerance
within = 0
for r, g, b in pixels:
dr = r - dom[0]
dg = g - dom[1]
db = b - dom[2]
if dr * dr + dg * dg + db * db <= tol_sq:
within += 1
return (within / n) > threshold
@@ -0,0 +1,27 @@
"""Transparency audit: matches images whose transparent-pixel fraction
exceeds the threshold. Animated images short-circuit (skipped) to avoid
the multi-frame PIL decode that hits Celery's hard time limit."""
def evaluate(pil_image, *, threshold: float) -> bool:
"""True iff the image's transparent-pixel fraction exceeds threshold.
False for non-alpha modes and animated images. Mirrors the import-side
Importer._transparency_pct logic so retroactive enforcement matches
prospective filtering.
"""
if getattr(pil_image, "is_animated", False):
return False
if pil_image.mode not in ("RGBA", "LA") and not (
pil_image.mode == "P" and "transparency" in pil_image.info
):
return False
im = pil_image
if im.mode != "RGBA":
im = im.convert("RGBA")
alpha = im.getchannel("A")
histogram = alpha.histogram()
transparent = histogram[0]
total = sum(histogram)
pct = transparent / total if total else 0.0
return pct > threshold
+147 -2
View File
@@ -12,12 +12,14 @@ re-exports from this module and then delete the wrapper.
"""
from __future__ import annotations
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from sqlalchemy import func, select
from sqlalchemy import func, select, update
from sqlalchemy.orm import Session
from ..models import Artist, ImageRecord, Tag
from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
from ..models.series_page import SeriesPage
from ..models.tag import image_tag
@@ -365,3 +367,146 @@ def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
)
session.commit()
return {"deleted": len(ids), "sample_names": sample}
# ---------------------------------------------------------------------------
# FC-Cleanup additions (2026-05-26): retroactive audit of import-filter rules.
# ---------------------------------------------------------------------------
_MIN_DIM_SAMPLE_CAP = 50
def project_min_dimension_violations(
session: Session, *, min_width: int, min_height: int,
) -> dict:
"""Return {count, sample_ids} for image_record rows with width or
height below the thresholds. Synchronous SQL — no PIL inspection
needed since width/height are stored columns."""
base = select(ImageRecord.id).where(
(ImageRecord.width < min_width) | (ImageRecord.height < min_height)
)
count = session.execute(
select(func.count()).select_from(base.subquery())
).scalar_one()
sample_ids = session.execute(
base.order_by(ImageRecord.id).limit(_MIN_DIM_SAMPLE_CAP)
).scalars().all()
return {"count": count, "sample_ids": list(sample_ids)}
def delete_min_dimension_violations(
session: Session, *, min_width: int, min_height: int, images_root: Path,
) -> int:
"""Delete every image_record where width<min_w OR height<min_h.
Routes through delete_images so file-unlink + cascading FKs
(image_tag / image_provenance / etc.) are handled uniformly."""
ids = session.execute(
select(ImageRecord.id).where(
(ImageRecord.width < min_width) | (ImageRecord.height < min_height)
)
).scalars().all()
if not ids:
return 0
result = delete_images(
session, image_ids=list(ids), images_root=images_root,
)
return result["images_deleted"]
# ---------------------------------------------------------------------------
# Audit lifecycle (transparency + single_color async scans).
# ---------------------------------------------------------------------------
class AuditAlreadyRunning(Exception):
"""Another audit_run is currently in status='running' — wait or
cancel it before starting a new one. Surfaces as HTTP 409 in the
/api/cleanup/audit POST endpoint."""
class AuditNotReady(Exception):
"""apply_audit_run called on an audit whose status is not 'ready'."""
class ConfirmTokenMismatch(Exception):
"""Operator-supplied confirm token did not match server-recomputed token."""
_VALID_RULES = ("transparency", "single_color")
def start_audit_run(
session: Session, *, rule: str, params: dict[str, Any],
) -> int:
"""Create a LibraryAuditRun row in status='running' and dispatch the
scan_library_for_rule Celery task. Returns the new audit_id.
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
has status='running'. Operator must cancel or wait."""
if rule not in _VALID_RULES:
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
existing = session.execute(
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
).scalar_one_or_none()
if existing is not None:
raise AuditAlreadyRunning(existing)
audit = LibraryAuditRun(
rule=rule,
params=params,
status="running",
scanned_count=0,
matched_count=0,
matched_ids=[],
)
session.add(audit)
session.flush()
audit_id = audit.id
# Dispatch after flush so audit_id is populated; commit happens in
# the API handler so the audit row + dispatch are visible together.
from ..tasks.library_audit import scan_library_for_rule
scan_library_for_rule.delay(audit_id)
return audit_id
def apply_audit_run(
session: Session, *, audit_id: int, confirm_token: str, images_root: Path,
) -> int:
"""Delete all images in audit_run.matched_ids after confirming token.
Marks audit status='applied'. Routes through delete_images so files
+ cascading FK rows are handled uniformly."""
audit = session.execute(
select(LibraryAuditRun).where(LibraryAuditRun.id == audit_id)
).scalar_one_or_none()
if audit is None:
raise ValueError(f"audit_run {audit_id} not found")
if audit.status != "ready":
raise AuditNotReady(audit.status)
# Token format matches modal/DestructiveConfirmModal.vue convention:
# ${action}-${kind}-${runId}. The modal hardcodes action ∈ {'restore',
# 'delete'}; "apply audit" is semantically a delete of the matched
# images, so we use 'delete-audit-<id>' (not 'apply-audit-<id>').
expected = f"delete-audit-{audit_id}"
if confirm_token != expected:
raise ConfirmTokenMismatch(expected)
ids = list(audit.matched_ids or [])
deleted = 0
if ids:
result = delete_images(session, image_ids=ids, images_root=images_root)
deleted = result["images_deleted"]
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(status="applied", finished_at=datetime.now(UTC))
)
return deleted
def cancel_audit_run(session: Session, *, audit_id: int) -> None:
"""Flip a running audit_run to 'cancelled'. The scan task checks
for status=='cancelled' between batches and exits cleanly."""
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.where(LibraryAuditRun.status == "running")
.values(status="cancelled", finished_at=datetime.now(UTC))
)
+44 -7
View File
@@ -11,6 +11,7 @@ from __future__ import annotations
import re
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, Source
@@ -97,20 +98,40 @@ class ExtensionService:
raise UnknownPlatformError(f"no platform pattern matched {url!r}")
async def _find_or_create_artist(self, raw_name: str) -> tuple[Artist, bool]:
"""Race-safe find-or-create on Artist by slug. Mirrors the
savepoint + IntegrityError recovery pattern used in
Importer._find_or_create_source/post (see
reference_scalar_one_or_none_duplicates memory). Without this,
two concurrent quick-add-source calls hitting the same artist
would both miss the existence check and the second INSERT would
500 against uq_artist_slug.
"""
slug = slugify(raw_name)
existing = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one_or_none()
if existing is not None:
return existing, False
artist = Artist(name=raw_name, slug=slug, is_subscription=True)
self.session.add(artist)
await self.session.flush()
return artist, True
sp = await self.session.begin_nested()
try:
artist = Artist(name=raw_name, slug=slug, is_subscription=True)
self.session.add(artist)
await self.session.flush()
await sp.commit()
return artist, True
except IntegrityError:
await sp.rollback()
recovered = (await self.session.execute(
select(Artist).where(Artist.slug == slug)
)).scalar_one()
return recovered, False
async def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> tuple[Source, bool]:
"""Race-safe — same pattern as _find_or_create_artist above. The
uq_source_artist_platform_url constraint catches the duplicate
insert; we roll the savepoint back and re-select."""
existing = (await self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
@@ -120,8 +141,24 @@ class ExtensionService:
)).scalar_one_or_none()
if existing is not None:
return existing, False
src = Source(artist_id=artist_id, platform=platform, url=url, enabled=True)
self.session.add(src)
await self.session.flush()
sp = await self.session.begin_nested()
try:
src = Source(
artist_id=artist_id, platform=platform,
url=url, enabled=True,
)
self.session.add(src)
await self.session.flush()
await sp.commit()
except IntegrityError:
await sp.rollback()
recovered = (await self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
)).scalar_one()
return recovered, False
await self.session.commit()
return src, True
+113 -54
View File
@@ -1,26 +1,34 @@
"""Cursor-paginated gallery queries.
Cursor format: opaque base64-encoded "<iso8601_created_at>:<image_id>".
Pagination key is (created_at DESC, id DESC) so we don't drift when new
imports arrive between page loads. Decoding rejects malformed cursors with
a ValueError; the API layer translates that to HTTP 400.
Cursor format: opaque base64-encoded "<iso8601_effective_date>:<image_id>".
Pagination key is (effective_date DESC, id DESC) where effective_date is
COALESCE(post.post_date, image_record.created_at) so the gallery surfaces
images by ORIGINAL publish date when known, falling back to FC's scan
date. Important for migrated content: ~57k IR images scanned in a single
week would otherwise all share the same created_at and pile up in one
month bucket. The effective_date spreads them across the years they
were originally published.
Decoding rejects malformed cursors with a ValueError; the API layer
translates that to HTTP 400.
"""
import base64
from dataclasses import dataclass
from datetime import datetime
from sqlalchemy import and_, exists, func, or_, select
from sqlalchemy import Select, and_, exists, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ImageProvenance, ImageRecord, Source, Tag
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models.tag import image_tag
CURSOR_SEPARATOR = "|"
def encode_cursor(created_at: datetime, image_id: int) -> str:
raw = f"{created_at.isoformat()}{CURSOR_SEPARATOR}{image_id}"
def encode_cursor(effective_date: datetime, image_id: int) -> str:
raw = f"{effective_date.isoformat()}{CURSOR_SEPARATOR}{image_id}"
return base64.urlsafe_b64encode(raw.encode()).decode()
@@ -33,6 +41,26 @@ def decode_cursor(cursor: str) -> tuple[datetime, int]:
raise ValueError(f"invalid cursor: {cursor!r}") from exc
def _effective_date_col():
"""SQL expression: COALESCE(post.post_date, image_record.created_at).
Used as the canonical sort/group/filter key across the gallery so
images backfilled with primary_post_id (e.g. via tag_apply phase 4)
surface at their original publish date, not their FC import date.
Images without a Post (or with Post.post_date NULL) fall back to
image_record.created_at and still order coherently against
post-attached ones.
"""
return func.coalesce(Post.post_date, ImageRecord.created_at)
def _outer_join_primary_post(stmt: Select) -> Select:
"""LEFT JOIN Post on ImageRecord.primary_post_id so the COALESCE
above sees Post.post_date when available. Images without a post
survive the join as NULL on the Post side; COALESCE handles it."""
return stmt.outerjoin(Post, Post.id == ImageRecord.primary_post_id)
@dataclass(frozen=True)
class GalleryImage:
id: int
@@ -41,7 +69,9 @@ class GalleryImage:
mime: str
width: int | None
height: int | None
created_at: datetime
created_at: datetime # FC's row-insert time
effective_date: datetime # COALESCE(post.post_date, created_at)
posted_at: datetime | None # post.post_date if known, else None
thumbnail_url: str
artist: dict | None = None
@@ -78,7 +108,7 @@ def _require_single_filter(tag_id, post_id, artist_id) -> None:
def _provenance_clause(post_id, artist_id):
"""Correlated EXISTS clause (NOT a join) so an image with multiple
matching provenance rows is returned exactly once and the
(created_at DESC, id DESC) cursor ordering is unaffected."""
(effective_date DESC, id DESC) cursor ordering is unaffected."""
if post_id is not None:
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
@@ -125,7 +155,9 @@ class GalleryService:
raise ValueError("limit must be between 1 and 200")
_require_single_filter(tag_id, post_id, artist_id)
stmt = select(ImageRecord)
eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
stmt = _outer_join_primary_post(stmt)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
image_tag.c.tag_id == tag_id
@@ -138,34 +170,38 @@ class GalleryService:
cur_ts, cur_id = decode_cursor(cursor)
stmt = stmt.where(
or_(
ImageRecord.created_at < cur_ts,
and_(ImageRecord.created_at == cur_ts, ImageRecord.id < cur_id),
eff < cur_ts,
and_(eff == cur_ts, ImageRecord.id < cur_id),
)
)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).scalars().all()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(limit + 1)
rows = (await self.session.execute(stmt)).all()
next_cursor = None
if len(rows) > limit:
last = rows[limit - 1]
next_cursor = encode_cursor(last.created_at, last.id)
last_record, _last_posted_at, last_eff = rows[limit - 1]
next_cursor = encode_cursor(last_eff, last_record.id)
rows = rows[:limit]
artists = await _artists_for(self.session, [r.id for r in rows])
artists = await _artists_for(
self.session, [r[0].id for r in rows]
)
images = [
GalleryImage(
id=r.id,
path=r.path,
sha256=r.sha256,
mime=r.mime,
width=r.width,
height=r.height,
created_at=r.created_at,
thumbnail_url=thumbnail_url(r.sha256, r.mime),
artist=artists.get(r.id),
id=record.id,
path=record.path,
sha256=record.sha256,
mime=record.mime,
width=record.width,
height=record.height,
created_at=record.created_at,
effective_date=eff_date,
posted_at=posted_at,
thumbnail_url=thumbnail_url(record.sha256, record.mime),
artist=artists.get(record.id),
)
for r in rows
for record, posted_at, eff_date in rows
]
return GalleryPage(
images=images,
@@ -179,11 +215,13 @@ class GalleryService:
post_id: int | None = None,
artist_id: int | None = None,
) -> list[TimelineBucket]:
year_col = func.date_part("year", ImageRecord.created_at).label("yr")
month_col = func.date_part("month", ImageRecord.created_at).label("mo")
eff = _effective_date_col()
year_col = func.date_part("year", eff).label("yr")
month_col = func.date_part("month", eff).label("mo")
stmt = select(
year_col, month_col, func.count(ImageRecord.id).label("cnt")
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -201,14 +239,17 @@ class GalleryService:
post_id: int | None = None, artist_id: int | None = None,
) -> str | None:
"""Returns a cursor that, when passed to scroll(), positions at the
first image of the given year-month. None if the bucket is empty.
first image of the given year-month (by effective_date, not
created_at). None if the bucket is empty.
"""
from sqlalchemy import extract
stmt = select(ImageRecord).where(
extract("year", ImageRecord.created_at) == year,
extract("month", ImageRecord.created_at) == month,
eff = _effective_date_col()
stmt = select(ImageRecord, eff.label("eff")).where(
extract("year", eff) == year,
extract("month", eff) == month,
)
stmt = _outer_join_primary_post(stmt)
_require_single_filter(tag_id, post_id, artist_id)
if tag_id is not None:
stmt = stmt.join(image_tag, image_tag.c.image_record_id == ImageRecord.id).where(
@@ -217,13 +258,14 @@ class GalleryService:
prov = _provenance_clause(post_id, artist_id)
if prov is not None:
stmt = stmt.where(prov)
stmt = stmt.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).scalar_one_or_none()
stmt = stmt.order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
first = (await self.session.execute(stmt)).first()
if first is None:
return None
record, eff_date = first
# Cursor is exclusive; we encode a cursor with id+1 so the row itself
# is the first result in the next scroll().
return encode_cursor(first.created_at, first.id + 1)
return encode_cursor(eff_date, record.id + 1)
async def get_image_with_tags(self, image_id: int) -> dict | None:
record = await self.session.get(ImageRecord, image_id)
@@ -236,6 +278,13 @@ class GalleryService:
.order_by(Tag.kind.asc(), Tag.name.asc())
)
tags = (await self.session.execute(tag_stmt)).scalars().all()
# Fetch the canonical post.post_date for this image (if any) so
# the modal can show "Posted on <date>" alongside import date.
posted_at = None
if record.primary_post_id is not None:
posted_at = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
neighbors = await self._neighbors(record)
# Direct artist FK — used by the modal's ProvenancePanel as a
# fallback when ImageProvenance is empty (i.e., filesystem-
@@ -256,6 +305,7 @@ class GalleryService:
"size_bytes": record.size_bytes,
"integrity_status": record.integrity_status,
"created_at": record.created_at.isoformat(),
"posted_at": posted_at.isoformat() if posted_at else None,
"thumbnail_url": thumbnail_url(record.sha256, record.mime),
"image_url": f"/images/{record.path.split('/images/', 1)[-1]}",
"artist": (
@@ -275,34 +325,41 @@ class GalleryService:
}
async def _neighbors(self, record: ImageRecord) -> dict:
prev_stmt = (
select(ImageRecord.id)
.where(
# Compute the boundary image's effective_date in Python (one query
# below + the SELECT we already have on `record`) and use it for
# the neighbor comparison. Cheaper than re-deriving in SQL via
# correlated subquery.
boundary_eff = record.created_at
if record.primary_post_id is not None:
post_date = (await self.session.execute(
select(Post.post_date).where(Post.id == record.primary_post_id)
)).scalar_one_or_none()
if post_date is not None:
boundary_eff = post_date
eff = _effective_date_col()
prev_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at > record.created_at,
eff > boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id > record.id,
),
)
)
.order_by(ImageRecord.created_at.asc(), ImageRecord.id.asc())
.limit(1)
)
next_stmt = (
select(ImageRecord.id)
.where(
).order_by(eff.asc(), ImageRecord.id.asc()).limit(1)
next_stmt = _outer_join_primary_post(
select(ImageRecord.id).where(
or_(
ImageRecord.created_at < record.created_at,
eff < boundary_eff,
and_(
ImageRecord.created_at == record.created_at,
eff == boundary_eff,
ImageRecord.id < record.id,
),
)
)
.order_by(ImageRecord.created_at.desc(), ImageRecord.id.desc())
.limit(1)
)
).order_by(eff.desc(), ImageRecord.id.desc()).limit(1)
prev_id = (await self.session.execute(prev_stmt)).scalar_one_or_none()
next_id = (await self.session.execute(next_stmt)).scalar_one_or_none()
return {"prev_id": prev_id, "next_id": next_id}
@@ -311,9 +368,11 @@ class GalleryService:
def _group_by_year_month(
images: list[GalleryImage],
) -> list[tuple[int, int, list[int]]]:
"""Group by effective_date's year/month so migrated content surfaces
in the publish-date buckets, not the FC-scan-date bucket."""
groups: list[tuple[int, int, list[int]]] = []
for img in images:
y, m = img.created_at.year, img.created_at.month
y, m = img.effective_date.year, img.effective_date.month
if groups and groups[-1][0] == y and groups[-1][1] == m:
groups[-1][2].append(img.id)
else:
+333 -87
View File
@@ -18,6 +18,7 @@ from pathlib import Path
from PIL import Image
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from ..models import (
@@ -51,7 +52,14 @@ class SkipReason(StrEnum):
@dataclass(frozen=True)
class ImportResult:
status: str # 'imported'|'skipped'|'failed'|'superseded'|'attached'
# 'imported' — new ImageRecord row created
# 'superseded' — existing ImageRecord row got the new file (larger) + sidecar
# 'attached' — non-media saved as PostAttachment
# 'refreshed' — deep scan re-applied sidecar / filled NULL phash / NULL
# artist on an already-imported row (no new ImageRecord)
# 'skipped' — no work done (true duplicate, too small, etc.)
# 'failed' — pipeline error
status: str
image_id: int | None = None
skip_reason: SkipReason | None = None
error: str | None = None
@@ -71,6 +79,31 @@ def is_video(path: Path) -> bool:
return path.suffix.lower() in VIDEO_EXTS
def _safe_ext(path: Path) -> str:
"""Conservatively extract a file extension for PostAttachment.ext
(varchar(32)).
gallery-dl produces some filenames with URL-encoded query-string
artifacts embedded into the basename (e.g.
`79507046_media_..._https___www.patreon.com_media-u_Z0FBQUFBQm5q...`).
`Path.suffix` finds the LAST dot and returns everything after, which
in those cases yields a 50+ char "extension" of mostly base64-ish
junk. That blows the column. Operator-flagged 2026-05-25.
Real extensions are short and alphanumeric. We accept anything ≤ 16
chars where every post-dot character is alphanumeric; anything else
means the input wasn't a real extension and we return the empty
string. ext is nullable-ish (empty string still satisfies NOT NULL)
and consumers should treat "" as "no known extension".
"""
suffix = path.suffix.lower()
if not suffix or len(suffix) > 16:
return ""
if not all(c.isalnum() for c in suffix[1:]):
return ""
return suffix
def _mime_for(path: Path) -> str:
suffix = path.suffix.lower()
image_mimes = {
@@ -126,6 +159,179 @@ class Importer:
self.settings = settings
self.deep = deep
self.attachments = AttachmentStore(images_root)
# phash near-dup candidate cache. Archive imports call _import_media
# per-member; without this cache the per-member SELECT *FROM
# image_record WHERE phash IS NOT NULL fetch repeats N times and a
# large library × many-member archive blew past soft_time_limit
# (300s) — operator-flagged 2026-05-25. Loaded lazily on first
# need, appended to on every imported/superseded outcome, never
# invalidated mid-Importer (Importer instances are per-task /
# per-archive-import so cross-instance staleness is harmless).
self._phash_candidates: list[tuple] | None = None
def _phash_candidates_cache(self) -> list[tuple]:
"""Cached `(phash, width, height, id)` rows from image_record.
Loaded on first call, appended-to on subsequent imported/
superseded outcomes. Soft-timeout pattern: an archive with N
members + a library of M existing rows used to do N × M-row
fetches (operator-flagged 2026-05-25); now it's exactly one.
The per-task lifecycle of Importer (instantiated fresh by
import_media_file) bounds the cache's staleness window: cross-
process changes (other workers importing concurrently) won't
be reflected, but that's the same race the un-cached version
had — `find_similar` is best-effort anyway."""
if self._phash_candidates is None:
rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
self._phash_candidates = [
(r.phash, r.width or 0, r.height or 0, r.id) for r in rows
]
return self._phash_candidates
def _phash_cache_append(self, phash, width, height, image_id) -> None:
"""Append a freshly-imported row to the cache so subsequent
members of the same archive can match against it."""
if self._phash_candidates is not None and phash is not None:
self._phash_candidates.append(
(phash, width or 0, height or 0, image_id)
)
def _find_or_create_source(
self, *, artist_id: int, platform: str, url: str,
) -> Source:
"""Race-safe find-or-create on `source` keyed by
(artist_id, platform, url) — the same key as the
`uq_source_artist_platform_url` constraint.
Two concurrent workers processing different files in the same
post can both find no existing Source row then both INSERT,
which trips the unique constraint and poisons the session with
`psycopg.errors.UniqueViolation`. Operator-flagged 2026-05-26.
Pattern: select; if absent, open a savepoint and INSERT.
On IntegrityError, roll the savepoint back (NOT the outer
transaction, which would lose the surrounding scan's progress)
and re-select — the concurrent op just created the row we
wanted, so the second select will find it.
"""
existing = self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Source(artist_id=artist_id, platform=platform, url=url)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source).where(
Source.artist_id == artist_id,
Source.platform == platform,
Source.url == url,
)
).scalar_one()
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Filesystem-import sidecar Source resolver.
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). The filesystem importer
used to call _find_or_create_source(url=sd.post_url), which created
one Source row per post URL — 100s of junk Sources per artist, all
with enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26.
New behaviour: if any Source row exists for (artist_id, platform),
reuse it regardless of its URL — the artist's real subscription Source
(created by the downloader / extension / UI) is the canonical
attachment point for filesystem-imported posts. If none exists, create
ONE synthetic anchor with url='sidecar:<platform>:<artist_slug>' and
enabled=False (so the subscription checker doesn't poll it).
"""
existing = self.session.execute(
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one_or_none()
if existing is not None:
return existing
synthetic_url = f"sidecar:{platform}:{artist_slug}"
sp = self.session.begin_nested()
try:
row = Source(
artist_id=artist_id,
platform=platform,
url=synthetic_url,
enabled=False,
)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
)
.order_by(Source.id.asc())
.limit(1)
).scalar_one()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
existing = self.session.execute(
select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one_or_none()
if existing is not None:
return existing
sp = self.session.begin_nested()
try:
row = Post(source_id=source_id, external_post_id=external_post_id)
self.session.add(row)
self.session.flush()
sp.commit()
return row
except IntegrityError:
sp.rollback()
return self.session.execute(
select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
).scalar_one()
def import_one(self, source: Path) -> ImportResult:
"""Dispatch by kind. Media → normal pipeline. Archive → extract
@@ -165,30 +371,13 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
url = sd.post_url or f"sidecar:{platform}"
src = self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if src is None:
src = Source(artist_id=artist.id, platform=platform, url=url)
self.session.add(src)
self.session.flush()
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self.session.execute(
select(Post).where(
Post.source_id == src.id,
Post.external_post_id == epid,
)
).scalar_one_or_none()
if post is None:
post = Post(source_id=src.id, external_post_id=epid)
self.session.add(post)
self.session.flush()
return post
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
)
def _capture_attachment(
self, source: Path, *, post: Post | None = None,
@@ -209,7 +398,7 @@ class Importer:
sha256=sha,
path=stored,
original_filename=source.name,
ext=source.suffix.lower(),
ext=_safe_ext(source),
mime=_mime_for(source),
size_bytes=source.stat().st_size,
))
@@ -280,7 +469,18 @@ class Importer:
)
if self.settings.skip_transparent and has_alpha:
pct = self._transparency_pct(source)
try:
pct = self._transparency_pct(source)
except OSError as exc:
# PIL.verify() at line 263 only validates header structure;
# truncated/corrupt pixel data only surfaces when load()
# actually decodes (here via getchannel('A')). Convert to
# invalid_image skip so the Celery autoretry loop doesn't
# bounce the same broken file forever.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during transparency check: {exc}",
)
if pct >= self.settings.transparency_threshold:
return ImportResult(
status="skipped", skip_reason=SkipReason.too_transparent,
@@ -302,21 +502,18 @@ class Importer:
# Perceptual near-dup (images only; videos keep phash NULL).
phash = None
if not is_video(source):
with Image.open(source) as im:
phash = compute_phash(im)
try:
with Image.open(source) as im:
phash = compute_phash(im)
except OSError as exc:
# Same rationale as the transparency-check guard above:
# broken-pixel-data files pass verify() but blow up here.
return ImportResult(
status="skipped", skip_reason=SkipReason.invalid_image,
error=f"PIL load failed during phash compute: {exc}",
)
if phash is not None:
cand_rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
candidates = [
(c.phash, c.width or 0, c.height or 0, c.id)
for c in cand_rows
]
candidates = self._phash_candidates_cache()
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
@@ -350,6 +547,7 @@ class Importer:
)
self.session.add(record)
self.session.flush()
self._phash_cache_append(phash, width, height, record.id)
# Folder→artist (anchored to attribution_path).
artist = None
@@ -373,13 +571,27 @@ class Importer:
) -> ImportResult:
"""Deep scan: backfill phash/provenance/artist on an
already-imported record. METADATA ONLY — never re-runs the pHash
near-dup / supersede path. NULL-only, idempotent."""
near-dup / supersede path. NULL-only on phash/artist, additive on
sidecar Post/Source/ImageProvenance (via _apply_sidecar).
Idempotent: a second deep-scan over the same file finds nothing
to refresh and is a no-op.
Returns status="refreshed" so the UI can surface the work done
instead of the prior misleading "skipped/duplicate_hash" reading.
Operator-flagged 2026-05-25 — IR has had this; FC inherited it
as a no-op skip during the original port and the UI showed deep
scan as "completed with no changes" even when sidecar metadata
actually got re-applied to N existing rows.
"""
if existing.phash is None and not is_video(source):
try:
with Image.open(source) as im:
ph = compute_phash(im)
if ph is not None:
existing.phash = ph
# Promoted from NULL to non-NULL → cache is now stale
# (this row would newly qualify for the candidates set).
self._phash_candidates = None
except Exception as exc:
log.warning("deep rephash failed for %s: %s", source, exc)
@@ -392,10 +604,7 @@ class Importer:
self._apply_sidecar(existing, attribution_path, artist)
self.session.commit()
return ImportResult(
status="skipped", skip_reason=SkipReason.duplicate_hash,
image_id=existing.id, error="deep: re-derived",
)
return ImportResult(status="refreshed", image_id=existing.id)
def attach_in_place(
self,
@@ -479,18 +688,7 @@ class Importer:
except Exception:
phash = None
if phash is not None:
cand_rows = self.session.execute(
select(
ImageRecord.phash,
ImageRecord.width,
ImageRecord.height,
ImageRecord.id,
).where(ImageRecord.phash.is_not(None))
).all()
candidates = [
(c.phash, c.width or 0, c.height or 0, c.id)
for c in cand_rows
]
candidates = self._phash_candidates_cache()
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
@@ -525,6 +723,7 @@ class Importer:
record.artist_id = artist.id
self.session.add(record)
self.session.flush()
self._phash_cache_append(phash, width, height, record.id)
# Sidecar provenance (best-effort). When `source` is passed, link
# the post to that subscription Source instead of creating a new
@@ -619,30 +818,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
url = sd.post_url or f"sidecar:{platform}"
src = self.session.execute(
select(Source).where(
Source.artist_id == artist.id,
Source.platform == platform,
Source.url == url,
)
).scalar_one_or_none()
if src is None:
src = Source(artist_id=artist.id, platform=platform, url=url)
self.session.add(src)
self.session.flush()
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self.session.execute(
select(Post).where(
Post.source_id == src.id,
Post.external_post_id == epid,
)
).scalar_one_or_none()
if post is None:
post = Post(source_id=src.id, external_post_id=epid)
self.session.add(post)
self.session.flush()
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
)
if sd.post_url is not None:
post.post_url = sd.post_url
if sd.post_title is not None:
@@ -655,6 +839,15 @@ class Importer:
post.attachment_count = sd.attachment_count
post.raw_metadata = sd.raw
# Race-safe (image_record_id, post_id) upsert — mirrors the
# _find_or_create_source/post savepoint pattern. The plain
# SELECT-then-INSERT pattern lost a race when two workers ran
# _apply_sidecar on the same (image, post) pair (e.g. the 5-min
# recovery sweep re-enqueued a still-running long import), planting
# duplicates that then broke .scalar_one_or_none() on every later
# deep-scan rederive (MultipleResultsFound). Alembic 0021 adds the
# uq_image_provenance_image_post UNIQUE so this savepoint actually
# trips on collision.
exists = self.session.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == record.id,
@@ -662,14 +855,20 @@ class Importer:
)
).scalar_one_or_none()
if exists is None:
self.session.add(
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
captured_metadata=sd.raw,
sp = self.session.begin_nested()
try:
self.session.add(
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
captured_metadata=sd.raw,
)
)
)
self.session.flush()
sp.commit()
except IntegrityError:
sp.rollback()
if record.primary_post_id is None:
record.primary_post_id = post.id
self.session.flush()
@@ -703,6 +902,15 @@ class Importer:
row id (so tags/series/curation stay attached). ML is cleared so
the import task re-derives it on the new pixels.
After the file swap, the new file's adjacent gallery-dl sidecar
(if any) is applied via _apply_sidecar — operator-flagged
2026-05-25: scanning a GS download dir with smaller IR-migrated
images on the receiving end used to swap files but lose the GS
sidecar's post metadata entirely. _apply_sidecar is additive
(find-or-create Post / Source / ImageProvenance, NULL-only
primary_post_id update) so any pre-existing Post linkage
survives untouched.
If `new_path` is provided, `source` is assumed to ALREADY be at
that path (FC-3c attach_in_place case) — skip the copy step.
Otherwise the file is copied via _copy_to_library."""
@@ -731,6 +939,26 @@ class Importer:
# created_at intentionally preserved; updated_at auto-bumps.
self.session.flush()
self.session.commit()
# The phash candidate cache (used to avoid N+1 selects during
# archive imports) is now stale for `existing.id` — the row's
# phash/dimensions changed. Invalidate; the next call re-fetches.
self._phash_candidates = None
# Sidecar enrichment from the new (larger) file's location.
# _apply_sidecar resolves artist from the sidecar itself if the
# existing row has none, and is internally guarded against
# missing-or-malformed sidecars (silent return).
try:
self._apply_sidecar(existing, source, None)
except Exception as exc:
# Don't unwind the supersede DB swap if sidecar parsing
# blows up unexpectedly — the file replacement is the
# critical operation, sidecar is enrichment.
log.warning(
"sidecar enrichment failed during supersede of "
"image_record.id=%s from %s: %s",
existing.id, source, exc,
)
for stale in (old_path, old_thumb):
if not stale or stale == str(dest):
@@ -746,8 +974,26 @@ class Importer:
pass
def _transparency_pct(self, source: Path) -> float:
"""Fraction of fully-transparent pixels in the image. 0.0 if no alpha."""
"""Fraction of fully-transparent pixels in the image. 0.0 if no alpha.
For animated formats (multi-frame WebP / GIF / APNG), short-circuit
to 0.0 instead of decoding every frame. PIL's `getchannel("A")`
forces a full decode of all frames in an animated image, which for
a large animated WebP takes 5+ minutes and blows past the Celery
soft+hard time limits (300s/360s → SIGKILL). Operator-flagged
2026-05-26. Transparency analysis on a multi-frame image isn't
meaningful for art-curation purposes anyway — different frames
have different alpha — so the existing too_transparent skip rule
is bypassed entirely for animated content.
"""
with Image.open(source) as im:
if getattr(im, "is_animated", False):
log.info(
"skipping transparency check for animated image %s "
"(n_frames=%d) — avoids multi-frame decode timeout",
source, getattr(im, "n_frames", 0),
)
return 0.0
if im.mode not in ("RGBA", "LA") and not (
im.mode == "P" and "transparency" in im.info
):
+35 -4
View File
@@ -37,13 +37,25 @@ from ...utils.slug import slugify
from .ir_ingest import manifest_path
# Per-platform artist-profile URL — used as Source.url when restoring
# IR PostMetadata into FC. Keep this table in sync with
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS and
# extension/lib/platforms.js.
# IR PostMetadata into FC. Must cover every platform that
# backend/app/services/extension_service.py:_PLATFORM_PATTERNS
# recognizes; an entry missing here silently drops ALL PostMetadata for
# that platform during phase 4 (operator hit this 2026-05-25:
# DeviantArt + Pixiv posts in the IR migration produced empty
# ImageProvenance because they fell through this table).
#
# Pixiv caveat: the real profile URL takes a numeric user_id
# (https://www.pixiv.net/users/12345), but IR's PostMetadata.artist
# stores the display name not the id. We use the slugified name here
# so we preserve the artist→post→image linkage; the resulting Source.url
# won't resolve in a browser and the operator may want to manually fix
# it via Settings → Subscriptions once the migration lands.
_PLATFORM_PROFILE_URL = {
"patreon": "https://www.patreon.com/{slug}",
"subscribestar": "https://www.subscribestar.com/{slug}",
"hentaifoundry": "https://www.hentai-foundry.com/user/{slug}",
"deviantart": "https://www.deviantart.com/{slug}",
"pixiv": "https://www.pixiv.net/users/{slug}",
}
@@ -110,7 +122,14 @@ async def _ensure_provenance(
db: AsyncSession, *,
image_id: int, post_id: int, source_id: int, dry_run: bool,
) -> bool:
"""Returns True if a new ImageProvenance row was inserted."""
"""Returns True if a new ImageProvenance row was inserted.
Also sets ImageRecord.primary_post_id to this post if the image
doesn't already have one — preserves any primary_post_id already
assigned at download time by the importer (don't clobber). This is
the linkage gallery_service.py uses to surface Post.post_date as
the image's effective date for sort/group/jump/neighbor nav.
"""
existing = (await db.execute(
select(ImageProvenance.id).where(
ImageProvenance.image_record_id == image_id,
@@ -118,6 +137,18 @@ async def _ensure_provenance(
ImageProvenance.source_id == source_id,
)
)).scalar_one_or_none()
# Whether-or-not the provenance row already exists, ensure the
# image's primary_post_id is set so the gallery date-coalesce works.
# Idempotent: only writes when currently NULL.
if not dry_run:
await db.execute(
ImageRecord.__table__.update()
.where(ImageRecord.id == image_id)
.where(ImageRecord.primary_post_id.is_(None))
.values(primary_post_id=post_id)
)
if existing is not None:
return False
if dry_run:
+13 -2
View File
@@ -34,10 +34,21 @@ class Embedder:
if self._model is not None:
return
import torch
from transformers import AutoModel, AutoProcessor
from transformers import AutoModel, SiglipImageProcessor
self._torch = torch
self._processor = AutoProcessor.from_pretrained(str(self._model_dir))
# FC's embedder only does IMAGE inference — never text. AutoProcessor
# loads the full processor including SiglipTokenizer, which requires
# the sentencepiece library at import time even if we never call it.
# SiglipImageProcessor loads ONLY preprocessor_config.json (image
# side) and skips the tokenizer config entirely. Operator hit the
# ImportError 2026-05-25 once the ml-worker started actually running
# tag_and_embed; switching to the image-only loader avoids the
# tokenizer dep without adding ~30 MB of unused C++ build to the
# lean ml-worker image.
self._processor = SiglipImageProcessor.from_pretrained(
str(self._model_dir)
)
self._model = AutoModel.from_pretrained(str(self._model_dir))
self._model.eval()
+82 -44
View File
@@ -4,12 +4,14 @@ CPU-only, single-image at a time. Loaded lazily inside the ml-worker
process; NOT thread-safe — the ml queue worker must run --concurrency=1
(set by the FC-1 entrypoint).
Camie's selected_tags.csv columns: tag_id,name,category,count
where category is a string: general|character|copyright|artist|meta|rating|year
(unlike WD14's integer Danbooru category ids).
v2 layout reference: HuggingFace Camais03/camie-tagger-v2 root has
camie-tagger-v2.onnx (789 MB) + camie-tagger-v2-metadata.json (7.77 MB)
+ config.json. Tags ship as nested JSON, not CSV. Preprocessing and
output handling follow the published onnx_inference.py reference:
ImageNet normalize, NCHW layout, sigmoid on refined logits (output[1]).
"""
import csv
import json
import os
from dataclasses import dataclass
from pathlib import Path
@@ -28,6 +30,8 @@ ImageFile.LOAD_TRUNCATED_IMAGES = True
MODEL_NAME = os.environ.get("CAMIE_MODEL_NAME", "camie-tagger-v2")
_MODEL_DIR = Path(os.environ.get("ML_MODEL_DIR", "/models")) / "camie"
_MODEL_FILE = f"{MODEL_NAME}.onnx"
_METADATA_FILE = f"{MODEL_NAME}-metadata.json"
# Below this confidence, predictions aren't stored (keeps the JSON compact).
STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
@@ -39,6 +43,12 @@ STORE_FLOOR = float(os.environ.get("TAGGER_STORE_FLOOR", "0.05"))
# stored at STORE_FLOOR but artist never surfaces.
SURFACED_CATEGORIES = {"character", "copyright", "general"}
# ImageNet preprocessing constants (per Camie v2 onnx_inference.py).
_IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
_IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
# Square-pad color ≈ ImageNet mean × 255 (matches reference inference).
_PAD_COLOR = (124, 116, 104)
@dataclass(frozen=True)
class TagPrediction:
@@ -51,34 +61,48 @@ class Tagger:
def __init__(self, model_dir: Path | None = None):
self._model_dir = model_dir or _MODEL_DIR
self._session = None # onnxruntime.InferenceSession once load()ed
self._tag_meta: list[dict] | None = None
self._tag_names: list[str] | None = None
self._tag_categories: list[str] | None = None
self._input_name: str | None = None
self._output_name: str | None = None
self._input_size: int = 448
self._input_size: int = 512
def load(self) -> None:
if self._session is not None:
return
model_path = self._model_dir / "model.onnx"
tags_path = self._model_dir / "selected_tags.csv"
model_path = self._model_dir / _MODEL_FILE
meta_path = self._model_dir / _METADATA_FILE
if not model_path.is_file():
raise RuntimeError(
f"Camie model.onnx missing at {model_path}. "
f"Camie {_MODEL_FILE} missing at {model_path}. "
f"Populate /models via the ml-worker downloader."
)
if not tags_path.is_file():
if not meta_path.is_file():
raise RuntimeError(
f"Camie selected_tags.csv missing at {tags_path}. "
f"Camie {_METADATA_FILE} missing at {meta_path}. "
f"Populate /models via the ml-worker downloader."
)
tag_meta: list[dict] = []
with open(tags_path, newline="") as f:
reader = csv.DictReader(f)
for row in reader:
tag_meta.append(
{"name": row["name"], "category": row["category"]}
)
with open(meta_path) as f:
metadata = json.load(f)
# Per Camie v2 onnx_inference.py: idx_to_tag is keyed by str(idx);
# tag_to_category maps tag_name -> category. Project to two parallel
# lists indexed by output position for O(1) lookup in the hot path.
ds = metadata["dataset_info"]
idx_to_tag = ds["tag_mapping"]["idx_to_tag"]
tag_to_category = ds["tag_mapping"]["tag_to_category"]
total = ds["total_tags"]
names: list[str] = []
cats: list[str] = []
for i in range(total):
name = idx_to_tag.get(str(i), f"unknown-{i}")
names.append(name)
cats.append(tag_to_category.get(name, "general"))
# Input size from metadata; fall back to 512 (the v2 default).
self._input_size = int(
metadata.get("model_info", {}).get("img_size", 512)
)
# Lazy import — kept after the file-existence checks so the
# missing-model RuntimeError still fires first in environments
@@ -89,51 +113,65 @@ class Tagger:
str(model_path), providers=["CPUExecutionProvider"]
)
self._input_name = session.get_inputs()[0].name
self._output_name = session.get_outputs()[0].name
input_shape = session.get_inputs()[0].shape
for dim in input_shape:
if isinstance(dim, int) and dim > 1:
self._input_size = dim
break
# Assign sentinels last so a partial load isn't observable.
self._tag_meta = tag_meta
self._tag_names = names
self._tag_categories = cats
self._session = session
def _preprocess(self, image_path: Path) -> np.ndarray:
img = Image.open(image_path)
# Camie handles RGBA natively but we still composite onto white so
# transparency doesn't bias the model (same as IR's WD14 path).
if img.mode != "RGBA":
img = img.convert("RGBA")
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
# Composite RGBA onto neutral so transparency doesn't bias the model.
if img.mode == "RGBA":
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
bg.paste(img, mask=img.split()[3])
img = bg.convert("RGB")
elif img.mode != "RGB":
img = img.convert("RGB")
# Pad to square with ImageNet-mean color, then bicubic resize.
w, h = img.size
side = max(w, h)
square = Image.new("RGB", (side, side), (255, 255, 255))
square = Image.new("RGB", (side, side), _PAD_COLOR)
square.paste(img, ((side - w) // 2, (side - h) // 2))
square = square.resize(
(self._input_size, self._input_size), Image.BICUBIC
)
arr = np.array(square, dtype=np.float32)
return arr[np.newaxis, :, :, :] # NHWC
arr = np.array(square, dtype=np.float32) / 255.0 # HWC, [0,1]
arr = (arr - _IMAGENET_MEAN) / _IMAGENET_STD # ImageNet normalize
arr = arr.transpose(2, 0, 1) # HWC -> CHW
return arr[np.newaxis, :, :, :] # NCHW
def infer(self, image_path: Path) -> dict[str, TagPrediction]:
"""Run Camie on one image. Returns {name: TagPrediction}, only
entries with confidence >= STORE_FLOOR (across all categories —
the suggestion service does category filtering later)."""
"""Run Camie v2 on one image. Returns {name: TagPrediction} with
confidence >= STORE_FLOOR (across all categories — the suggestion
service does category filtering later).
v2 emits multiple outputs; we use the refined predictions
(output[1] per onnx_inference.py). Sigmoid is applied to raw
logits to produce [0,1] confidence scores.
"""
self.load()
x = self._preprocess(image_path)
out = self._session.run([self._output_name], {self._input_name: x})[0][0]
outputs = self._session.run(None, {self._input_name: x})
# Refined predictions if present (v2 emits initial + refined),
# fall back to initial for single-output forks.
logits = outputs[1] if len(outputs) > 1 else outputs[0]
# Squeeze batch dim, apply sigmoid.
probs = 1.0 / (1.0 + np.exp(-logits[0]))
results: dict[str, TagPrediction] = {}
for idx, score in enumerate(out):
names = self._tag_names
cats = self._tag_categories
for idx, score in enumerate(probs):
conf = float(score)
if conf < STORE_FLOOR:
continue
meta = self._tag_meta[idx]
results[meta["name"]] = TagPrediction(
name=meta["name"], category=meta["category"], confidence=conf
if idx >= len(names):
# Output longer than metadata declared — shouldn't happen but
# don't crash the import pipeline if v2 metadata desynchronizes.
continue
results[names[idx]] = TagPrediction(
name=names[idx], category=cats[idx], confidence=conf
)
return results
+22 -12
View File
@@ -109,7 +109,10 @@ class PostFeedService:
if row is None:
return None
post, artist, source = row
thumbs_map = await self._thumbnails_for([post.id])
# Detail endpoint returns the FULL image list for PostModal's
# masonry grid — feed query still caps at THUMBNAIL_LIMIT via
# the default arg.
thumbs_map = await self._thumbnails_for([post.id], limit=None)
atts_map = await self._attachments_for([post.id])
item = self._to_dict(post, artist, source, thumbs_map, atts_map)
item["description_full"] = html_to_plain(post.description)
@@ -117,15 +120,21 @@ class PostFeedService:
# --- composition helpers ---------------------------------------------
async def _thumbnails_for(self, post_ids: list[int]) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to 6], "more": int}.
async def _thumbnails_for(
self, post_ids: list[int], *, limit: int | None = THUMBNAIL_LIMIT,
) -> dict[int, dict]:
"""post_id -> {"thumbs": [...up to limit], "more": int}.
Selects THUMBNAIL_LIMIT+1 images per post via window function so we
can detect overflow in a single query.
Selects up to `limit` images per post via window function so we
can detect overflow in a single query. Pass `limit=None` to
return ALL thumbnails per post (used by `get_post` for PostModal's
masonry grid; the feed pass keeps the default cap so payloads
stay small).
"""
if not post_ids:
return {}
# Rank images within each post and fetch only the top THUMBNAIL_LIMIT+1.
# Rank images within each post; cap at `limit` rows per post when
# limit is set, return all when limit is None.
ranked = (
select(
ImageRecord.id,
@@ -143,12 +152,13 @@ class PostFeedService:
.where(ImageRecord.primary_post_id.in_(post_ids))
.subquery()
)
rows = (await self.session.execute(
select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.total,
).where(ranked.c.rn <= THUMBNAIL_LIMIT)
)).all()
stmt = select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.total,
)
if limit is not None:
stmt = stmt.where(ranked.c.rn <= limit)
rows = (await self.session.execute(stmt)).all()
out: dict[int, dict] = {pid: {"thumbs": [], "more": 0} for pid in post_ids}
for img_id, pid, sha, mime, total in rows:
+13 -2
View File
@@ -29,10 +29,12 @@ IMAGES_ROOT = Path("/images")
def _map_result_to_status(result):
"""(ImportTask.status, should_requeue_ml_and_thumb) for an ImportResult.
'superseded' = the kept row's file/ML changed → complete + re-derive.
'attached' = a non-art file preserved → complete, no ML/thumb."""
'attached' = a non-art file preserved → complete, no ML/thumb.
'refreshed' = deep scan refreshed sidecar/phash on an existing row →
complete, no ML/thumb re-derive (file/pixels unchanged)."""
if result.status in ("imported", "superseded"):
return ("complete", True)
if result.status == "attached":
if result.status in ("attached", "refreshed"):
return ("complete", False)
if result.status == "skipped":
return ("skipped", False)
@@ -138,6 +140,15 @@ def _do_import(session, task, import_task_id: int) -> dict:
task.result_image_id = result.image_id
counter_col_name = "imported"
counter_col = ImportBatch.imported
elif result.status == "refreshed":
# Deep-scan rederive: existing row got phash/artist/sidecar
# refreshed. Task is complete (no further work), but counted in
# `refreshed` not `imported` so the UI can surface the actual
# work done. operator-flagged 2026-05-25.
task.status = "complete"
task.result_image_id = result.image_id
counter_col_name = "refreshed"
counter_col = ImportBatch.refreshed
elif result.status == "attached":
task.status = "complete"
counter_col_name = "attachments"
+167
View File
@@ -0,0 +1,167 @@
"""scan_library_for_rule Celery task — iterates image_record in keyset-
paginated batches, evaluates the audit rule per image, populates
LibraryAuditRun.matched_ids. Runs on the maintenance queue with a 2h soft
time limit (plenty of margin for 100k+ image libraries at ~100ms PIL
decode + histogram per image).
State machine:
start: status='running'
end success: status='ready'
end error: status='error', error=traceback
oversize: status='error', error='matched too many images; tighten threshold'
external cancel: scan sees status='cancelled' between batches, exits.
"""
import logging
import traceback
from datetime import UTC, datetime
from celery.exceptions import SoftTimeLimitExceeded
from PIL import Image
from sqlalchemy import select, update
from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..models import ImageRecord, LibraryAuditRun
from ..services.audits import single_color, transparency
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
_BATCH = 500
_PROGRESS_TICK = 100
_MAX_MATCHED = 50_000
_RULES = {
"transparency": transparency.evaluate,
"single_color": single_color.evaluate,
}
@celery.task(
name="backend.app.tasks.library_audit.scan_library_for_rule",
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=5,
retry_backoff_max=60,
retry_jitter=True,
max_retries=3,
soft_time_limit=7200,
time_limit=7500,
)
def scan_library_for_rule(self, audit_id: int) -> dict:
"""See module docstring. Returns a small summary dict for eager-mode
test assertions (real workers ignore the return value)."""
SessionLocal = _sync_session_factory()
try:
with SessionLocal() as session:
audit = session.get(LibraryAuditRun, audit_id)
if audit is None:
return {"audit_id": audit_id, "status": "missing"}
evaluate = _RULES.get(audit.rule)
if evaluate is None:
_mark_error(session, audit_id, f"unknown rule {audit.rule!r}")
return {"audit_id": audit_id, "status": "error"}
params = dict(audit.params or {})
matched: list[int] = []
scanned = 0
last_id = 0
while True:
# Cancellation check between batches.
current_status = session.execute(
select(LibraryAuditRun.status)
.where(LibraryAuditRun.id == audit_id)
).scalar_one()
if current_status == "cancelled":
return {"audit_id": audit_id, "status": "cancelled"}
rows = session.execute(
select(ImageRecord.id, ImageRecord.path)
.where(ImageRecord.id > last_id)
.where(ImageRecord.mime.like("image/%"))
.order_by(ImageRecord.id.asc())
.limit(_BATCH)
).all()
if not rows:
break
for image_id, image_path in rows:
last_id = image_id
scanned += 1
try:
with Image.open(image_path) as im:
try:
if evaluate(im, **params):
matched.append(image_id)
except Exception as exc: # noqa: BLE001
log.warning(
"audit %s: rule evaluate failed on %s: %s",
audit_id, image_path, exc,
)
except FileNotFoundError:
log.warning(
"audit %s: image_record %s file missing at %s; skipping",
audit_id, image_id, image_path,
)
except OSError as exc:
log.warning(
"audit %s: PIL load failed for %s: %s",
audit_id, image_path, exc,
)
if len(matched) > _MAX_MATCHED:
_mark_error(
session, audit_id,
f"matched > {_MAX_MATCHED} images; "
"tighten threshold and re-run",
)
return {"audit_id": audit_id, "status": "error"}
if scanned % _PROGRESS_TICK == 0:
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(scanned_count=scanned)
)
session.commit()
# Final state.
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(
scanned_count=scanned,
matched_count=len(matched),
matched_ids=matched,
status="ready",
finished_at=datetime.now(UTC),
)
)
session.commit()
return {
"audit_id": audit_id,
"status": "ready",
"scanned": scanned,
"matched": len(matched),
}
except SoftTimeLimitExceeded:
with SessionLocal() as session:
_mark_error(session, audit_id, "soft_time_limit exceeded (>7200s)")
raise
except (OperationalError, DBAPIError):
# Retryable per the decorator; leave row in 'running' and let
# autoretry try again. Recovery sweep catches if all retries fail.
raise
except Exception: # noqa: BLE001
tb = traceback.format_exc()
with SessionLocal() as session:
_mark_error(session, audit_id, tb)
raise
def _mark_error(session, audit_id: int, error_msg: str) -> None:
session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.id == audit_id)
.values(
status="error",
error=error_msg,
finished_at=datetime.now(UTC),
)
)
session.commit()
+62 -25
View File
@@ -16,6 +16,7 @@ from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
STUCK_THRESHOLD_MINUTES = 5
ORPHAN_PENDING_THRESHOLD_MINUTES = 30
OLD_TASK_DAYS = 7
PHASH_PAGE = 500
VERIFY_PAGE = 200
@@ -26,40 +27,76 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
@celery.task(name="backend.app.tasks.maintenance.recover_interrupted_tasks")
def recover_interrupted_tasks() -> int:
"""Find ImportTask rows stuck in 'processing' for >5 min and re-queue them.
"""Recover stuck ImportTask rows. Two distinct stuck states:
Why 5 min: import_media_file is sub-second for the vast majority of
files; even a large-video transcode caps at the per-task soft_time_limit
(5 min) defined on the task itself. Anything still 'processing' after
that window is a confirmed crash (worker died, DB disconnect mid-flush,
OOM) and must be recycled. Was 30 min historically; tightened
2026-05-24 after operator hit a 2224-row zombie pile during the IR
migration scan.
1. 'processing' > 5 min — worker crash mid-import. Re-queue via
.delay() and let the import retry. Was 30 min historically;
tightened 2026-05-24 after operator hit a 2224-row zombie pile.
import_media_file is sub-second for the vast majority of files and
capped at the per-task soft_time_limit (5 min), so anything still
'processing' after that window is a confirmed crash.
2. 'pending' or 'queued' > 30 min — enqueue-phase crash. scan_directory
creates rows with status='pending' (commit), then in a second pass
transitions to 'queued' and calls .delay() (commit). If the scanner
crashes between those two commits, rows are orphaned in 'pending'
(never enqueued) with no recovery path — invisible to the
'processing' sweep above. Flagged 2026-05-25 by operator hitting a
5490-row orphan pile. Flip these to 'failed' (not re-enqueue) so
the operator drains them via /api/import/retry-failed at their own
pace; bulk-re-enqueueing 5000+ rows would thundering-herd the
import worker.
Returns total rows touched (recovered + marked failed).
"""
SessionLocal = _sync_session_factory()
cutoff = datetime.now(UTC) - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
now = datetime.now(UTC)
processing_cutoff = now - timedelta(minutes=STUCK_THRESHOLD_MINUTES)
orphan_cutoff = now - timedelta(minutes=ORPHAN_PENDING_THRESHOLD_MINUTES)
with SessionLocal() as session:
stuck_ids = session.execute(
select(ImportTask.id)
.where(ImportTask.status == "processing")
.where(ImportTask.started_at < cutoff)
).scalars().all()
if not stuck_ids:
return 0
session.execute(
# Both sweeps used to be SELECT ids → UPDATE WHERE id IN (...) which
# blew past psycopg's 65535-parameter ceiling once a sweep covered
# tens of thousands of rows (operator hit it 2026-05-26 after the
# /import deep scan piled up orphans). Folding the SELECT into the
# UPDATE eliminates the IN-list entirely. RETURNING gives us back
# exactly the ids that flipped so the stuck sweep can still
# .delay() each one.
stuck_result = session.execute(
update(ImportTask)
.where(ImportTask.id.in_(stuck_ids))
.values(status="queued", started_at=None, error="recovered from stuck state")
.where(ImportTask.status == "processing")
.where(ImportTask.started_at < processing_cutoff)
.values(
status="queued",
started_at=None,
error="recovered from stuck state",
)
.returning(ImportTask.id)
)
stuck_ids = [row[0] for row in stuck_result.all()]
orphan_result = session.execute(
update(ImportTask)
.where(ImportTask.status.in_(["pending", "queued"]))
.where(ImportTask.created_at < orphan_cutoff)
.values(
status="failed",
error=(
"orphan pending/queued swept by recover_interrupted_tasks "
"(scanner likely crashed mid-enqueue); retry via "
"/api/import/retry-failed"
),
)
)
orphan_count = orphan_result.rowcount or 0
session.commit()
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
if stuck_ids:
from .import_file import import_media_file
for tid in stuck_ids:
import_media_file.delay(tid)
return len(stuck_ids)
return len(stuck_ids) + orphan_count
@celery.task(name="backend.app.tasks.maintenance.cleanup_old_tasks")
+17 -8
View File
@@ -59,16 +59,25 @@ def scan_directory(self, triggered_by: str = "manual",
session.flush()
batch_id = batch.id
# Skip-set: any source_path that already has a non-failed ImportTask
# row. Re-running scan_directory must not re-enqueue files the
# importer has already handled (or is currently handling); doing so
# creates duplicate work and inflates the queue. Failed prior tasks
# are eligible for retry.
# Skip-set behavior splits by mode (operator-flagged 2026-05-25):
#
# quick: any non-failed prior ImportTask (active OR finished) is
# skipped — quick scan only does new-file enqueue, so re-touching
# already-imported files is wasted work.
#
# deep: ONLY currently-in-flight tasks (pending/queued/processing)
# are skipped. Completed and skipped tasks ARE re-queued because
# deep scan exists precisely to re-touch already-imported files
# (refresh sidecar metadata, fill NULL phash, fill NULL artist
# via Importer._deep_rederive). Matches IR's deep-scan behavior.
active_statuses = ["pending", "queued", "processing"]
if mode == "deep":
skip_statuses = active_statuses
else:
skip_statuses = active_statuses + ["complete", "skipped"]
non_failed_existing = set(session.execute(
select(ImportTask.source_path).where(
ImportTask.status.in_(
["pending", "queued", "processing", "complete", "skipped"]
),
ImportTask.status.in_(skip_statuses),
)
).scalars().all())
+79
View File
@@ -17,6 +17,30 @@ from ._sync_engine import sync_session_factory as _sync_session_factory
IMAGES_ROOT = Path("/images")
THUMB_MAGIC_JPEG = b"\xff\xd8\xff"
THUMB_MAGIC_PNG = b"\x89PNG\r\n\x1a\n"
def _thumb_is_valid(path: Path) -> bool:
"""Return True iff `path` exists and starts with a JPEG or PNG magic header.
The on-disk thumbnail format is set by services/thumbnailer.py — JPEG for
opaque sources, PNG for alpha sources. Anything else (missing file, OSError,
truncated, wrong magic) is invalid.
"""
try:
with path.open("rb") as f:
head = f.read(12)
except OSError:
return False
if len(head) < 8:
return False
if head[:3] == THUMB_MAGIC_JPEG:
return True
if head[:8] == THUMB_MAGIC_PNG:
return True
return False
@celery.task(
name="backend.app.tasks.thumbnail.generate_thumbnail",
@@ -50,3 +74,58 @@ def generate_thumbnail(self, image_id: int) -> dict:
session.add(record)
session.commit()
return {"status": "ok", "image_id": image_id, "path": str(result.path)}
@celery.task(
name="backend.app.tasks.thumbnail.backfill_thumbnails",
bind=True,
)
def backfill_thumbnails(self) -> dict:
"""Scan ImageRecord and enqueue generate_thumbnail for rows whose
thumbnail is missing, gone from disk, or has wrong magic bytes.
Keyset paginates by id ASC, page size 500. NULLs out thumbnail_path for
rows that point at a missing or corrupt file before enqueueing — keeps
the DB self-consistent on partial runs and makes re-runs safe.
Returns {"enqueued": N, "ok": M, "regenerated": K} where:
- enqueued = total generate_thumbnail.delay() calls
- ok = rows whose existing thumbnail file is valid (skipped)
- regenerated = subset of enqueued that had a non-NULL thumbnail_path
cleared (i.e. missing + corrupt)
"""
from sqlalchemy import select, update
SessionLocal = _sync_session_factory()
enqueued = 0
ok = 0
regenerated = 0
last_id = 0
with SessionLocal() as session:
while True:
rows = session.execute(
select(ImageRecord.id, ImageRecord.thumbnail_path)
.where(ImageRecord.id > last_id)
.order_by(ImageRecord.id.asc())
.limit(500)
).all()
if not rows:
break
for image_id, thumb_path in rows:
if thumb_path is None:
generate_thumbnail.delay(image_id)
enqueued += 1
elif _thumb_is_valid(Path(thumb_path)):
ok += 1
else:
session.execute(
update(ImageRecord)
.where(ImageRecord.id == image_id)
.values(thumbnail_path=None)
)
generate_thumbnail.delay(image_id)
enqueued += 1
regenerated += 1
session.commit()
last_id = rows[-1][0]
return {"enqueued": enqueued, "ok": ok, "regenerated": regenerated}
+14 -1
View File
@@ -13,8 +13,21 @@ HASH_SIZE = 8
def compute_phash(pil_image) -> str | None:
"""Perceptual hash of an opened PIL image, as a hex string. None on any
failure (videos/unreadable/non-image)."""
failure (videos/unreadable/non-image).
For animated images (multi-frame WebP/GIF/APNG), explicitly seek to
frame 0 first. Without this, some PIL operations downstream of
imagehash.phash (convert("L"), resize) can iterate all frames and
blow past Celery's hard time limit on large animations
(operator-flagged 2026-05-26 against animated WebPs). The pHash of
frame 0 is the conventional choice for animated content.
"""
try:
if getattr(pil_image, "is_animated", False):
try:
pil_image.seek(0)
except Exception:
pass
return str(imagehash.phash(pil_image, hash_size=HASH_SIZE))
except Exception:
return None
+19
View File
@@ -4,6 +4,7 @@ No per-platform branching: a small common key set with fallbacks; the
full JSON is kept in raw so anything unmapped is recoverable later.
"""
import re
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
@@ -21,10 +22,28 @@ class SidecarData:
raw: dict
# gallery-dl prefixes media filenames with `NN_` for in-post ordering
# (`01_HOLLOW-ICHIGO.png`, `02_HOLOW ICHIGO.zip`) but writes the sidecar
# under the attachment's stem WITHOUT that ordering prefix
# (`HOLLOW-ICHIGO.json`). Strip the prefix when looking for sidecars.
# Confirmed against real Patreon downloads 2026-05-26 — without this
# strip, every gallery-dl post-level sidecar was invisible to FC since
# FC-3 shipped (24 deep-refresh calls produced 0 Posts in operator's DB).
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
def find_sidecar(media: Path) -> Path | None:
# Attachment-level sidecars (image.jpg.json, image.json).
for cand in (media.with_suffix(".json"), Path(str(media) + ".json")):
if cand.is_file():
return cand
# gallery-dl post-numbered convention: strip the `NN_` prefix from
# the stem and look for that.json in the same directory.
m = _NUMBERING_PREFIX.match(media.stem)
if m:
cand = media.parent / f"{m.group(1)}.json"
if cand.is_file():
return cand
return None
+46
View File
@@ -0,0 +1,46 @@
"""Parse the user-facing `kind:name` shortcut used by the add-tag input.
Mirrors IR's app/utils/tag_prefix.py. Tag.name in FC is stored bare;
the `kind:` prefix only exists as an input convention at user-facing
places (image-modal add-tag input, future bulk-add forms). The parser
is the single owner of the kind-string list — anything not in
KNOWN_KINDS keeps its colon as literal text.
"""
from __future__ import annotations
# Kinds the user can type as a prefix at the input boundary.
# Exclusions:
# - `general` is the default for un-prefixed input (never typed as prefix)
# - `archive`, `post` are system-managed
# - `artist` was retired in FC-2d-vii-c — artists are first-class
# entities (Artist row + ImageRecord.artist_id), browsed via the
# provenance axis rather than as tags. See project_provenance_separation.
# - `meta`, `rating` retired as user-typeable per operator 2026-05-26 —
# content classification only needs character/fandom/series.
KNOWN_KINDS: frozenset[str] = frozenset({
"character",
"fandom",
"series",
})
def parse_kind_prefix(raw: str) -> tuple[str | None, str]:
"""Split a raw user-typed tag string into (kind, name).
Returns (kind, name) where kind is lowercase canonical and in
KNOWN_KINDS, or (None, raw.strip()) if no recognized prefix is
present. `name` is always whitespace-stripped.
Examples:
parse_kind_prefix("character:Saber") -> ("character", "Saber")
parse_kind_prefix("Character:Saber") -> ("character", "Saber")
parse_kind_prefix("sunset") -> (None, "sunset")
parse_kind_prefix("http://example") -> (None, "http://example")
parse_kind_prefix("fandom: FSN ") -> ("fandom", "FSN")
"""
if ":" in raw:
prefix, rest = raw.split(":", 1)
if prefix.lower() in KNOWN_KINDS:
return prefix.lower(), rest.strip()
return None, raw.strip()
+6 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "FabledCurator",
"version": "1.0.0",
"version": "1.0.4",
"description": "Export cookies from supported platforms to FabledCurator and add creators as sources in one click.",
"browser_specific_settings": {
@@ -11,6 +11,11 @@
}
},
"content_security_policy": {
"_comment": "Override the MV3 default CSP to OMIT upgrade-insecure-requests. FC runs over plain HTTP per the homelab posture (feedback_homelab_http), and the default MV3 CSP would silently upgrade every fetch(http://curator.../...) to https:// and fail with NS_ERROR_GENERATE_FAILURE. Operator-flagged 2026-05-26 after the 'Test connection' button errored despite a working CORS preflight on the backend.",
"extension_pages": "script-src 'self'; object-src 'self';"
},
"permissions": [
"cookies",
"storage",
+5 -5
View File
@@ -1,13 +1,13 @@
{
"name": "fabledcurator-extension",
"version": "1.0.0",
"version": "1.0.4",
"private": true,
"description": "Firefox extension for FabledCurator",
"scripts": {
"lint": "web-ext lint --source-dir=.",
"start": "web-ext run --source-dir=. --firefox=firefox",
"build": "web-ext build --source-dir=. --overwrite-dest",
"sign": "web-ext sign --source-dir=. --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
"lint": "web-ext lint --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore",
"start": "web-ext run --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --firefox=firefox",
"build": "web-ext build --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --overwrite-dest",
"sign": "web-ext sign --source-dir=. --no-config-discovery --ignore-files package.json package-lock.json web-ext-artifacts node_modules README.md .gitignore --channel=unlisted --api-key=$WEB_EXT_API_KEY --api-secret=$WEB_EXT_API_SECRET"
},
"devDependencies": {
"web-ext": "^8.0.0"
-13
View File
@@ -1,13 +0,0 @@
module.exports = {
sourceDir: '.',
artifactsDir: './web-ext-artifacts',
ignoreFiles: [
'package.json',
'package-lock.json',
'web-ext-config.cjs',
'web-ext-artifacts',
'node_modules',
'README.md',
'.gitignore',
],
};
+2
View File
@@ -4,6 +4,7 @@
<RouterView />
</AppShell>
<ImageViewer v-if="modal.isOpen" @close="modal.close()" />
<PostModal />
<AppSnackbar ref="snackbar" />
</v-app>
</template>
@@ -13,6 +14,7 @@ import { onMounted, ref } from 'vue'
import AppShell from './components/AppShell.vue'
import AppSnackbar from './components/AppSnackbar.vue'
import ImageViewer from './components/modal/ImageViewer.vue'
import PostModal from './components/posts/PostModal.vue'
import { useModalStore } from './stores/modal.js'
const modal = useModalStore()
@@ -0,0 +1,48 @@
<template>
<div class="fc-artist-gallery">
<MasonryGrid
:items="store.images"
:loading="store.imagesLoading"
:has-more="store.hasMoreImages"
@load-more="store.loadMoreImages(props.slug)"
@open="openImage"
/>
</div>
</template>
<script setup>
import { onMounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../../stores/artist.js'
import { useModalStore } from '../../stores/modal.js'
import MasonryGrid from '../discovery/MasonryGrid.vue'
const props = defineProps({
slug: { type: String, required: true },
})
const store = useArtistStore()
const modal = useModalStore()
const route = useRoute()
const router = useRouter()
onMounted(() => {
const initial = parseInt(route.query.image, 10)
if (!isNaN(initial)) modal.open(initial)
})
watch(() => route.query.image, (q) => {
const id = parseInt(q, 10)
if (!isNaN(id) && id !== modal.currentImageId) modal.open(id)
else if (isNaN(id) && modal.currentImageId !== null) modal.close()
})
function openImage (id) {
router.push({ query: { ...route.query, image: id } })
}
</script>
<style scoped>
.fc-artist-gallery { min-width: 0; }
</style>
@@ -0,0 +1,128 @@
<template>
<header class="fc-artist-header">
<div class="fc-artist-header__left">
<h1 class="fc-artist-header__name">{{ name }}</h1>
<span v-if="stats" class="fc-artist-header__stats">{{ stats }}</span>
</div>
<v-tabs
:model-value="modelValue"
color="accent"
density="compact"
class="fc-artist-header__tabs"
@update:model-value="$emit('update:modelValue', $event)"
>
<v-tab value="posts">
Posts
<span v-if="postCount != null" class="fc-artist-header__tab-count">
({{ postCount }})
</span>
</v-tab>
<v-tab value="gallery">
Gallery
<span v-if="imageCount != null" class="fc-artist-header__tab-count">
({{ imageCount }})
</span>
</v-tab>
<v-tab value="management">Management</v-tab>
</v-tabs>
<!-- Right-side spacer: balances the left cell's flex weight so the
centered tabs stay geometrically centered regardless of the
artist-name length. Mirrors TopNav's 1fr | auto | 1fr layout. -->
<div class="fc-artist-header__right" />
</header>
</template>
<script setup>
import { computed } from 'vue'
const props = defineProps({
name: { type: String, required: true },
imageCount: { type: Number, default: null },
postCount: { type: Number, default: null },
lastAdded: { type: String, default: null },
modelValue: { type: String, required: true },
})
defineEmits(['update:modelValue'])
const stats = computed(() => {
const parts = []
if (props.imageCount != null) {
parts.push(`${props.imageCount} image${props.imageCount === 1 ? '' : 's'}`)
}
if (props.lastAdded) {
parts.push(`last added ${props.lastAdded.slice(0, 10)}`)
}
return parts.join(' · ')
})
</script>
<style scoped>
/* Matches TopNav.vue's frosted recipe exactly. top:48px parks it flush
against TopNav's bottom edge (TopNav is 0.75rem padding + ~24px content
= ~48px tall; operator-flagged 2026-05-26 that top:64px left a visible
gap). The two bars now read as one continuous frosted strip. */
.fc-artist-header {
position: sticky;
top: 48px;
z-index: 4;
display: flex;
align-items: center;
gap: 1rem;
padding: 0.5rem 1rem;
background: linear-gradient(
to bottom,
rgba(20, 23, 26, 0.92) 0%,
rgba(20, 23, 26, 0.65) 60%,
rgba(20, 23, 26, 0) 100%
);
backdrop-filter: blur(2px);
-webkit-backdrop-filter: blur(2px);
}
.fc-artist-header__left {
flex: 1 1 0;
min-width: 0;
display: flex;
align-items: baseline;
gap: 12px;
overflow: hidden;
}
.fc-artist-header__name {
font-family: 'Fraunces', Georgia, serif;
font-size: 24px;
font-weight: 500;
margin: 0;
color: rgb(var(--v-theme-on-surface));
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.fc-artist-header__stats {
font-size: 13px;
color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.fc-artist-header__tabs {
flex: 0 0 auto;
}
.fc-artist-header__right {
flex: 1 1 0;
min-width: 0;
}
.fc-artist-header__tab-count {
margin-left: 4px;
font-size: 12px;
color: rgb(var(--v-theme-on-surface-variant));
font-variant-numeric: tabular-nums;
}
</style>
@@ -0,0 +1,133 @@
<template>
<div class="fc-artist-mgmt">
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Overview</h2>
<div class="fc-artist-mgmt__chips">
<v-chip
size="small"
:variant="overview.is_subscription ? 'flat' : 'outlined'"
:color="overview.is_subscription ? 'accent' : undefined"
prepend-icon="mdi-rss"
>{{ overview.is_subscription ? 'Subscription' : 'One-off' }}</v-chip>
<v-chip
size="small" variant="outlined" prepend-icon="mdi-link-variant"
:to="`/subscriptions?artist_id=${overview.id}`"
>{{ overview.sources.length }} subscription{{ overview.sources.length === 1 ? '' : 's' }}</v-chip>
</div>
</section>
<section v-if="overview.cooccurring_tags.length" class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Frequent tags</h2>
<div class="fc-artist-mgmt__tags">
<v-chip
v-for="t in overview.cooccurring_tags" :key="t.id"
size="small" @click="openTag(t.id)"
>{{ t.name }} <span class="fc-artist-mgmt__tagc">{{ t.count }}</span></v-chip>
</div>
</section>
<section v-if="overview.activity.length" class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Activity</h2>
<svg
class="fc-artist-mgmt__spark" :viewBox="`0 0 ${sparkW} ${sparkH}`"
preserveAspectRatio="none" role="img" aria-label="posts over time"
>
<polyline :points="sparkPoints" fill="none"
stroke="rgb(var(--v-theme-accent))" stroke-width="2" />
</svg>
</section>
<section v-if="overview.sources.length" class="fc-artist-mgmt__sec">
<div class="fc-artist-mgmt__sec-head">
<h2 class="fc-h2">Subscriptions</h2>
<RouterLink
:to="`/subscriptions?artist_id=${overview.id}`"
class="fc-artist-mgmt__manage"
>Manage subscriptions </RouterLink>
</div>
<v-table density="compact">
<thead>
<tr><th>Platform</th><th>URL</th><th class="text-right">Images</th></tr>
</thead>
<tbody>
<tr v-for="s in overview.sources" :key="s.id">
<td>{{ s.platform }}</td>
<td class="fc-artist-mgmt__url">{{ s.url }}</td>
<td class="text-right">{{ s.image_count }}</td>
</tr>
</tbody>
</v-table>
</section>
<section class="fc-artist-mgmt__sec">
<h2 class="fc-h2">Danger zone</h2>
<ArtistDangerZone
:slug="overview.slug"
:artist-id="overview.id"
:artist-name="overview.name"
/>
</section>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { useRouter, RouterLink } from 'vue-router'
import ArtistDangerZone from './ArtistDangerZone.vue'
const props = defineProps({
overview: { type: Object, required: true },
})
const router = useRouter()
const sparkW = 600
const sparkH = 80
const sparkPoints = computed(() => {
const a = props.overview.activity ?? []
if (a.length === 0) return ''
const max = Math.max(...a.map(p => p.count), 1)
const stepX = a.length > 1 ? sparkW / (a.length - 1) : 0
return a.map((p, i) => {
const x = i * stepX
const y = sparkH - (p.count / max) * (sparkH - 4) - 2
return `${x.toFixed(1)},${y.toFixed(1)}`
}).join(' ')
})
function openTag (tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } })
}
</script>
<style scoped>
.fc-artist-mgmt { padding-top: 1rem; }
.fc-h2 {
font-family: 'Fraunces', Georgia, serif;
font-size: 20px; font-weight: 500; margin-bottom: 8px;
}
.fc-artist-mgmt__sec { margin-bottom: 28px; }
.fc-artist-mgmt__chips { display: flex; gap: 8px; flex-wrap: wrap; }
.fc-artist-mgmt__tags { display: flex; flex-wrap: wrap; gap: 6px; }
.fc-artist-mgmt__tagc {
color: rgb(var(--v-theme-on-surface-variant));
margin-left: 4px;
font-variant-numeric: tabular-nums;
}
.fc-artist-mgmt__spark { width: 100%; height: 80px; }
.fc-artist-mgmt__url {
max-width: 380px; overflow: hidden; text-overflow: ellipsis;
white-space: nowrap;
}
.fc-artist-mgmt__sec-head {
display: flex; align-items: baseline; justify-content: space-between;
margin-bottom: 0.25rem;
}
.fc-artist-mgmt__manage {
font-size: 0.85rem;
color: rgb(var(--v-theme-accent));
text-decoration: none;
}
.fc-artist-mgmt__manage:hover { text-decoration: underline; }
</style>
@@ -0,0 +1,90 @@
<template>
<div class="fc-artist-posts">
<v-alert v-if="store.error" type="error" variant="tonal" closable class="mb-3">
{{ String(store.error) }}
</v-alert>
<div v-if="store.loading && store.items.length === 0" class="fc-artist-posts__loading">
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<div v-else-if="store.items.length === 0 && store.done" class="fc-artist-posts__empty">
<p>No posts for this artist yet. Switch to
<a href="#" @click.prevent="$emit('switch-tab', 'gallery')">Gallery</a>
to see imported images, or visit
<RouterLink to="/subscriptions">Subscriptions</RouterLink>
to start capturing posts.
</p>
</div>
<div v-else>
<PostCard v-for="p in store.items" :key="p.id" :post="p" />
<div ref="sentinel" class="fc-artist-posts__sentinel">
<v-progress-circular v-if="store.loading" indeterminate color="accent" size="24" />
<span v-else-if="store.done" class="fc-artist-posts__end">End of stream</span>
</div>
</div>
</div>
</template>
<script setup>
import { onMounted, onUnmounted, ref, watch } from 'vue'
import { RouterLink } from 'vue-router'
import { usePostsStore } from '../../stores/posts.js'
import PostCard from '../posts/PostCard.vue'
const props = defineProps({
artistId: { type: Number, required: true },
})
defineEmits(['switch-tab'])
const store = usePostsStore()
const sentinel = ref(null)
let observer = null
async function reload () {
await store.loadInitial({ artist_id: props.artistId, platform: null })
}
watch(() => props.artistId, reload)
onMounted(async () => {
await reload()
observer = new IntersectionObserver((entries) => {
if (entries.some(e => e.isIntersecting)) {
store.loadMore()
}
}, { rootMargin: '400px 0px' })
if (sentinel.value) observer.observe(sentinel.value)
})
onUnmounted(() => {
if (observer) observer.disconnect()
})
</script>
<style scoped>
.fc-artist-posts {
max-width: 1600px;
margin: 0 auto;
}
.fc-artist-posts__loading,
.fc-artist-posts__empty {
display: flex;
justify-content: center;
padding: 2rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-artist-posts__sentinel {
display: flex;
justify-content: center;
padding: 1.5rem 0;
min-height: 2rem;
}
.fc-artist-posts__end {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.85rem;
}
</style>
@@ -0,0 +1,127 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-image-size-select-small" size="small" />
<span>Minimum dimensions</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Find and delete images smaller than the threshold. Mirrors the
import-time <code>min_width</code> / <code>min_height</code>
filter, applied retroactively to the existing library.
</p>
<v-row dense>
<v-col cols="6">
<v-text-field
v-model.number="minW" label="Min width (px)" type="number"
min="0" density="compact" hide-details
/>
</v-col>
<v-col cols="6">
<v-text-field
v-model.number="minH" label="Min height (px)" type="number"
min="0" density="compact" hide-details
/>
</v-col>
</v-row>
<div class="d-flex align-center mt-3" style="gap: 10px;">
<v-btn
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify"
:loading="busy"
@click="onPreview"
>Preview</v-btn>
<span v-if="preview" class="text-body-2">
<strong>{{ preview.count }}</strong> image(s) would be deleted.
</span>
</div>
<v-btn
v-if="preview && preview.count > 0"
class="mt-3"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onDeleteClick"
>Delete {{ preview.count }} matching...</v-btn>
</v-card-text>
<DestructiveConfirmModal
v-model="showModal"
action="delete"
kind="min-dim"
:run-id="tokenSha8"
tier="C"
:projected-counts="projectedCounts"
:description="`Width < ${minW} OR height < ${minH}`"
@confirm="onConfirmedDelete"
/>
</v-card>
</template>
<script setup>
import { onMounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
const store = useCleanupStore()
const minW = ref(0)
const minH = ref(0)
const preview = ref(null)
const busy = ref(false)
const showModal = ref(false)
const tokenSha8 = ref('')
const projectedCounts = ref({})
onMounted(async () => {
await store.loadDefaults()
minW.value = store.defaults.min_width
minH.value = store.defaults.min_height
})
// SHA-256 truncated to 8 hex chars — matches the backend's
// _min_dim_token() exactly. Web Crypto rejects MD5 as insecure.
async function sha8(canon) {
const enc = new TextEncoder()
const buf = await crypto.subtle.digest('SHA-256', enc.encode(canon))
const hex = Array.from(new Uint8Array(buf)).map(b => b.toString(16).padStart(2, '0')).join('')
return hex.slice(0, 8)
}
async function onPreview() {
busy.value = true
try {
preview.value = await store.previewMinDim(minW.value, minH.value)
} catch (e) {
window.__fcToast?.({ text: `Preview failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
async function onDeleteClick() {
tokenSha8.value = await sha8(`${minW.value}x${minH.value}`)
projectedCounts.value = { 'Images to delete': preview.value.count }
showModal.value = true
}
async function onConfirmedDelete(token) {
try {
const res = await store.deleteMinDim(minW.value, minH.value, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
preview.value = null
} catch (e) {
window.__fcToast?.({ text: `Delete failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,183 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-palette-swatch" size="small" />
<span>Single-color audit</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Scan library for images dominated by one color within the
tolerance. Catches placeholder / solid-fill / error-page images
that slipped through the import filter. Same background-scan
cadence as the transparency audit.
</p>
<v-row dense>
<v-col cols="6">
<v-text-field
v-model.number="threshold" label="Threshold (01)"
type="number" min="0" max="1" step="0.01"
density="compact" hide-details
:disabled="audit && audit.status === 'running'"
/>
</v-col>
<v-col cols="6">
<v-text-field
v-model.number="tolerance" label="Color tolerance (0441)"
type="number" min="0" max="441"
density="compact" hide-details
:disabled="audit && audit.status === 'running'"
/>
</v-col>
</v-row>
<v-btn
v-if="!audit || audit.status !== 'running'"
class="mt-3"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify-scan"
:loading="busy"
@click="onStart"
>Scan library</v-btn>
<div v-if="audit && audit.status === 'running'" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 d-flex align-center" style="gap: 10px;">
<span>
Scanning {{ audit.scanned_count }} checked,
{{ audit.matched_count }} matched
</span>
<v-btn
variant="text" size="small" color="warning" rounded="pill"
@click="onCancel"
>Cancel</v-btn>
</div>
</div>
<div v-if="audit && audit.status === 'ready'" class="mt-3">
<p class="text-body-2 mb-2">
Scan complete. <strong>{{ audit.matched_count }}</strong>
image(s) match.
</p>
<v-btn
v-if="audit.matched_count > 0"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onApplyClick"
>Delete {{ audit.matched_count }} matching...</v-btn>
</div>
<v-alert
v-if="audit && audit.status === 'error'"
type="error" variant="tonal" density="compact" class="mt-3"
>Scan failed: {{ audit.error }}</v-alert>
<v-alert
v-if="audit && audit.status === 'applied'"
type="success" variant="tonal" density="compact" class="mt-3"
>Applied matched images deleted.</v-alert>
</v-card-text>
<DestructiveConfirmModal
v-if="audit"
v-model="showModal"
action="delete"
kind="audit"
:run-id="audit.id"
tier="C"
:projected-counts="projectedCounts"
description="Permanently deletes images matched by the single-color scan."
@confirm="onConfirmedApply"
/>
</v-card>
</template>
<script setup>
import { onMounted, onUnmounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
const store = useCleanupStore()
const threshold = ref(0.95)
const tolerance = ref(30)
const audit = ref(null)
const busy = ref(false)
const showModal = ref(false)
const projectedCounts = ref({})
let pollTimer = null
onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.single_color_threshold
tolerance.value = store.defaults.single_color_tolerance
})
onUnmounted(() => stopPoll())
function startPoll(id) {
stopPoll()
pollTimer = setInterval(async () => {
try {
const fresh = await store.getAudit(id)
audit.value = fresh
if (fresh.status !== 'running') stopPoll()
} catch (e) {
stopPoll()
window.__fcToast?.({ text: `Audit poll failed: ${e.message}`, type: 'error' })
}
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onStart() {
busy.value = true
try {
const res = await store.startAudit('single_color', {
threshold: threshold.value, tolerance: tolerance.value,
})
audit.value = await store.getAudit(res.audit_id)
startPoll(res.audit_id)
} catch (e) {
window.__fcToast?.({ text: `Scan start failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
async function onCancel() {
if (!audit.value) return
try {
await store.cancelAudit(audit.value.id)
audit.value = await store.getAudit(audit.value.id)
stopPoll()
} catch (e) {
window.__fcToast?.({ text: `Cancel failed: ${e.message}`, type: 'error' })
}
}
function onApplyClick() {
projectedCounts.value = { 'Images to delete': audit.value.matched_count }
showModal.value = true
}
async function onConfirmedApply(token) {
try {
const res = await store.applyAudit(audit.value.id, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
audit.value = await store.getAudit(audit.value.id)
} catch (e) {
window.__fcToast?.({ text: `Apply failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,166 @@
<template>
<v-card class="fc-clean-card">
<v-card-title class="d-flex align-center" style="gap: 10px;">
<v-icon icon="mdi-checkerboard" size="small" />
<span>Transparency audit</span>
</v-card-title>
<v-card-text>
<p class="fc-muted text-body-2 mb-3">
Scan library for images whose transparent-pixel fraction exceeds
the threshold. Animated WebPs / GIFs are skipped (the import-side
rule does the same). Runs as a background task ~50ms per image,
so a 57k library takes ~50 minutes.
</p>
<v-text-field
v-model.number="threshold" label="Transparency threshold (01)"
type="number" min="0" max="1" step="0.01" density="compact" hide-details
:disabled="audit && audit.status === 'running'"
class="mb-3"
/>
<v-btn
v-if="!audit || audit.status !== 'running'"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-magnify-scan"
:loading="busy"
@click="onStart"
>Scan library</v-btn>
<div v-if="audit && audit.status === 'running'" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 d-flex align-center" style="gap: 10px;">
<span>
Scanning {{ audit.scanned_count }} checked,
{{ audit.matched_count }} matched
</span>
<v-btn
variant="text" size="small" color="warning" rounded="pill"
@click="onCancel"
>Cancel</v-btn>
</div>
</div>
<div v-if="audit && audit.status === 'ready'" class="mt-3">
<p class="text-body-2 mb-2">
Scan complete. <strong>{{ audit.matched_count }}</strong>
image(s) match.
</p>
<v-btn
v-if="audit.matched_count > 0"
color="error" variant="flat" rounded="pill"
prepend-icon="mdi-delete"
@click="onApplyClick"
>Delete {{ audit.matched_count }} matching...</v-btn>
</div>
<v-alert
v-if="audit && audit.status === 'error'"
type="error" variant="tonal" density="compact" class="mt-3"
>Scan failed: {{ audit.error }}</v-alert>
<v-alert
v-if="audit && audit.status === 'applied'"
type="success" variant="tonal" density="compact" class="mt-3"
>Applied matched images deleted.</v-alert>
</v-card-text>
<DestructiveConfirmModal
v-if="audit"
v-model="showModal"
action="delete"
kind="audit"
:run-id="audit.id"
tier="C"
:projected-counts="projectedCounts"
description="Permanently deletes images matched by the transparency scan."
@confirm="onConfirmedApply"
/>
</v-card>
</template>
<script setup>
import { onMounted, onUnmounted, ref } from 'vue'
import DestructiveConfirmModal from '../modal/DestructiveConfirmModal.vue'
import { useCleanupStore } from '../../stores/cleanup.js'
const store = useCleanupStore()
const threshold = ref(0.9)
const audit = ref(null)
const busy = ref(false)
const showModal = ref(false)
const projectedCounts = ref({})
let pollTimer = null
onMounted(async () => {
await store.loadDefaults()
threshold.value = store.defaults.transparency_threshold
})
onUnmounted(() => stopPoll())
function startPoll(id) {
stopPoll()
pollTimer = setInterval(async () => {
try {
const fresh = await store.getAudit(id)
audit.value = fresh
if (fresh.status !== 'running') stopPoll()
} catch (e) {
stopPoll()
window.__fcToast?.({ text: `Audit poll failed: ${e.message}`, type: 'error' })
}
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onStart() {
busy.value = true
try {
const res = await store.startAudit('transparency', { threshold: threshold.value })
audit.value = await store.getAudit(res.audit_id)
startPoll(res.audit_id)
} catch (e) {
window.__fcToast?.({ text: `Scan start failed: ${e.message}`, type: 'error' })
} finally {
busy.value = false
}
}
async function onCancel() {
if (!audit.value) return
try {
await store.cancelAudit(audit.value.id)
audit.value = await store.getAudit(audit.value.id)
stopPoll()
} catch (e) {
window.__fcToast?.({ text: `Cancel failed: ${e.message}`, type: 'error' })
}
}
function onApplyClick() {
projectedCounts.value = { 'Images to delete': audit.value.matched_count }
showModal.value = true
}
async function onConfirmedApply(token) {
try {
const res = await store.applyAudit(audit.value.id, token)
window.__fcToast?.({
text: `Deleted ${res.deleted} image(s)`, type: 'success',
})
audit.value = await store.getAudit(audit.value.id)
} catch (e) {
window.__fcToast?.({ text: `Apply failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-clean-card { border-radius: 8px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,172 @@
<template>
<v-dialog :model-value="modelValue" max-width="900"
@update:model-value="$emit('update:modelValue', $event)">
<v-card>
<v-card-title class="d-flex align-center" style="gap: 12px;">
<v-icon icon="mdi-alert-circle-outline" color="error" />
<span>{{ displayTitle }}</span>
<v-spacer />
<v-btn icon variant="text" size="small" @click="close">
<v-icon>mdi-close</v-icon>
</v-btn>
</v-card-title>
<v-card-text>
<dl v-if="contextRows.length" class="fc-err-context">
<template v-for="(row, idx) in contextRows" :key="idx">
<dt>{{ row[0] }}</dt>
<dd>{{ row[1] }}</dd>
</template>
</dl>
<pre class="fc-err-pre">{{ displayMessage || '(no error message)' }}</pre>
</v-card-text>
<v-card-actions>
<v-btn
variant="text" rounded="pill" size="small"
:prepend-icon="copied ? 'mdi-check' : 'mdi-content-copy'"
@click="onCopy"
>{{ copied ? 'Copied' : 'Copy' }}</v-btn>
<v-spacer />
<v-btn variant="text" rounded="pill" @click="close">Close</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed, ref, watch } from 'vue'
import { copyText } from '../../utils/clipboard.js'
const props = defineProps({
modelValue: { type: Boolean, default: false },
// Legacy mode: pass title + message strings. Used by callers whose row
// shape lacks structured context (e.g. ImportTaskList where `error` is
// a plain string on the import_task row, not a TaskRun).
title: { type: String, default: 'Error details' },
message: { type: String, default: '' },
// Row mode: pass the full TaskRun-shaped dict from /api/system_activity.
// When set, displayTitle/displayMessage derive from the row and a context
// panel of task_name/queue/target/duration/etc. renders above the error.
row: { type: Object, default: null },
})
const emit = defineEmits(['update:modelValue'])
const copied = ref(false)
let copiedTimer = null
const displayTitle = computed(() => {
if (props.row) return props.row.error_type || 'Error details'
return props.title
})
const displayMessage = computed(() => {
if (props.row) return props.row.error_message || ''
return props.message
})
function _shortTaskName (name) {
if (!name) return ''
const parts = String(name).split('.')
return parts[parts.length - 1]
}
function _formatDuration (ms) {
if (ms == null) return null
if (ms < 1000) return `${ms} ms`
if (ms < 60_000) return `${(ms / 1000).toFixed(1)} s`
return `${(ms / 60_000).toFixed(1)} min`
}
const contextRows = computed(() => {
const r = props.row
if (!r) return []
const rows = []
if (r.task_name) rows.push(['Task', _shortTaskName(r.task_name)])
if (r.queue) rows.push(['Queue', r.queue])
if (r.target_id != null) rows.push(['Target', r.target_id])
const dur = _formatDuration(r.duration_ms)
if (dur != null) rows.push(['Duration', dur])
if (r.started_at) rows.push(['Started', r.started_at])
if (r.finished_at) rows.push(['Finished', r.finished_at])
if (r.retry_count) rows.push(['Retries', r.retry_count])
if (r.worker_hostname) rows.push(['Worker', r.worker_hostname])
if (r.celery_task_id) rows.push(['Celery ID', r.celery_task_id])
if (r.args_summary) rows.push(['Args', r.args_summary])
return rows
})
watch(() => props.modelValue, (open) => {
if (!open) {
copied.value = false
if (copiedTimer) { clearTimeout(copiedTimer); copiedTimer = null }
}
})
function close () {
emit('update:modelValue', false)
}
async function onCopy () {
let text = displayMessage.value || ''
if (contextRows.value.length) {
const header = contextRows.value.map(([k, v]) => `${k}: ${v}`).join('\n')
text = `${header}\n\nError: ${displayTitle.value}\n${text}`
}
try {
await copyText(text)
copied.value = true
if (copiedTimer) clearTimeout(copiedTimer)
copiedTimer = setTimeout(() => { copied.value = false }, 1500)
} catch (e) {
window.__fcToast?.({ text: `Copy failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
/* Context panel: muted labels in vellum, crisp values in parchment. The
2-column key/value grid keeps rows visually scannable when there are
many fields (Task, Queue, Target, Duration, Started, Finished, Retries,
Worker, Celery ID, Args). */
.fc-err-context {
display: grid;
grid-template-columns: max-content 1fr;
column-gap: 14px;
row-gap: 4px;
margin: 0 0 14px 0;
font-size: 13px;
}
.fc-err-context dt {
color: rgb(var(--v-theme-on-surface-variant));
font-weight: 500;
white-space: nowrap;
}
.fc-err-context dd {
color: rgb(var(--v-theme-on-surface));
margin: 0;
font-variant-numeric: tabular-nums;
word-break: break-all;
}
/* Error pre block: high-contrast pairing. The page's `background` token
(obsidian #14171A) is darker than the modal card's `surface` (iron
#1E2228), so parchment text reads crisply against it. The prior pairing
used `surface-variant` which Vuetify auto-derives to a near-parchment
light value in this theme — pale-on-pale and unreadable. Operator-
flagged 2026-05-26 ("ui contrast is poor"). */
.fc-err-pre {
font-family: 'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 12px;
line-height: 1.5;
white-space: pre-wrap;
word-break: break-word;
background: rgb(var(--v-theme-background));
color: rgb(var(--v-theme-on-surface));
padding: 12px 14px;
border-radius: 6px;
max-height: 60vh;
overflow: auto;
margin: 0;
}
</style>
@@ -28,6 +28,7 @@
<script setup>
import { onMounted, ref } from 'vue'
import { useCredentialsStore } from '../../stores/credentials.js'
import { copyText } from '../../utils/clipboard.js'
const store = useCredentialsStore()
const showRotateConfirm = ref(false)
@@ -37,7 +38,7 @@ onMounted(() => store.loadKey())
async function copyKey() {
if (!store.extensionKey) return
try {
await navigator.clipboard.writeText(store.extensionKey)
await copyText(store.extensionKey)
globalThis.window?.__fcToast?.({ text: 'Copied', type: 'success' })
} catch {
globalThis.window?.__fcToast?.({ text: 'Copy failed', type: 'error' })
@@ -38,9 +38,18 @@ function onCardClick() {
.fc-artistcard { cursor: pointer; }
.fc-artistcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute. */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-artistcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-artistcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-artistcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
+12 -2
View File
@@ -106,9 +106,19 @@ function submit() {
.fc-tagcard { cursor: pointer; }
.fc-tagcard__previews {
display: grid; grid-template-columns: repeat(3, 1fr);
gap: 2px; aspect-ratio: 3 / 1; background: rgb(var(--v-theme-surface-light));
gap: 2px; aspect-ratio: 3 / 1;
/* Explicit floor + ceiling so tall source images can't escape the
preview slot even on browsers where aspect-ratio doesn't compute
(older Safari, embedded webviews). */
min-height: 150px; max-height: 220px;
overflow: hidden;
background: rgb(var(--v-theme-surface-light));
}
.fc-tagcard__previews img {
display: block;
width: 100%; height: 100%;
object-fit: cover; object-position: center;
}
.fc-tagcard__previews img { width: 100%; height: 100%; object-fit: cover; }
.fc-tagcard__noimg {
grid-column: 1 / -1; display: flex; align-items: center;
justify-content: center;
@@ -1,17 +1,18 @@
<template>
<div class="fc-tag-autocomplete">
<div class="d-flex" style="gap: 6px;">
<v-select
v-model="kind" :items="kindOptions" :item-title="(k) => k.label" :item-value="(k) => k.value"
density="compact" hide-details style="max-width: 140px;"
/>
<v-text-field
v-model="query" placeholder="Add tag…" density="compact" hide-details
@keydown.down.prevent="moveHighlight(1)" @keydown.up.prevent="moveHighlight(-1)"
@keydown.enter.prevent="onEnter" @keydown.esc="$emit('cancel')"
/>
</div>
<v-list v-if="hits.length || allowCreate" density="compact" class="fc-tag-autocomplete__list">
<v-text-field
v-model="query"
placeholder="Add tag (or kind:name — character/fandom/series)"
density="compact" hide-details
@keydown.down.prevent="moveHighlight(1)"
@keydown.up.prevent="moveHighlight(-1)"
@keydown.enter.prevent="onEnter"
@keydown.esc="$emit('cancel')"
/>
<v-list
v-if="hits.length || allowCreate"
density="compact" class="fc-tag-autocomplete__list"
>
<v-list-item
v-for="(h, idx) in hits" :key="h.id"
:active="idx === highlight" @click="onPick(h)"
@@ -26,7 +27,7 @@
<span v-if="h.fandom_name" class="text-caption"> {{ h.fandom_name }}</span>
</v-list-item-title>
<template #append>
<span class="text-caption">{{ h.image_count }}</span>
<span class="text-caption">{{ h.kind }}</span>
</template>
</v-list-item>
<v-list-item
@@ -34,9 +35,13 @@
@click="onCreate"
>
<template #prepend>
<v-icon size="small" :color="store.colorFor(kind)">{{ iconFor(kind) }}</v-icon>
<v-icon size="small" :color="store.colorFor(parsedKind)">
{{ iconFor(parsedKind) }}
</v-icon>
</template>
<v-list-item-title>Create "{{ query }}" as {{ kind }}</v-list-item-title>
<v-list-item-title>
Create "{{ parsedName }}" as {{ parsedKind }}
</v-list-item-title>
</v-list-item>
</v-list>
@@ -54,65 +59,95 @@ import FandomPicker from './FandomPicker.vue'
const emit = defineEmits(['pick-existing', 'pick-new', 'cancel'])
const store = useTagStore()
const kind = ref('general')
// Single text input; no kind dropdown. Client-side mirror of the
// backend's parse_kind_prefix lives below — kept in sync with
// KNOWN_KINDS in backend/app/utils/tag_prefix.py. The backend is the
// canonical parser; this mirror just powers the live Create-label
// preview ("Create 'Eric' as artist") and the autocomplete query.
const query = ref('')
const hits = ref([])
const highlight = ref(0)
const fandomDialog = ref(false)
let pendingNewName = null
const kindOptions = store.kindOptions()
const KNOWN_KINDS = new Set([
'character', 'fandom', 'series',
])
const KIND_ICONS = {
general: 'mdi-tag', artist: 'mdi-palette', character: 'mdi-account-circle',
general: 'mdi-tag', character: 'mdi-account-circle',
fandom: 'mdi-book-open-page-variant', series: 'mdi-bookshelf',
meta: 'mdi-cog-outline', rating: 'mdi-shield-check-outline'
}
function iconFor(k) { return KIND_ICONS[k] || 'mdi-tag' }
function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
const parsed = computed(() => {
const raw = query.value.trim()
if (raw.includes(':')) {
const idx = raw.indexOf(':')
const prefix = raw.slice(0, idx).toLowerCase()
if (KNOWN_KINDS.has(prefix)) {
return { kind: prefix, name: raw.slice(idx + 1).trim() }
}
}
return { kind: 'general', name: raw }
})
const parsedKind = computed(() => parsed.value.kind)
const parsedName = computed(() => parsed.value.name)
let debounceId = null
watch([query, kind], () => {
watch(query, () => {
highlight.value = 0
if (debounceId) clearTimeout(debounceId)
debounceId = setTimeout(async () => {
const q = query.value.trim()
const q = parsedName.value
if (!q) { hits.value = []; return }
hits.value = await store.autocomplete(q, kind.value, 10)
// Autocomplete across ALL kinds. When the user typed a prefix the
// matches list is naturally narrower because the parsed name is
// shorter; we don't filter server-side by kind.
hits.value = await store.autocomplete(q, null, 10)
}, 200)
})
const allowCreate = computed(() => {
const q = query.value.trim()
return q && !hits.value.some(h => h.name.toLowerCase() === q.toLowerCase() && h.kind === kind.value)
const q = parsedName.value
if (!q) return false
return !hits.value.some(h =>
h.name.toLowerCase() === q.toLowerCase() && h.kind === parsedKind.value,
)
})
function moveHighlight(delta) {
function moveHighlight (delta) {
const total = hits.value.length + (allowCreate.value ? 1 : 0)
if (total === 0) return
highlight.value = (highlight.value + delta + total) % total
}
function onPick(hit) { emit('pick-existing', hit); reset() }
function onPick (hit) { emit('pick-existing', hit); reset() }
function onCreate() {
const name = query.value.trim()
if (kind.value === 'character') {
// Character requires a fandom — open the picker.
function onCreate () {
const name = parsedName.value
const kind = parsedKind.value
if (kind === 'character') {
pendingNewName = name
fandomDialog.value = true
return
}
emit('pick-new', { name, kind: kind.value, fandom_id: null })
// Pass explicit kind here; the backend accepts both shapes. Passing
// it makes the parsed kind preview match the actual server outcome
// for users who didn't use a prefix (general goes through cleanly).
emit('pick-new', { name, kind, fandom_id: null })
reset()
}
function onFandomChosen(fandom) {
function onFandomChosen (fandom) {
fandomDialog.value = false
emit('pick-new', { name: pendingNewName, kind: 'character', fandom_id: fandom.id })
emit('pick-new', {
name: pendingNewName, kind: 'character', fandom_id: fandom.id,
})
pendingNewName = null
reset()
}
function onEnter() {
function onEnter () {
if (highlight.value < hits.value.length) {
onPick(hits.value[highlight.value])
} else if (allowCreate.value) {
@@ -120,7 +155,7 @@ function onEnter() {
}
}
function reset() { query.value = ''; hits.value = []; highlight.value = 0 }
function reset () { query.value = ''; hits.value = []; highlight.value = 0 }
</script>
<style scoped>
+172 -122
View File
@@ -1,10 +1,17 @@
<template>
<v-card class="fc-post-card" variant="outlined">
<v-card
class="fc-post-card"
variant="outlined"
tabindex="0"
@click="onCardClick"
@keydown.enter="onCardClick"
>
<div class="fc-post-card__head">
<v-chip size="x-small" variant="tonal">{{ post.source.platform }}</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-card__artist"
@click.stop
>{{ post.artist.name }}</RouterLink>
<span class="fc-post-card__date" :title="absoluteDate">{{ relativeDate }}</span>
<v-spacer />
@@ -13,72 +20,74 @@
:href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="x-small" variant="text"
:aria-label="`open original post on ${post.source.platform}`"
@click.stop
/>
</div>
<h3 v-if="post.post_title" class="fc-post-card__title">{{ post.post_title }}</h3>
<div class="fc-post-card__body">
<div class="fc-post-card__media">
<template v-if="post.thumbnails?.length">
<div class="fc-post-card__hero">
<img :src="hero.thumbnail_url" :alt="`hero thumbnail`" loading="lazy" />
</div>
<div v-if="rail.length" class="fc-post-card__rail">
<div v-for="t in rail" :key="t.image_id" class="fc-post-card__rail-cell">
<img :src="t.thumbnail_url" :alt="`thumbnail`" loading="lazy" />
</div>
<div v-if="moreCount > 0" class="fc-post-card__rail-more">
+{{ moreCount }}
</div>
</div>
</template>
<PostEmptyThumbs v-else />
</div>
<div v-if="descriptionToShow" class="fc-post-card__desc">
<span class="fc-post-card__desc-text">{{ descriptionToShow }}</span>
<button
v-if="post.description_truncated && !expanded"
class="fc-post-card__more" type="button"
@click="expand"
>Show more</button>
<button
v-if="expanded"
class="fc-post-card__more" type="button"
@click="expanded = false"
>Show less</button>
</div>
<div class="fc-post-card__text">
<h3 v-if="post.post_title" class="fc-post-card__title">
{{ post.post_title }}
</h3>
<h3 v-else class="fc-post-card__title fc-post-card__title--missing">
Post {{ post.external_post_id }}
</h3>
<div v-if="post.thumbnails.length" class="fc-post-card__thumbs">
<RouterLink
v-for="t in post.thumbnails"
:key="t.image_id"
:to="{ path: '/gallery', query: { post_id: post.id } }"
class="fc-post-card__thumb"
>
<v-img
:src="t.thumbnail_url" cover :alt="`thumbnail`"
width="96" height="96" class="fc-post-card__thumb-img"
/>
</RouterLink>
<RouterLink
v-if="post.thumbnails_more > 0"
:to="{ path: '/gallery', query: { post_id: post.id } }"
class="fc-post-card__more-thumbs"
>+{{ post.thumbnails_more }} more</RouterLink>
</div>
<p v-if="post.description_plain" class="fc-post-card__desc">
{{ post.description_plain }}
</p>
<p v-else class="fc-post-card__desc fc-post-card__desc--missing">
(no description)
</p>
<div v-if="post.attachments.length" class="fc-post-card__attachments">
<a
v-for="att in post.attachments"
:key="att.id"
:href="att.download_url"
download
class="fc-post-card__att"
>
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
<span class="fc-post-card__att-name">{{ att.original_filename }}</span>
<span class="fc-post-card__att-size">({{ formatBytes(att.size_bytes) }})</span>
</a>
<div v-if="post.attachments?.length" class="fc-post-card__atts">
<v-icon size="small" class="fc-post-card__att-icon">mdi-paperclip</v-icon>
{{ post.attachments.length }} attachment{{ post.attachments.length === 1 ? '' : 's' }}
</div>
</div>
</div>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { usePostsStore } from '../../stores/posts.js'
import { computed } from 'vue'
import { RouterLink } from 'vue-router'
import { usePostModalStore } from '../../stores/postModal.js'
import PostEmptyThumbs from './PostEmptyThumbs.vue'
const props = defineProps({
post: { type: Object, required: true },
})
const postsStore = usePostsStore()
const postModal = usePostModalStore()
const expanded = ref(false)
const fullDescription = ref(null)
const hero = computed(() => props.post.thumbnails?.[0])
const rail = computed(() => (props.post.thumbnails || []).slice(1, 4))
const moreCount = computed(() => {
const more = props.post.thumbnails_more || 0
const railLen = rail.value.length
// If feed returned more than (1 hero + 3 rail = 4), extra spills into "+N".
const extraShown = Math.max(0, (props.post.thumbnails?.length || 0) - 1 - railLen)
return more + extraShown
})
const sortDateIso = computed(() => props.post.post_date || props.post.downloaded_at)
const absoluteDate = computed(() => new Date(sortDateIso.value).toLocaleString())
@@ -92,25 +101,8 @@ const relativeDate = computed(() => {
return new Date(sortDateIso.value).toLocaleDateString()
})
const descriptionToShow = computed(() => {
if (expanded.value && fullDescription.value) return fullDescription.value
return props.post.description_plain
})
async function expand() {
if (!fullDescription.value) {
const detail = await postsStore.getPostFull(props.post.id)
fullDescription.value = detail?.description_full || props.post.description_plain
}
expanded.value = true
}
function formatBytes(n) {
if (!n) return '0 B'
if (n < 1024) return `${n} B`
if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)} KB`
if (n < 1024 * 1024 * 1024) return `${(n / 1024 / 1024).toFixed(1)} MB`
return `${(n / 1024 / 1024 / 1024).toFixed(1)} GB`
function onCardClick () {
postModal.open(props.post)
}
</script>
@@ -118,13 +110,25 @@ function formatBytes(n) {
.fc-post-card {
padding: 1rem;
margin-bottom: 1rem;
cursor: pointer;
container-type: inline-size;
transition: border-color 0.15s ease;
}
.fc-post-card:hover {
border-color: rgb(var(--v-theme-accent));
}
.fc-post-card:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 2px;
}
.fc-post-card__head {
display: flex;
align-items: center;
gap: 0.6rem;
font-size: 0.85rem;
font-size: 0.8rem;
color: rgb(var(--v-theme-on-surface-variant));
margin-bottom: 12px;
}
.fc-post-card__artist {
color: rgb(var(--v-theme-on-surface));
@@ -133,72 +137,118 @@ function formatBytes(n) {
}
.fc-post-card__artist:hover { color: rgb(var(--v-theme-accent)); }
.fc-post-card__date { white-space: nowrap; }
.fc-post-card__title {
font-size: 1.05rem;
margin: 0.6rem 0 0.4rem;
}
.fc-post-card__desc {
white-space: pre-wrap;
font-size: 0.92rem;
color: rgb(var(--v-theme-on-surface));
margin-bottom: 0.6rem;
}
.fc-post-card__more {
background: none;
border: none;
color: rgb(var(--v-theme-accent));
cursor: pointer;
padding: 0 0.25rem;
font-size: inherit;
}
.fc-post-card__thumbs {
.fc-post-card__body {
display: flex;
flex-wrap: wrap;
gap: 0.4rem;
align-items: center;
margin: 0.5rem 0;
flex-direction: column;
gap: 16px;
}
.fc-post-card__thumb { line-height: 0; }
.fc-post-card__thumb-img {
border-radius: 4px;
@container (min-width: 800px) {
.fc-post-card__body {
flex-direction: row;
gap: 24px;
}
.fc-post-card__media {
flex: 0 0 50%;
}
.fc-post-card__text {
flex: 1 1 0;
min-width: 0;
}
}
.fc-post-card__hero {
width: 100%;
aspect-ratio: 16 / 10;
overflow: hidden;
border-radius: 6px;
}
.fc-post-card__more-thumbs {
display: inline-flex;
.fc-post-card__hero img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
.fc-post-card__rail {
display: flex;
gap: 6px;
margin-top: 6px;
}
.fc-post-card__rail-cell {
width: 80px;
height: 80px;
overflow: hidden;
border-radius: 4px;
}
.fc-post-card__rail-cell img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
.fc-post-card__rail-more {
width: 80px;
height: 80px;
display: flex;
align-items: center;
justify-content: center;
width: 96px;
height: 96px;
border: 1px dashed rgb(var(--v-theme-on-surface-variant));
border-radius: 4px;
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: none;
font-size: 0.85rem;
}
.fc-post-card__more-thumbs:hover {
color: rgb(var(--v-theme-accent));
border-color: rgb(var(--v-theme-accent));
.fc-post-card__title {
font-family: 'Fraunces', Georgia, serif;
font-size: 18px;
font-weight: 500;
margin: 0 0 8px 0;
color: rgb(var(--v-theme-on-surface));
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
.fc-post-card__attachments {
display: flex;
flex-wrap: wrap;
gap: 0.4rem;
margin-top: 0.6rem;
.fc-post-card__title--missing {
font-style: italic;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__att {
@container (min-width: 800px) {
.fc-post-card__title {
font-size: 20px;
-webkit-line-clamp: 1;
white-space: nowrap;
text-overflow: ellipsis;
}
}
.fc-post-card__desc {
font-size: 0.9rem;
line-height: 1.5;
color: rgb(var(--v-theme-on-surface));
margin: 0 0 12px 0;
display: -webkit-box;
-webkit-line-clamp: 3;
-webkit-box-orient: vertical;
overflow: hidden;
}
.fc-post-card__desc--missing {
font-style: italic;
color: rgb(var(--v-theme-on-surface-variant));
}
@container (min-width: 800px) {
.fc-post-card__desc {
-webkit-line-clamp: 5;
}
}
.fc-post-card__atts {
display: inline-flex;
align-items: center;
gap: 0.35rem;
padding: 0.2rem 0.5rem;
border: 1px solid rgb(var(--v-theme-on-surface-variant));
border-radius: 999px;
color: rgb(var(--v-theme-on-surface));
text-decoration: none;
font-size: 0.82rem;
gap: 6px;
font-size: 0.85rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-card__att:hover {
color: rgb(var(--v-theme-accent));
border-color: rgb(var(--v-theme-accent));
}
.fc-post-card__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-post-card__att-icon { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,35 @@
<template>
<div class="fc-post-empty">
<v-icon size="48" class="fc-post-empty__icon">mdi-image-off-outline</v-icon>
<div class="fc-post-empty__text">No images attached to this post</div>
</div>
</template>
<script setup>
// No props — pure presentational placeholder.
</script>
<style scoped>
.fc-post-empty {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 12px;
padding: 32px 16px;
border: 1px dashed rgb(var(--v-theme-on-surface-variant));
border-radius: 8px;
background: rgba(20, 23, 26, 0.3);
width: 100%;
height: 100%;
min-height: 200px;
}
.fc-post-empty__icon {
color: rgb(var(--v-theme-on-surface-variant));
opacity: 0.6;
}
.fc-post-empty__text {
color: rgb(var(--v-theme-on-surface-variant));
font-size: 0.9rem;
}
</style>
@@ -0,0 +1,63 @@
<template>
<div class="fc-post-grid">
<button
v-for="(t, idx) in thumbnails"
:key="t.image_id"
type="button"
class="fc-post-grid__cell"
:aria-label="`Open image ${idx + 1} of ${thumbnails.length}`"
@click="openImage(t.image_id, idx)"
>
<img
:src="t.thumbnail_url"
:alt="`thumbnail ${idx + 1}`"
loading="lazy"
/>
</button>
</div>
</template>
<script setup>
import { computed } from 'vue'
import { useModalStore } from '../../stores/modal.js'
const props = defineProps({
thumbnails: { type: Array, required: true }, // [{ image_id, thumbnail_url, ... }]
})
const modal = useModalStore()
const imageIds = computed(() => props.thumbnails.map(t => t.image_id))
function openImage (id, idx) {
modal.open(id, { postImageIds: imageIds.value, initialIndex: idx })
}
</script>
<style scoped>
.fc-post-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 6px;
}
.fc-post-grid__cell {
aspect-ratio: 4 / 3;
overflow: hidden;
border-radius: 4px;
cursor: pointer;
border: 0;
padding: 0;
background: rgb(var(--v-theme-background));
transition: transform 0.15s ease, box-shadow 0.15s ease;
}
.fc-post-grid__cell:hover {
transform: scale(1.02);
box-shadow: 0 0 0 2px rgb(var(--v-theme-accent));
}
.fc-post-grid__cell img {
width: 100%;
height: 100%;
object-fit: cover;
display: block;
}
</style>
+211
View File
@@ -0,0 +1,211 @@
<template>
<v-dialog
:model-value="store.isOpen"
max-width="1100"
scrollable
@update:model-value="(v) => { if (!v) close() }"
>
<v-card v-if="store.currentPost" class="fc-post-modal">
<div class="fc-post-modal__head">
<v-chip size="x-small" variant="tonal">{{ post.source.platform }}</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-modal__artist"
@click="close"
>{{ post.artist.name }}</RouterLink>
<span class="fc-post-modal__date" :title="absoluteDate">
{{ relativeDate }}
</span>
<span v-if="post.thumbnails?.length" class="fc-post-modal__meta">
· {{ post.thumbnails.length }} image{{ post.thumbnails.length === 1 ? '' : 's' }}
</span>
<span v-if="post.attachments?.length" class="fc-post-modal__meta">
· {{ post.attachments.length }} attachment{{ post.attachments.length === 1 ? '' : 's' }}
</span>
<v-spacer />
<v-btn
v-if="post.post_url"
:href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="small" variant="text"
:aria-label="`open original post on ${post.source.platform}`"
/>
<v-btn
icon="mdi-close" size="small" variant="text"
aria-label="Close"
@click="close"
/>
</div>
<v-card-text class="fc-post-modal__body">
<section v-if="post.thumbnails?.length" class="fc-post-modal__sec">
<PostImageGrid :thumbnails="post.thumbnails" />
<div v-if="!store.detailLoaded" class="fc-post-modal__loading-hint">
Loading full image list…
</div>
</section>
<section v-if="post.post_title" class="fc-post-modal__sec">
<h2 class="fc-post-modal__title">{{ post.post_title }}</h2>
</section>
<section v-if="descriptionHtml" class="fc-post-modal__sec">
<div class="fc-post-modal__desc" v-html="descriptionHtml" />
</section>
<section v-if="post.attachments?.length" class="fc-post-modal__sec">
<h3 class="fc-post-modal__h3">Attachments</h3>
<div class="fc-post-modal__atts">
<a
v-for="att in post.attachments" :key="att.id"
:href="att.download_url" download
class="fc-post-modal__att"
>
<v-icon size="small" class="fc-post-modal__att-icon">mdi-paperclip</v-icon>
<span class="fc-post-modal__att-name">{{ att.original_filename }}</span>
<span class="fc-post-modal__att-size">({{ formatBytes(att.size_bytes) }})</span>
</a>
</div>
</section>
</v-card-text>
</v-card>
</v-dialog>
</template>
<script setup>
import { computed } from 'vue'
import { RouterLink } from 'vue-router'
import { usePostModalStore } from '../../stores/postModal.js'
import { sanitizeHtml } from '../../utils/htmlSanitize.js'
import PostImageGrid from './PostImageGrid.vue'
const store = usePostModalStore()
const post = computed(() => store.currentPost || {})
const sortDateIso = computed(() => post.value.post_date || post.value.downloaded_at)
const absoluteDate = computed(() => sortDateIso.value
? new Date(sortDateIso.value).toLocaleString()
: '')
const relativeDate = computed(() => {
if (!sortDateIso.value) return ''
const then = new Date(sortDateIso.value).getTime()
const diff = (Date.now() - then) / 1000
if (diff < 60) return `${Math.floor(diff)}s ago`
if (diff < 3600) return `${Math.floor(diff / 60)}m ago`
if (diff < 86400) return `${Math.floor(diff / 3600)}h ago`
if (diff < 86400 * 30) return `${Math.floor(diff / 86400)}d ago`
return new Date(sortDateIso.value).toLocaleDateString()
})
const descriptionHtml = computed(() => {
// Detail endpoint returns description_full as plain text (the
// existing service uses html_to_plain on the stored description).
// Render the plain text in <p> wrappers; the sanitizer below is
// defensive for the case where the backend ever switches to raw HTML.
const raw = post.value.description_full || post.value.description_plain
if (!raw) return ''
if (/[<>]/.test(raw)) return sanitizeHtml(raw)
const esc = raw
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
return esc
.split(/\n\s*\n/)
.map(p => `<p>${p.replace(/\n/g, '<br>')}</p>`)
.join('')
})
function close () {
store.close()
}
function formatBytes (n) {
if (!n) return '0 B'
if (n < 1024) return `${n} B`
if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)} KB`
if (n < 1024 * 1024 * 1024) return `${(n / 1024 / 1024).toFixed(1)} MB`
return `${(n / 1024 / 1024 / 1024).toFixed(1)} GB`
}
</script>
<style scoped>
.fc-post-modal {
background: rgb(var(--v-theme-surface));
max-height: 90vh;
display: flex;
flex-direction: column;
}
.fc-post-modal__head {
display: flex;
align-items: center;
gap: 10px;
padding: 12px 16px;
font-size: 0.85rem;
color: rgb(var(--v-theme-on-surface-variant));
border-bottom: 1px solid rgb(var(--v-theme-on-surface-variant) / 0.15);
}
.fc-post-modal__artist {
color: rgb(var(--v-theme-on-surface));
text-decoration: none;
font-weight: 600;
}
.fc-post-modal__artist:hover { color: rgb(var(--v-theme-accent)); }
.fc-post-modal__date { white-space: nowrap; }
.fc-post-modal__meta { white-space: nowrap; }
.fc-post-modal__body {
padding: 16px;
overflow-y: auto;
flex: 1 1 auto;
}
.fc-post-modal__sec {
margin-bottom: 24px;
}
.fc-post-modal__sec:last-child { margin-bottom: 0; }
.fc-post-modal__title {
font-family: 'Fraunces', Georgia, serif;
font-size: 24px;
font-weight: 500;
margin: 0;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-modal__h3 {
font-family: 'Fraunces', Georgia, serif;
font-size: 16px;
font-weight: 500;
margin: 0 0 8px 0;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-modal__desc {
font-size: 0.95rem;
line-height: 1.55;
color: rgb(var(--v-theme-on-surface));
}
.fc-post-modal__desc :deep(p) { margin: 0 0 12px 0; }
.fc-post-modal__desc :deep(a) { color: rgb(var(--v-theme-accent)); }
.fc-post-modal__loading-hint {
margin-top: 8px;
font-size: 0.8rem;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-post-modal__atts {
display: flex;
flex-wrap: wrap;
gap: 8px;
}
.fc-post-modal__att {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
border: 1px solid rgb(var(--v-theme-on-surface-variant));
border-radius: 999px;
color: rgb(var(--v-theme-on-surface));
text-decoration: none;
font-size: 0.85rem;
}
.fc-post-modal__att:hover {
color: rgb(var(--v-theme-accent));
border-color: rgb(var(--v-theme-accent));
}
.fc-post-modal__att-size { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -34,11 +34,18 @@
<template v-else-if="manifest?.installed">
<div class="fc-ext-install mt-3">
<!-- Install button: direct :href anchor click (no programmatic
window.location.assign). Firefox's XPI-install gesture
requires a user-clicked anchor pointing at an
application/x-xpinstall response; programmatic navigation
sometimes triggered nothing instead of the install dialog
(operator-flagged 2026-05-26). No `download` attribute —
that would force a save dialog instead of install. -->
<v-btn
v-if="isFirefox"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-firefox"
@click="installXpi"
:href="manifest.latest_url"
>Install Firefox extension</v-btn>
<v-btn
@@ -103,6 +110,7 @@
<script setup>
import { computed, onMounted, ref } from 'vue'
import { useApi } from '../../composables/useApi.js'
import { copyText } from '../../utils/clipboard.js'
const api = useApi()
@@ -146,11 +154,6 @@ async function loadKey() {
}
}
function installXpi() {
if (!manifest.value?.latest_url) return
window.location.assign(manifest.value.latest_url)
}
async function rotateKey() {
rotating.value = true
try {
@@ -170,7 +173,7 @@ async function rotateKey() {
async function copy(text, label) {
try {
await navigator.clipboard.writeText(text)
await copyText(text)
window.__fcToast?.({ text: `${label} copied.`, type: 'success' })
} catch (e) {
window.__fcToast?.({ text: `Copy failed: ${e.message}`, type: 'error' })
@@ -17,6 +17,12 @@
>
Retry failed
</v-btn>
<v-btn
variant="text" rounded="pill" size="small" color="warning"
:disabled="!hasStuck" @click="onClearStuckOpen"
>
Clear stuck
</v-btn>
<v-btn
variant="text" rounded="pill" size="small" color="error"
@click="onClearOpen"
@@ -39,9 +45,11 @@
<template #item.size_bytes="{ item }">{{ formatBytes(item.size_bytes) }}</template>
<template #item.created_at="{ item }">{{ formatDate(item.created_at) }}</template>
<template #item.error="{ item }">
<span v-if="item.error" :title="item.error" class="text-caption">
{{ shorten(item.error, 60) }}
</span>
<button
v-if="item.error" type="button" class="fc-err-link text-caption"
@click="openError(`Task ${item.id} failed`, item.error)"
title="Click for full error"
>{{ shorten(item.error, 60) }}</button>
</template>
</v-data-table-virtual>
<div v-if="store.hasMore" class="d-flex justify-center py-3">
@@ -69,17 +77,62 @@
</v-card-actions>
</v-card>
</v-dialog>
<v-dialog v-model="clearStuckDialog" max-width="480">
<v-card>
<v-card-title>Clear stuck tasks</v-card-title>
<v-card-text>
<v-alert type="warning" variant="tonal" density="compact" class="mb-3">
Force every <strong>pending / queued / processing</strong> task to
<strong>failed</strong> and finalize any active batch that
has no remaining work. Use this when the automatic recovery
sweep keeps re-queueing the same row (e.g., corrupt file in
an autoretry loop, or worker model missing).
</v-alert>
<p class="text-body-2">
Tasks remain in the database with status=<code>failed</code>;
click <em>Retry failed</em> once the underlying cause is
resolved to re-queue them.
</p>
</v-card-text>
<v-card-actions>
<v-spacer />
<v-btn @click="clearStuckDialog = false">Cancel</v-btn>
<v-btn color="warning" rounded="pill" @click="onClearStuckConfirm">Clear stuck</v-btn>
</v-card-actions>
</v-card>
</v-dialog>
<ErrorDetailModal
v-model="showErrorModal"
:title="errorModalTitle"
:message="errorModalMessage"
/>
</v-card>
</template>
<script setup>
import { computed, ref } from 'vue'
import { useImportStore } from '../../stores/import.js'
import ErrorDetailModal from '../common/ErrorDetailModal.vue'
const store = useImportStore()
const statusFilter = ref(null)
const clearDialog = ref(false)
// Click-to-open modal for full error text (operator-flagged 2026-05-26
// — the prior :title="..." tooltip cramped multi-line SQLAlchemy
// tracebacks into an unusable popup with no copy-paste affordance).
const showErrorModal = ref(false)
const errorModalTitle = ref('')
const errorModalMessage = ref('')
function openError(title, message) {
errorModalTitle.value = title
errorModalMessage.value = message || ''
showErrorModal.value = true
}
const clearAgeDays = ref(7)
const clearStuckDialog = ref(false)
const statusOptions = [
{ title: 'All', value: null },
@@ -100,6 +153,9 @@ const headers = [
]
const hasFailed = computed(() => store.tasks.some(t => t.status === 'failed'))
const hasStuck = computed(() => store.tasks.some(
t => t.status === 'pending' || t.status === 'queued' || t.status === 'processing'
))
function statusColor(s) {
return {
@@ -138,4 +194,27 @@ async function onClearConfirm() {
await store.clearCompleted(clearAgeDays.value)
clearDialog.value = false
}
function onClearStuckOpen() { clearStuckDialog.value = true }
async function onClearStuckConfirm() {
await store.clearStuck()
clearStuckDialog.value = false
}
</script>
<style scoped>
.fc-err-link {
/* Truncated error preview as a clickable button — opens
ErrorDetailModal with the full text. Inherits the row's font
sizing so it doesn't visually drift from the prior tooltip-bearing
span. */
color: rgb(var(--v-theme-error, 220 80 80));
background: transparent;
border: 0;
padding: 0;
font: inherit;
text-align: left;
text-decoration: underline dotted;
cursor: pointer;
}
.fc-err-link:hover { text-decoration: underline; }
</style>
@@ -2,31 +2,71 @@
<v-card>
<v-card-title>Trigger scan</v-card-title>
<v-card-text>
<div v-if="store.activeBatch" class="d-flex align-center" style="gap: 12px;">
<div v-if="store.activeBatch" class="d-flex align-center mb-3" style="gap: 12px;">
<v-progress-circular
indeterminate color="accent" size="20"
/>
<span>
Scanning {{ store.activeBatch.source_path }}
{{ store.activeBatch.scan_mode === 'deep' ? 'Deep scanning' : 'Scanning' }}
{{ store.activeBatch.source_path || '/import' }}
imported {{ store.activeBatch.imported }},
<template v-if="store.activeBatch.scan_mode === 'deep'">
refreshed {{ store.activeBatch.refreshed || 0 }},
</template>
skipped {{ store.activeBatch.skipped }},
failed {{ store.activeBatch.failed }} /
{{ store.activeBatch.total_files }} files
</span>
<v-spacer />
<v-btn
variant="text" rounded="pill" size="small" color="warning"
:loading="clearing" @click="onClearStuck"
>
Clear stuck
</v-btn>
</div>
<div v-else>
<p class="text-body-2 mb-3">
Run a quick scan of the import directory. Deep scan (pHash dedup,
archives) lands in FC-2d.
</p>
<v-btn color="primary" rounded="pill" @click="trigger" :loading="busy">
<p class="text-body-2 mb-3">
<span v-if="!store.activeBatch">
<strong>Quick scan</strong> walks <code>/import</code> and enqueues
new files only.
<strong>Deep scan</strong> additionally re-walks already-imported
files so updated sidecar metadata (post title/date/attribution) and
previously-NULL phashes / artist links get refreshed. Use after
bulk-downloading fresh sidecars for existing content. Both modes
route non-media + sidecar pairs through PostAttachment capture.
</span>
<span v-else>
An active batch is in progress. Wait for it to finish, or click
<em>Clear stuck</em> above if it has been wedged with no
measurable progress.
</span>
</p>
<div class="d-flex flex-wrap" style="gap: 12px;">
<v-btn
color="primary" rounded="pill"
:disabled="!!store.activeBatch"
:loading="busy === 'quick'"
@click="trigger('quick')"
>
<v-icon start>mdi-magnify-scan</v-icon>
Quick scan
</v-btn>
<v-alert v-if="store.triggerError" type="error" variant="tonal" class="mt-3" closable>
{{ store.triggerError }}
</v-alert>
<v-btn
color="secondary" rounded="pill" variant="tonal"
:disabled="!!store.activeBatch"
:loading="busy === 'deep'"
@click="trigger('deep')"
>
<v-icon start>mdi-magnify-plus-outline</v-icon>
Deep scan
</v-btn>
</div>
<v-alert v-if="store.triggerError" type="error" variant="tonal" class="mt-3" closable>
{{ store.triggerError }}
</v-alert>
</v-card-text>
</v-card>
</template>
@@ -36,10 +76,22 @@ import { ref } from 'vue'
import { useImportStore } from '../../stores/import.js'
const store = useImportStore()
const busy = ref(false)
const busy = ref(null)
const clearing = ref(false)
async function trigger() {
busy.value = true
try { await store.triggerScan() } catch {} finally { busy.value = false }
async function trigger(mode) {
busy.value = mode
try { await store.triggerScan(mode) } catch {} finally { busy.value = null }
}
async function onClearStuck() {
clearing.value = true
try {
await store.clearStuck()
} catch {
// store surfaces error via triggerError if needed
} finally {
clearing.value = false
}
}
</script>
@@ -1,20 +1,23 @@
<template>
<div class="fc-maint">
<p class="text-body-2 mb-4">
Machine-assisted tagging controls. Backfill and centroid recompute run
nightly automatically; the allowlist auto-applies accepted tags to new
and existing images.
Operational backfills and tagging controls. The ML backfill and centroid
recompute run nightly automatically; the allowlist auto-applies accepted
tags to new and existing images. Use the cards below to trigger a
one-off pass.
</p>
<div class="fc-maint__grid">
<MLBackfillCard />
<CentroidRecomputeCard />
<ThumbnailBackfillCard />
</div>
<MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" />
<AliasTable class="mt-4" />
<BackupCard class="mt-6" />
<TagMaintenanceCard class="mt-6" />
<BrowserExtensionCard class="mt-6" />
<!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) it
operates on the existing library which fits the Cleanup-tab
theme, and clusters with the other audit cards. -->
<LegacyMigrationCard class="mt-6" />
</div>
</template>
@@ -22,12 +25,11 @@
<script setup>
import MLBackfillCard from './MLBackfillCard.vue'
import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue'
import BackupCard from './BackupCard.vue'
import TagMaintenanceCard from './TagMaintenanceCard.vue'
import BrowserExtensionCard from './BrowserExtensionCard.vue'
import LegacyMigrationCard from './LegacyMigrationCard.vue'
</script>
@@ -59,8 +59,12 @@
<td>{{ r.queue }}</td>
<td><code>{{ shortTaskName(r.task_name) }}</code></td>
<td class="fc-tabular">{{ r.target_id ?? '—' }}</td>
<td class="fc-err" :title="r.error_message">
{{ r.error_type }}
<td>
<button
type="button" class="fc-err-link"
@click="openError(r)"
:title="'Click for full error'"
>{{ r.error_type }}</button>
</td>
</tr>
<tr v-if="!filteredFailures.length">
@@ -138,6 +142,11 @@
</div>
</v-card-text>
</v-card>
<ErrorDetailModal
v-model="showErrorModal"
:row="errorModalRow"
/>
</div>
</template>
@@ -145,8 +154,21 @@
import { computed, onMounted, onUnmounted, ref } from 'vue'
import { useSystemActivityStore } from '../../stores/systemActivity.js'
import ErrorDetailModal from '../common/ErrorDetailModal.vue'
import QueuesTable from './QueuesTable.vue'
// Click-to-open modal for full error text. Replaces the unusable
// :title="..." tooltip (operator-flagged 2026-05-26: SQLAlchemy
// rollback + traceback content rendered as a cramped browser tooltip
// you couldn't copy from or scroll within).
const showErrorModal = ref(false)
const errorModalRow = ref(null)
function openError(row) {
errorModalRow.value = row
showErrorModal.value = true
}
const store = useSystemActivityStore()
const filterQueue = ref(null)
@@ -276,5 +298,18 @@ function formatRelative(iso) {
font-feature-settings: 'tnum';
}
.fc-err { color: rgb(var(--v-theme-error, 220 80 80)); }
.fc-err-link {
/* Styled as a text-only button so the error_type cell stays
visually identical to the prior tooltip-bearing row, but is
now a real clickable target with hover affordance. */
color: rgb(var(--v-theme-error, 220 80 80));
background: transparent;
border: 0;
padding: 0;
font: inherit;
text-decoration: underline dotted;
cursor: pointer;
}
.fc-err-link:hover { text-decoration: underline; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -0,0 +1,30 @@
<template>
<v-card>
<v-card-title>Thumbnail backfill</v-card-title>
<v-card-text>
<p class="text-body-2 mb-3">
Scan the library for images with no thumbnail, or whose thumbnail file
is missing or corrupt on disk. Repair candidates are re-enqueued for
thumbnail generation. Safe to re-run.
</p>
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
<v-icon start>mdi-image-refresh</v-icon> Run backfill now
</v-btn>
<span v-if="done" class="ml-3 text-caption">Enqueued.</span>
</v-card-text>
</v-card>
</template>
<script setup>
import { ref } from 'vue'
import { useThumbnailsStore } from '../../stores/thumbnails.js'
const store = useThumbnailsStore()
const busy = ref(false)
const done = ref(false)
async function run () {
busy.value = true
try { await store.triggerBackfill(); done.value = true }
catch (e) { window.__fcToast?.({ text: e.message, type: 'error' }) }
finally { busy.value = false }
}
</script>
+13 -3
View File
@@ -1,6 +1,7 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { useApi } from '../composables/useApi.js'
import { usePostsStore } from './posts.js'
const PAGE = 60
@@ -15,7 +16,11 @@ export const useArtistStore = defineStore('artist', () => {
const notFound = ref(false)
let started = false
async function load(slug) {
async function load (slug) {
// Cross-artist reset: clear this store AND the posts store so the new
// artist doesn't briefly render with the previous artist's content
// when the user is on the Posts tab. (Gallery tab uses this artist
// store's own images list — cleared above.)
overview.value = null
images.value = []
nextCursor.value = null
@@ -23,6 +28,7 @@ export const useArtistStore = defineStore('artist', () => {
started = false
error.value = null
loading.value = true
usePostsStore().$reset?.()
try {
overview.value = await api.get(`/api/artist/${encodeURIComponent(slug)}`)
await loadMoreImages(slug)
@@ -34,7 +40,7 @@ export const useArtistStore = defineStore('artist', () => {
}
}
async function loadMoreImages(slug) {
async function loadMoreImages (slug) {
if (imagesLoading.value) return
if (started && nextCursor.value === null) return
imagesLoading.value = true
@@ -55,9 +61,13 @@ export const useArtistStore = defineStore('artist', () => {
}
const hasMoreImages = computed(() => !started || nextCursor.value !== null)
const postCount = computed(() => overview.value?.post_count ?? null)
const imageCount = computed(() => overview.value?.image_count ?? null)
const lastAdded = computed(() => overview.value?.date_range?.max ?? null)
return {
overview, images, loading, imagesLoading, error, notFound,
hasMoreImages, load, loadMoreImages
hasMoreImages, postCount, imageCount, lastAdded,
load, loadMoreImages,
}
})
+72
View File
@@ -0,0 +1,72 @@
import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
export const useCleanupStore = defineStore('cleanup', () => {
const api = useApi()
// Defaults sourced from ImportSettings on mount. Cards pre-fill from
// these so the common case ("apply current import filters
// retroactively") is one click; operator can override per-audit.
const defaults = ref({
min_width: 0,
min_height: 0,
transparency_threshold: 0.9,
single_color_threshold: 0.95,
single_color_tolerance: 30,
})
const recentRuns = ref([])
async function loadDefaults() {
const s = await api.get('/api/settings/import')
defaults.value = {
min_width: s.min_width ?? 0,
min_height: s.min_height ?? 0,
transparency_threshold: s.transparency_threshold ?? 0.9,
single_color_threshold: s.single_color_threshold ?? 0.95,
single_color_tolerance: s.single_color_tolerance ?? 30,
}
}
async function previewMinDim(min_width, min_height) {
return await api.post('/api/cleanup/min-dimension/preview', {
body: { min_width, min_height },
})
}
async function deleteMinDim(min_width, min_height, confirm) {
return await api.post('/api/cleanup/min-dimension/delete', {
body: { min_width, min_height, confirm },
})
}
async function startAudit(rule, params) {
return await api.post('/api/cleanup/audit', { body: { rule, params } })
}
async function getAudit(id) {
return await api.get(`/api/cleanup/audit/${id}`)
}
async function loadHistory(limit = 20) {
const body = await api.get(`/api/cleanup/audit?limit=${limit}`)
recentRuns.value = body.runs
return body.runs
}
async function applyAudit(id, confirm) {
return await api.post(`/api/cleanup/audit/${id}/apply`, { body: { confirm } })
}
async function cancelAudit(id) {
return await api.post(`/api/cleanup/audit/${id}/cancel`)
}
return {
defaults, recentRuns,
loadDefaults,
previewMinDim, deleteMinDim,
startAudit, getAudit, loadHistory, applyAudit, cancelAudit,
}
})
+57 -3
View File
@@ -52,11 +52,58 @@ export const useImportStore = defineStore('import', () => {
}
}
async function triggerScan() {
async function triggerScan(mode = 'quick') {
if (!['quick', 'deep', 'verify'].includes(mode)) {
throw new Error(`unsupported scan mode: ${mode}`)
}
triggerError.value = null
try {
await api.post('/api/import/trigger', { body: { mode: 'quick' } })
await api.post('/api/import/trigger', { body: { mode } })
// Acknowledge immediately so the click isn't invisible. scan_directory
// can finalize the batch synchronously when every file in /import is
// already on a non-failed ImportTask (operator-flagged 2026-05-25:
// 233k existing tasks → all paths in skip-set → files_seen=0 →
// batch flashes 'running' for <100ms then 'complete' before the
// first refreshStatus() lands; UI never sees the active state).
const label = mode === 'deep'
? 'Deep scan triggered (re-applying sidecar metadata + filling NULL phash/artist on existing rows)'
: mode === 'verify' ? 'Library verify triggered' : 'Quick scan triggered'
window.__fcToast?.({ text: label, type: 'success' })
await refreshStatus()
// Re-poll twice over ~5s and produce an HONEST follow-up toast.
// Operator-flagged 2026-05-25: the prior "no new files" message was
// misleading because deep scan IS doing work (refresh) even when
// there are no new files to import. Surface the real workload count
// (imported + refreshed + queued) instead. For quick scan + zero
// queued work, fall back to "up to date" instead of the old
// implementation-detail-leaking message.
setTimeout(async () => {
await refreshStatus()
await loadTasks(true)
if (activeBatch.value || mode === 'verify') return
// Batch finalized quickly; figure out what actually happened.
// The task list was just refreshed; the freshest row(s) carry
// the batch outcome.
const batchId = tasks.value[0]?.batch_id
const sameBatch = batchId
? tasks.value.filter(t => t.batch_id === batchId)
: []
const refreshedCount = sameBatch.filter(t => t.status === 'complete' && t.result_image_id).length
const newImported = sameBatch.filter(t => t.status === 'complete' && t.result_image_id && !t.error).length
if (mode === 'deep' && sameBatch.length > 0) {
window.__fcToast?.({
text: `Deep scan finished — ${sameBatch.length} file(s) processed`,
type: 'info',
})
} else if (mode === 'quick' && sameBatch.length > 0) {
window.__fcToast?.({
text: `Quick scan finished — ${newImported} new file(s) queued`,
type: 'info',
})
} else {
window.__fcToast?.({ text: 'Library is up to date', type: 'info' })
}
}, 2000)
} catch (e) {
triggerError.value = e.message
window.__fcToast?.({ text: `Scan failed: ${e.message}`, type: 'error' })
@@ -92,6 +139,13 @@ export const useImportStore = defineStore('import', () => {
await loadTasks(true)
}
async function clearStuck() {
const body = await api.post('/api/import/clear-stuck')
await loadTasks(true)
await refreshStatus()
return body
}
const hasMore = computed(() => tasksNextCursor.value !== null)
return {
@@ -101,6 +155,6 @@ export const useImportStore = defineStore('import', () => {
triggerError,
loadSettings, patchSettings,
refreshStatus, triggerScan,
loadTasks, setStatusFilter, retryFailed, clearCompleted
loadTasks, setStatusFilter, retryFailed, clearCompleted, clearStuck
}
})
+66 -21
View File
@@ -6,14 +6,31 @@ export const useModalStore = defineStore('modal', () => {
const api = useApi()
const currentImageId = ref(null)
const current = ref(null) // full image detail from API
const current = ref(null)
const loading = ref(false)
const error = ref(null)
async function open(id) {
// Post-scoped cycle. When set, prev/next cycles within this array
// (used by PostModal's PostImageGrid clicks). When null, prev/next
// falls back to current.value.neighbors (the gallery-store-driven
// /api/gallery/image/<id> neighbors).
const postImageIds = ref(null)
const postImageIndex = ref(0)
async function open (id, opts = {}) {
currentImageId.value = id
loading.value = true
error.value = null
// Update post-scoped state if caller passed it; otherwise clear so
// the next open() from gallery context uses neighbors mode.
if (opts.postImageIds != null) {
postImageIds.value = opts.postImageIds
postImageIndex.value = opts.postImageIds.indexOf(id)
if (postImageIndex.value < 0) postImageIndex.value = 0
} else if (opts.clearPostScope !== false && postImageIds.value != null) {
postImageIds.value = null
postImageIndex.value = 0
}
try {
current.value = await api.get(`/api/gallery/image/${id}`)
} catch (e) {
@@ -24,41 +41,58 @@ export const useModalStore = defineStore('modal', () => {
}
}
async function close() {
async function close () {
currentImageId.value = null
current.value = null
error.value = null
postImageIds.value = null
postImageIndex.value = 0
}
async function goPrev() {
if (current.value && current.value.neighbors.prev_id) {
async function goPrev () {
if (postImageIds.value != null) {
if (postImageIndex.value > 0) {
const newIdx = postImageIndex.value - 1
const newId = postImageIds.value[newIdx]
postImageIndex.value = newIdx
await open(newId, { postImageIds: postImageIds.value })
}
return
}
if (current.value && current.value.neighbors?.prev_id) {
await open(current.value.neighbors.prev_id)
}
}
async function goNext() {
if (current.value && current.value.neighbors.next_id) {
async function goNext () {
if (postImageIds.value != null) {
if (postImageIndex.value < postImageIds.value.length - 1) {
const newIdx = postImageIndex.value + 1
const newId = postImageIds.value[newIdx]
postImageIndex.value = newIdx
await open(newId, { postImageIds: postImageIds.value })
}
return
}
if (current.value && current.value.neighbors?.next_id) {
await open(current.value.neighbors.next_id)
}
}
async function reloadTags() {
async function reloadTags () {
if (!currentImageId.value) return
const tags = await api.get(`/api/images/${currentImageId.value}/tags`)
if (current.value) current.value.tags = tags
}
async function removeTag(tagId) {
async function removeTag (tagId) {
if (!currentImageId.value) return
// Optimistic UI: remove locally first, restore on error.
const prev = current.value.tags
current.value.tags = current.value.tags.filter(t => t.id !== tagId)
try {
// FC-2b: removal also records a per-image rejection (suggestions/dismiss
// is the rejection-recording endpoint) so the allowlist maintenance
// task won't re-apply this tag to this image.
await api.delete(`/api/images/${currentImageId.value}/tags/${tagId}`)
await api.post(`/api/images/${currentImageId.value}/suggestions/dismiss`, {
body: { tag_id: tagId }
body: { tag_id: tagId },
})
} catch (e) {
current.value.tags = prev
@@ -67,25 +101,36 @@ export const useModalStore = defineStore('modal', () => {
}
}
async function addExistingTag(tagId) {
async function addExistingTag (tagId) {
if (!currentImageId.value) return
await api.post(`/api/images/${currentImageId.value}/tags`, {
body: { tag_id: tagId, source: 'manual' }
body: { tag_id: tagId, source: 'manual' },
})
await reloadTags()
}
async function createAndAdd({ name, kind, fandom_id = null }) {
async function createAndAdd ({ name, kind, fandom_id = null }) {
const tag = await api.post('/api/tags', { body: { name, kind, fandom_id } })
await addExistingTag(tag.id)
}
const isOpen = computed(() => currentImageId.value !== null)
const canPrev = computed(() => current.value?.neighbors?.prev_id != null)
const canNext = computed(() => current.value?.neighbors?.next_id != null)
const canPrev = computed(() => {
if (postImageIds.value != null) return postImageIndex.value > 0
return current.value?.neighbors?.prev_id != null
})
const canNext = computed(() => {
if (postImageIds.value != null) {
return postImageIndex.value < (postImageIds.value.length - 1)
}
return current.value?.neighbors?.next_id != null
})
return {
currentImageId, current, loading, error, isOpen, canPrev, canNext,
open, close, goPrev, goNext, reloadTags, removeTag, addExistingTag, createAndAdd
currentImageId, current, loading, error,
postImageIds, postImageIndex,
isOpen, canPrev, canNext,
open, close, goPrev, goNext,
reloadTags, removeTag, addExistingTag, createAndAdd,
}
})
+39
View File
@@ -0,0 +1,39 @@
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'
import { usePostsStore } from './posts.js'
export const usePostModalStore = defineStore('postModal', () => {
const postsStore = usePostsStore()
// Feed-shape on open; upgraded to detail shape (full description +
// uncapped thumbnails) once getPostFull resolves.
const currentPost = ref(null)
const detailLoaded = ref(false)
const error = ref(null)
const isOpen = computed(() => currentPost.value !== null)
async function open (post) {
currentPost.value = post
detailLoaded.value = false
error.value = null
try {
const detail = await postsStore.getPostFull(post.id)
if (currentPost.value?.id === post.id) {
currentPost.value = detail
detailLoaded.value = true
}
} catch (e) {
// Keep feed-shape data; PostModal can still render title + truncated
// description + the up-to-6 feed thumbnails. Just surface the error.
error.value = e.message
}
}
function close () {
currentPost.value = null
detailLoaded.value = false
error.value = null
}
return { currentPost, detailLoaded, error, isOpen, open, close }
})
+5 -7
View File
@@ -2,24 +2,22 @@ import { defineStore } from 'pinia'
import { ref } from 'vue'
import { useApi } from '../composables/useApi.js'
// `artist` retired in FC-2d-vii-c (provenance is its own axis), `meta` +
// `rating` retired by operator 2026-05-26 (alembic 0023 drops them from
// the enum). KIND_COLOR keeps `archive` + `post` so any legacy
// system-managed tag still renders with a neutral color.
const KIND_OPTIONS = [
{ value: 'general', label: 'General', icon: 'mdi-tag' },
{ value: 'artist', label: 'Artist', icon: 'mdi-palette' },
{ value: 'character', label: 'Character', icon: 'mdi-account-circle' },
{ value: 'fandom', label: 'Fandom', icon: 'mdi-book-open-page-variant' },
{ value: 'series', label: 'Series', icon: 'mdi-bookshelf' },
{ value: 'meta', label: 'Meta', icon: 'mdi-cog-outline' },
{ value: 'rating', label: 'Rating', icon: 'mdi-shield-check-outline' }
{ value: 'series', label: 'Series', icon: 'mdi-bookshelf' }
]
const KIND_COLOR = {
artist: 'accent',
character: 'info',
fandom: 'secondary',
series: 'warning',
general: 'on-surface',
meta: 'on-surface',
rating: 'on-surface',
archive: 'on-surface',
post: 'on-surface'
}
+12
View File
@@ -0,0 +1,12 @@
import { defineStore } from 'pinia'
import { useApi } from '../composables/useApi.js'
export const useThumbnailsStore = defineStore('thumbnails', () => {
const api = useApi()
async function triggerBackfill () {
await api.post('/api/thumbnails/backfill')
}
return { triggerBackfill }
})
+35
View File
@@ -0,0 +1,35 @@
// Clipboard write that works on plain-HTTP self-hosted deployments.
//
// navigator.clipboard is gated by the browser's Secure Context restriction
// (HTTPS or localhost only). FabledCurator runs over plain HTTP per the
// homelab posture, so the modern API is undefined in production. We fall
// back to the legacy execCommand('copy') path via a temporary off-screen
// textarea — wide browser support, no HTTPS requirement.
export async function copyText (text) {
const str = text == null ? '' : String(text)
if (typeof navigator !== 'undefined' && navigator.clipboard?.writeText) {
try {
await navigator.clipboard.writeText(str)
return
} catch {
// Fall through to the legacy path. Some browsers throw even when
// the API exists (permission denied, focus loss, etc.).
}
}
const ta = document.createElement('textarea')
ta.value = str
ta.setAttribute('readonly', '')
ta.style.position = 'fixed'
ta.style.top = '0'
ta.style.left = '0'
ta.style.opacity = '0'
ta.style.pointerEvents = 'none'
document.body.appendChild(ta)
ta.select()
ta.setSelectionRange(0, str.length)
let ok = false
try { ok = document.execCommand('copy') } catch { ok = false }
document.body.removeChild(ta)
if (!ok) throw new Error('clipboard copy not supported')
}
+61
View File
@@ -0,0 +1,61 @@
// Whitelist-based HTML sanitizer for rendering third-party post
// descriptions (e.g. Patreon) via v-html. Strips script/style/iframe
// + event handlers + dangerous hrefs. Tag whitelist covers what
// Patreon, SubscribeStar, and similar platforms ship in normal posts.
//
// Not a substitute for server-side sanitization in higher-stakes
// contexts. This is for FC's single-operator homelab posture where
// the content source is the operator's own subscriptions.
const ALLOWED_TAGS = new Set([
'a', 'b', 'blockquote', 'br', 'code', 'div', 'em', 'figure',
'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'hr', 'i', 'img', 'li',
'ol', 'p', 'pre', 's', 'span', 'strong', 'sub', 'sup', 'u', 'ul',
])
const ALLOWED_ATTRS = {
a: new Set(['href', 'title', 'rel', 'target']),
img: new Set(['src', 'alt', 'title', 'width', 'height']),
// any tag → these always allowed
'*': new Set(['class']),
}
const SAFE_URL_RE = /^(https?:|mailto:|#|\/)/i
export function sanitizeHtml (html) {
if (typeof html !== 'string' || !html) return ''
const doc = new DOMParser().parseFromString(html, 'text/html')
_scrubNode(doc.body)
return doc.body.innerHTML
}
function _scrubNode (node) {
const children = Array.from(node.children)
for (const child of children) {
const tag = child.tagName.toLowerCase()
if (!ALLOWED_TAGS.has(tag)) {
// Strip the tag but keep its text content as a fallback.
const text = document.createTextNode(child.textContent || '')
child.replaceWith(text)
continue
}
const allowed = new Set([
...(ALLOWED_ATTRS[tag] || []),
...(ALLOWED_ATTRS['*'] || []),
])
for (const attr of Array.from(child.attributes)) {
const name = attr.name.toLowerCase()
if (name.startsWith('on') || !allowed.has(name)) {
child.removeAttribute(attr.name)
continue
}
if ((name === 'href' || name === 'src') && !SAFE_URL_RE.test(attr.value)) {
child.removeAttribute(attr.name)
}
}
if (tag === 'a' && child.getAttribute('target') === '_blank') {
child.setAttribute('rel', 'noopener noreferrer')
}
_scrubNode(child)
}
}
+73 -164
View File
@@ -1,187 +1,96 @@
<template>
<v-container fluid class="py-6">
<div v-if="store.loading && !store.overview" class="fc-artist__loading">
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<div v-if="store.loading && !store.overview" class="fc-artist__loading">
<v-progress-circular indeterminate color="accent" size="36" />
</div>
<v-alert v-else-if="store.notFound" type="warning" variant="tonal">
Artist not found.
</v-alert>
<v-alert v-else-if="store.error" type="error" variant="tonal" closable>
{{ store.error }}
</v-alert>
<template v-else-if="store.overview">
<header class="fc-artist__head">
<h1 class="fc-h1">{{ store.overview.name }}</h1>
<div class="fc-artist__stats">
<span>{{ store.overview.image_count }} images</span>
<span v-if="dateRange">· {{ dateRange }}</span>
</div>
</header>
<div class="fc-artist__fc4">
<v-chip
size="small"
:variant="store.overview.is_subscription ? 'flat' : 'outlined'"
:color="store.overview.is_subscription ? 'accent' : undefined"
prepend-icon="mdi-rss"
>{{ store.overview.is_subscription ? 'Subscription' : 'One-off' }}</v-chip>
<v-chip
size="small" variant="outlined" prepend-icon="mdi-link-variant"
:to="`/subscriptions?artist_id=${store.overview.id}`"
>{{ store.overview.sources.length }} source{{ store.overview.sources.length === 1 ? '' : 's' }}</v-chip>
<v-chip
size="small" variant="outlined" prepend-icon="mdi-rss"
:to="`/posts?artist_id=${store.overview.id}`"
>View posts</v-chip>
<v-chip
size="small" variant="outlined" disabled
prepend-icon="mdi-clock-outline"
>Credential health · FC-3b</v-chip>
</div>
<section v-if="store.overview.cooccurring_tags.length" class="fc-artist__sec">
<h2 class="fc-h2">Frequent tags</h2>
<div class="fc-artist__tags">
<v-chip
v-for="t in store.overview.cooccurring_tags" :key="t.id"
size="small" @click="openTag(t.id)"
>{{ t.name }} <span class="fc-artist__tagc">{{ t.count }}</span></v-chip>
</div>
</section>
<section v-if="store.overview.activity.length" class="fc-artist__sec">
<h2 class="fc-h2">Activity</h2>
<svg class="fc-artist__spark" :viewBox="`0 0 ${sparkW} ${sparkH}`"
preserveAspectRatio="none" role="img" aria-label="posts over time">
<polyline :points="sparkPoints" fill="none"
stroke="rgb(var(--v-theme-accent))" stroke-width="2" />
</svg>
</section>
<section v-if="store.overview.sources.length" class="fc-artist__sec">
<div class="fc-artist__sec-head">
<h2 class="fc-h2">Sources</h2>
<RouterLink
:to="`/subscriptions?artist_id=${store.overview.id}`"
class="fc-artist__manage"
>Manage subscriptions </RouterLink>
</div>
<v-table density="compact">
<thead>
<tr><th>Platform</th><th>URL</th><th class="text-right">Images</th></tr>
</thead>
<tbody>
<tr v-for="s in store.overview.sources" :key="s.id">
<td>{{ s.platform }}</td>
<td class="fc-artist__url">{{ s.url }}</td>
<td class="text-right">{{ s.image_count }}</td>
</tr>
</tbody>
</v-table>
</section>
<section class="fc-artist__sec">
<h2 class="fc-h2">Images</h2>
<MasonryGrid
:items="store.images"
:loading="store.imagesLoading"
:has-more="store.hasMoreImages"
@load-more="store.loadMoreImages(slug)"
@open="openImage"
/>
</section>
<ArtistDangerZone
:slug="slug"
:artist-id="store.overview.id"
:artist-name="store.overview.name"
/>
</template>
<v-container v-else-if="store.notFound" class="pt-2 pb-6">
<v-alert type="warning" variant="tonal">Artist not found.</v-alert>
</v-container>
<v-container v-else-if="store.error && !store.overview" class="pt-2 pb-6">
<v-alert type="error" variant="tonal" closable>{{ store.error }}</v-alert>
</v-container>
<template v-else-if="store.overview">
<ArtistHeader
v-model="tab"
:name="store.overview.name"
:image-count="store.imageCount"
:post-count="store.postCount"
:last-added="store.lastAdded"
/>
<v-container fluid class="pt-2 pb-4">
<v-window v-model="tab">
<v-window-item value="posts">
<ArtistPostsTab
:artist-id="store.overview.id"
@switch-tab="(t) => tab = t"
/>
</v-window-item>
<v-window-item value="gallery">
<ArtistGalleryTab :slug="slug" />
</v-window-item>
<v-window-item value="management">
<ArtistManagementTab :overview="store.overview" />
</v-window-item>
</v-window>
</v-container>
</template>
</template>
<script setup>
import { computed, watch } from 'vue'
import { useRoute, useRouter, RouterLink } from 'vue-router'
import { computed, ref, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router'
import { useArtistStore } from '../stores/artist.js'
import { useModalStore } from '../stores/modal.js'
import MasonryGrid from '../components/discovery/MasonryGrid.vue'
import ArtistDangerZone from '../components/artist/ArtistDangerZone.vue'
import ArtistHeader from '../components/artist/ArtistHeader.vue'
import ArtistPostsTab from '../components/artist/ArtistPostsTab.vue'
import ArtistGalleryTab from '../components/artist/ArtistGalleryTab.vue'
import ArtistManagementTab from '../components/artist/ArtistManagementTab.vue'
const VALID_TABS = ['posts', 'gallery', 'management']
const route = useRoute()
const router = useRouter()
const store = useArtistStore()
const modal = useModalStore()
const slug = computed(() => route.params.slug)
const tab = ref('posts')
watch(slug, (s) => { if (s) store.load(s) }, { immediate: true })
function resolveDefaultTab () {
const fromUrl = route.query.tab
if (VALID_TABS.includes(fromUrl)) return fromUrl
if ((store.postCount ?? 0) > 0) return 'posts'
return 'gallery'
}
const dateRange = computed(() => {
const r = store.overview?.date_range
if (!r || !r.min) return null
const fmt = (iso) => iso.slice(0, 10)
return r.min === r.max ? fmt(r.min) : `${fmt(r.min)}${fmt(r.max)}`
watch(slug, async (s) => {
if (!s) return
await store.load(s)
document.title = store.overview
? `${store.overview.name} — FabledCurator`
: 'FabledCurator'
tab.value = resolveDefaultTab()
}, { immediate: true })
// Reflect tab changes back into the URL so refresh/back/forward work.
watch(tab, (newTab) => {
if (route.query.tab === newTab) return
router.replace({
query: { ...route.query, tab: newTab },
})
})
const sparkW = 600
const sparkH = 80
const sparkPoints = computed(() => {
const a = store.overview?.activity ?? []
if (a.length === 0) return ''
const max = Math.max(...a.map(p => p.count), 1)
const stepX = a.length > 1 ? sparkW / (a.length - 1) : 0
return a.map((p, i) => {
const x = i * stepX
const y = sparkH - (p.count / max) * (sparkH - 4) - 2
return `${x.toFixed(1)},${y.toFixed(1)}`
}).join(' ')
// React to URL-tab changes (e.g., back/forward).
watch(() => route.query.tab, (q) => {
if (q && VALID_TABS.includes(q) && tab.value !== q) {
tab.value = q
}
})
function openImage(id) {
modal.open(id)
}
function openTag(tagId) {
router.push({ name: 'gallery', query: { tag_id: tagId } })
}
</script>
<style scoped>
.fc-h1 {
font-family: 'Fraunces', Georgia, serif;
font-size: 32px; font-weight: 500;
color: rgb(var(--v-theme-on-surface));
.fc-artist__loading {
display: flex; justify-content: center; padding: 64px 0;
}
.fc-h2 {
font-family: 'Fraunces', Georgia, serif;
font-size: 20px; font-weight: 500; margin-bottom: 8px;
}
.fc-artist__loading { display: flex; justify-content: center; padding: 64px 0; }
.fc-artist__head { margin-bottom: 12px; }
.fc-artist__stats { opacity: 0.75; margin-top: 4px; }
.fc-artist__fc4 { display: flex; gap: 8px; flex-wrap: wrap; margin-bottom: 24px; }
.fc-artist__sec { margin-bottom: 28px; }
.fc-artist__tags { display: flex; flex-wrap: wrap; gap: 6px; }
.fc-artist__tagc { opacity: 0.6; margin-left: 4px; font-variant-numeric: tabular-nums; }
.fc-artist__spark { width: 100%; height: 80px; }
.fc-artist__url {
max-width: 380px; overflow: hidden; text-overflow: ellipsis;
white-space: nowrap;
}
.fc-artist__sec-head {
display: flex; align-items: baseline; justify-content: space-between;
margin-bottom: 0.25rem;
}
.fc-artist__manage {
font-size: 0.85rem;
color: rgb(var(--v-theme-accent));
text-decoration: none;
}
.fc-artist__manage:hover { text-decoration: underline; }
</style>
+2 -2
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<div class="fc-artists__controls">
<v-text-field
@@ -84,7 +84,7 @@ onUnmounted(() => observer && observer.disconnect())
}
.fc-artists__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
gap: 12px;
}
.fc-artists__sentinel {
+31
View File
@@ -0,0 +1,31 @@
<template>
<div class="fc-cleanup">
<p class="fc-muted text-body-2 mb-4">
Retroactive enforcement of import-filter rules. Each card scans the
existing library for content that the current import filters would
now exclude. Destructive typed-token confirmation required.
</p>
<MinDimensionCard class="mb-4" />
<TransparencyAuditCard class="mb-4" />
<SingleColorAuditCard class="mb-4" />
<TagMaintenanceCard />
</div>
</template>
<script setup>
import MinDimensionCard from '../components/cleanup/MinDimensionCard.vue'
import TransparencyAuditCard from '../components/cleanup/TransparencyAuditCard.vue'
import SingleColorAuditCard from '../components/cleanup/SingleColorAuditCard.vue'
// Reuse existing TagMaintenanceCard (FC-3k) as-is — it already handles
// preview + commit of prune-unused-tags via the admin store. Operator
// confirmed 2026-05-26: don't duplicate into a new UnusedTagsCard.
// MaintenancePanel drops its TagMaintenanceCard reference in the
// SettingsView edit (Task 16) so this is now the sole rendering site.
import TagMaintenanceCard from '../components/settings/TagMaintenanceCard.vue'
</script>
<style scoped>
.fc-cleanup { max-width: 900px; }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<ExtensionKeyBar />
<v-alert v-if="credentialsStore.error" type="error" variant="tonal" closable class="mb-4">
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<FilterPills v-model="pill" />
<v-alert v-if="store.error" type="error" variant="tonal" closable class="my-4">
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<Teleport to="#fc-nav-actions">
<v-btn
:color="sel.isSelectMode ? 'accent' : undefined"
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container class="py-8">
<v-container class="pt-3 pb-8">
<h1 class="fc-h1 mb-4">{{ title }}</h1>
<v-alert type="info" variant="tonal" icon="mdi-toolbox">
This surface is a placeholder. It will be implemented in
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container class="py-6" max-width="900">
<v-container class="pt-2 pb-6" max-width="900">
<PostsFilterBar
:artist-id="artistFilter"
:platform="platformFilter"
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<div class="fc-series__head">
<span class="fc-series__name">{{ store.series?.name || 'Series' }}</span>
<span class="fc-series__count">{{ store.pages.length }} page(s)</span>
+29 -4
View File
@@ -1,9 +1,20 @@
<template>
<v-container fluid class="py-6">
<v-tabs v-model="tab" color="accent" class="mb-4">
<v-container fluid class="pt-2 pb-6">
<!-- Sticky tabs: operator-flagged 2026-05-25 long Import / Maintenance
panels pushed the tab strip out of the viewport, forcing a scroll-
to-top just to change tab. AppShell's TopNav is 64px sticky, so the
tab strip lives directly under it. Background uses the theme surface
token so it visually merges with the page rather than the
translucent v-tabs default. -->
<v-tabs
v-model="tab" color="accent" class="mb-4"
style="position: sticky; top: 64px; z-index: 4;
background: rgb(var(--v-theme-surface));"
>
<v-tab value="overview">Overview</v-tab>
<v-tab value="activity">Activity</v-tab>
<v-tab value="import">Import</v-tab>
<v-tab value="cleanup">Cleanup</v-tab>
<v-tab value="maintenance">Maintenance</v-tab>
</v-tabs>
@@ -21,6 +32,11 @@
{{ system.stats.tasks.pending + system.stats.tasks.queued }} import task(s) pending.
<v-btn variant="text" size="small" @click="tab = 'import'">Go to Import tab</v-btn>
</v-alert>
<!-- Browser-extension install/download lives on Overview (moved
from Maintenance 2026-05-25). Overview is the discovery
surface for "things to set up"; Maintenance is for
housekeeping of already-set-up systems. -->
<BrowserExtensionCard class="mt-6" />
</v-window-item>
<v-window-item value="activity">
@@ -28,11 +44,18 @@
</v-window-item>
<v-window-item value="import">
<!-- Order: trigger → recent tasks → filters. Tasks sit directly
below the trigger so operator sees hit/miss feedback without
scrolling past the filter card (operator-flagged 2026-05-25). -->
<ImportTriggerPanel />
<v-divider class="my-6" />
<ImportFiltersForm />
<v-divider class="my-6" />
<ImportTaskList />
<v-divider class="my-6" />
<ImportFiltersForm />
</v-window-item>
<v-window-item value="cleanup">
<CleanupView />
</v-window-item>
<v-window-item value="maintenance">
@@ -49,10 +72,12 @@ import { useImportStore } from '../stores/import.js'
import SystemStatsCards from '../components/settings/SystemStatsCards.vue'
import SystemActivitySummary from '../components/settings/SystemActivitySummary.vue'
import SystemActivityTab from '../components/settings/SystemActivityTab.vue'
import BrowserExtensionCard from '../components/settings/BrowserExtensionCard.vue'
import ImportTriggerPanel from '../components/settings/ImportTriggerPanel.vue'
import ImportFiltersForm from '../components/settings/ImportFiltersForm.vue'
import ImportTaskList from '../components/settings/ImportTaskList.vue'
import MaintenancePanel from '../components/settings/MaintenancePanel.vue'
import CleanupView from './CleanupView.vue'
import { useMLStore } from '../stores/ml.js'
const tab = ref('overview')
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<Teleport to="#fc-nav-actions">
<v-btn
prepend-icon="mdi-shuffle-variant" variant="tonal" color="accent"
+1 -1
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<div class="fc-subs__bar">
<v-btn color="accent" prepend-icon="mdi-plus" @click="openAddSource(null)">
Add subscription
+7 -6
View File
@@ -1,5 +1,5 @@
<template>
<v-container fluid class="py-6">
<v-container fluid class="pt-2 pb-6">
<div class="fc-tags__controls">
<v-text-field
@@ -99,10 +99,11 @@ import MergeConfirmDialog from '../components/discovery/MergeConfirmDialog.vue'
import DestructiveConfirmModal from '../components/modal/DestructiveConfirmModal.vue'
// Must stay a subset of the backend TagKind enum (character, fandom,
// general, series, archive, post, meta, rating). 'fandom' is this
// model's copyright/franchise concept (characters link via fandom_id).
// 'artist' retired in FC-2d-vii-c — artists are the Artist row, not a tag.
const KINDS = ['character', 'fandom', 'general', 'series', 'meta']
// general, series, archive, post). 'fandom' is this model's
// copyright/franchise concept (characters link via fandom_id).
// 'artist' retired in FC-2d-vii-c — artists are the Artist row, not
// a tag. 'meta' + 'rating' retired by operator 2026-05-26 (alembic 0023).
const KINDS = ['character', 'fandom', 'general', 'series']
const store = useTagDirectoryStore()
const router = useRouter()
@@ -236,7 +237,7 @@ async function onDeleteTagConfirm() {
}
.fc-tags__grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
grid-template-columns: repeat(auto-fill, minmax(440px, 1fr));
gap: 12px;
}
.fc-tags__sentinel {
+24
View File
@@ -29,6 +29,30 @@ async def test_artist_overview_ok(client, db):
body = await resp.get_json()
assert body["name"] == "Mira"
assert body["image_count"] == 0
assert body["post_count"] == 0
@pytest.mark.asyncio
async def test_artist_overview_post_count(client, db):
from backend.app.models import Post, Source
a = Artist(name="Lyra", slug="lyra")
db.add(a)
await db.flush()
s = Source(
artist_id=a.id, platform="patreon",
url="https://patreon.com/cw/lyra", enabled=True,
)
db.add(s)
await db.flush()
db.add(Post(source_id=s.id, external_post_id="p1"))
db.add(Post(source_id=s.id, external_post_id="p2"))
await db.flush()
await db.commit()
resp = await client.get("/api/artist/lyra")
assert resp.status_code == 200
body = await resp.get_json()
assert body["post_count"] == 2
@pytest.mark.asyncio
+215
View File
@@ -0,0 +1,215 @@
"""API tests for the /api/cleanup/* blueprint.
Per reference-async-coredml-test-assertions, post-DML state checks go
via column selects, not ORM entity access.
"""
import hashlib
from datetime import UTC, datetime
import pytest
from PIL import Image
from sqlalchemy import func, select
from backend.app import create_app
from backend.app.celery_app import celery
from backend.app.models import ImageRecord, LibraryAuditRun
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def disable_celery_eager(monkeypatch):
monkeypatch.setattr(celery.conf, "task_always_eager", False)
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
def _sha256_min_dim_token(min_w: int, min_h: int) -> str:
canon = f"{min_w}x{min_h}"
return f"delete-min-dim-{hashlib.sha256(canon.encode()).hexdigest()[:8]}"
async def _seed_image(db, tmp_path, *, w, h, name):
path = tmp_path / name
Image.new("RGB", (w, h), (w % 256, h % 256, 0)).save(path)
sha = f"api-cleanup-{name}".ljust(64, "x")[:64]
rec = ImageRecord(
path=str(path), sha256=sha,
size_bytes=path.stat().st_size, mime="image/png",
width=w, height=h, origin="imported_filesystem",
integrity_status="ok",
)
db.add(rec)
await db.flush()
return rec
@pytest.mark.asyncio
async def test_min_dimension_preview_returns_count(client, db, tmp_path):
await _seed_image(db, tmp_path, w=50, h=50, name="small.png")
await _seed_image(db, tmp_path, w=500, h=500, name="big.png")
await db.commit()
resp = await client.post(
"/api/cleanup/min-dimension/preview",
json={"min_width": 200, "min_height": 200},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["count"] == 1
@pytest.mark.asyncio
async def test_min_dimension_delete_with_token_removes_rows(client, db, tmp_path):
await _seed_image(db, tmp_path, w=50, h=50, name="s2.png")
await _seed_image(db, tmp_path, w=500, h=500, name="b2.png")
await db.commit()
token = _sha256_min_dim_token(200, 200)
resp = await client.post(
"/api/cleanup/min-dimension/delete",
json={"min_width": 200, "min_height": 200, "confirm": token},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["deleted"] == 1
remaining = await db.execute(select(func.count()).select_from(ImageRecord))
assert remaining.scalar_one() == 1
@pytest.mark.asyncio
async def test_min_dimension_delete_with_bad_token_returns_400(client, db, tmp_path):
await _seed_image(db, tmp_path, w=50, h=50, name="s3.png")
await db.commit()
resp = await client.post(
"/api/cleanup/min-dimension/delete",
json={"min_width": 200, "min_height": 200, "confirm": "nope"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
@pytest.mark.asyncio
async def test_audit_create_returns_id_and_running_status(
client, db, monkeypatch,
):
from backend.app.tasks import library_audit
monkeypatch.setattr(
library_audit.scan_library_for_rule, "delay", lambda audit_id: None,
)
resp = await client.post(
"/api/cleanup/audit",
json={"rule": "transparency", "params": {"threshold": 0.9}},
)
assert resp.status_code == 202
body = await resp.get_json()
assert body["status"] == "running"
assert isinstance(body["audit_id"], int)
@pytest.mark.asyncio
async def test_audit_create_returns_409_when_another_is_running(client, db):
audit = LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="running", matched_ids=[],
)
db.add(audit)
await db.commit()
resp = await client.post(
"/api/cleanup/audit",
json={"rule": "single_color", "params": {"threshold": 0.95, "tolerance": 30}},
)
assert resp.status_code == 409
@pytest.mark.asyncio
async def test_audit_get_by_id_returns_full_row(client, db):
audit = LibraryAuditRun(
rule="transparency", params={"threshold": 0.85},
status="ready", scanned_count=100, matched_count=3,
matched_ids=[1, 2, 3],
)
db.add(audit)
await db.commit()
resp = await client.get(f"/api/cleanup/audit/{audit.id}")
assert resp.status_code == 200
body = await resp.get_json()
assert body["rule"] == "transparency"
assert body["matched_count"] == 3
assert body["matched_ids"] == [1, 2, 3]
@pytest.mark.asyncio
async def test_audit_history_returns_recent_runs(client, db):
for _ in range(3):
db.add(LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="applied", matched_ids=[],
finished_at=datetime.now(UTC),
))
await db.commit()
resp = await client.get("/api/cleanup/audit?limit=5")
assert resp.status_code == 200
body = await resp.get_json()
assert len(body["runs"]) >= 3
@pytest.mark.asyncio
async def test_audit_apply_with_token_deletes(client, db, tmp_path):
rec = await _seed_image(db, tmp_path, w=100, h=100, name="apply.png")
audit = LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="ready", scanned_count=1, matched_count=1,
matched_ids=[rec.id],
)
db.add(audit)
await db.commit()
resp = await client.post(
f"/api/cleanup/audit/{audit.id}/apply",
json={"confirm": f"delete-audit-{audit.id}"},
)
assert resp.status_code == 200
body = await resp.get_json()
assert body["deleted"] == 1
@pytest.mark.asyncio
async def test_audit_apply_with_bad_token_returns_400(client, db):
audit = LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="ready", matched_ids=[],
)
db.add(audit)
await db.commit()
resp = await client.post(
f"/api/cleanup/audit/{audit.id}/apply",
json={"confirm": "wrong-token"},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "confirm_mismatch"
@pytest.mark.asyncio
async def test_audit_cancel_flips_status(client, db):
audit = LibraryAuditRun(
rule="transparency", params={"threshold": 0.9},
status="running", matched_ids=[],
)
db.add(audit)
await db.commit()
resp = await client.post(f"/api/cleanup/audit/{audit.id}/cancel")
assert resp.status_code == 200
new_status = await db.execute(
select(LibraryAuditRun.status).where(LibraryAuditRun.id == audit.id)
)
assert new_status.scalar_one() == "cancelled"
+55
View File
@@ -86,6 +86,61 @@ async def test_clear_completed(client, db):
assert body["deleted"] == 1
@pytest.mark.asyncio
async def test_clear_stuck_fails_non_terminal_and_finalizes_orphan_batch(client, db):
"""Operator-flagged 2026-05-25: 3 large PNGs got stuck in 'processing'
for 2 days, the active ImportBatch never finalized, and the UI's
'Scanning...' banner persisted with 0/0 files. /api/import/clear-stuck
is the escape hatch to break the autoretry loop manually."""
from sqlalchemy import select as _select
batch = ImportBatch(triggered_by="manual", source_path="/import", scan_mode="quick")
db.add(batch)
await db.flush()
# Three stuck rows in mixed non-terminal states.
db.add(ImportTask(
batch_id=batch.id, source_path="/p1", task_type="media", status="processing",
))
db.add(ImportTask(
batch_id=batch.id, source_path="/p2", task_type="media", status="queued",
))
db.add(ImportTask(
batch_id=batch.id, source_path="/p3", task_type="media", status="pending",
))
# One already-complete row should be untouched.
db.add(ImportTask(
batch_id=batch.id, source_path="/done", task_type="media",
status="complete", finished_at=datetime.now(UTC),
))
await db.commit()
resp = await client.post("/api/import/clear-stuck")
body = await resp.get_json()
assert resp.status_code == 200
assert body["tasks_failed"] == 3
assert body["batches_finalized"] == 1
statuses = {
row.status for row in
(await db.execute(_select(ImportTask).where(ImportTask.batch_id == batch.id)))
.scalars().all()
}
assert statuses == {"failed", "complete"}
batch_status = (await db.execute(
_select(ImportBatch.status).where(ImportBatch.id == batch.id)
)).scalar_one()
assert batch_status == "complete"
@pytest.mark.asyncio
async def test_clear_stuck_no_op_when_nothing_stuck(client, db):
resp = await client.post("/api/import/clear-stuck")
body = await resp.get_json()
assert resp.status_code == 200
assert body == {"tasks_failed": 0, "batches_finalized": 0}
@pytest.mark.asyncio
async def test_trigger_accepts_deep(client, monkeypatch):
# Stub the task dispatch — assert the API accepts 'deep' and forwards
+47
View File
@@ -117,3 +117,50 @@ async def test_detail_404_for_unknown(client):
assert resp.status_code == 404
body = await resp.get_json()
assert body["error"] == "not_found"
@pytest.mark.asyncio
async def test_detail_returns_uncapped_thumbnails(client, db):
"""Feed query caps thumbnails at 6 for previews; detail endpoint
returns the full list so PostModal can render the masonry grid."""
from backend.app.models import ImageRecord
a = Artist(name="yuki-api", slug="yuki-api")
db.add(a)
await db.flush()
s = Source(
artist_id=a.id, platform="patreon",
url="https://patreon.com/cw/yuki-api", enabled=True,
)
db.add(s)
await db.flush()
p = Post(
source_id=s.id, external_post_id="DETAIL10",
post_title="big post", description="<p>body</p>",
)
db.add(p)
await db.flush()
# Seed 10 ImageRecord rows linked to this post via primary_post_id.
for i in range(10):
sha = f"y{i:x}".ljust(64, "0")[:64]
rec = ImageRecord(
path=f"/images/test-yuki-{i}.jpg",
sha256=sha,
size_bytes=1,
mime="image/jpeg",
width=64,
height=64,
origin="downloaded",
integrity_status="unknown",
primary_post_id=p.id,
artist_id=a.id,
)
db.add(rec)
await db.commit()
resp = await client.get(f"/api/posts/{p.id}")
assert resp.status_code == 200
body = await resp.get_json()
# Detail returns ALL 10 thumbnails (feed would return 6 + thumbnails_more).
assert len(body["thumbnails"]) == 10
assert body["description_full"] == "body"
+48 -2
View File
@@ -55,6 +55,52 @@ async def test_create_character_with_bad_fandom_id(client):
@pytest.mark.asyncio
async def test_create_tag_missing_required(client):
resp = await client.post("/api/tags", json={"name": "Bob"})
async def test_create_tag_missing_name_400(client):
"""name is still required; only `kind` became optional."""
resp = await client.post("/api/tags", json={})
assert resp.status_code == 400
resp = await client.post("/api/tags", json={"kind": "artist"})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_create_tag_name_only_defaults_to_general(client):
"""IR-style: name without kind and without `kind:` prefix → general."""
resp = await client.post("/api/tags", json={"name": "sunset"})
assert resp.status_code == 201
body = await resp.get_json()
assert body["name"] == "sunset"
assert body["kind"] == "general"
@pytest.mark.asyncio
async def test_create_tag_with_kind_prefix(client):
"""IR-style: `name="character:Saber"` without explicit kind → parsed as character."""
resp = await client.post("/api/tags", json={"name": "character:Saber"})
assert resp.status_code == 201
body = await resp.get_json()
assert body["name"] == "Saber"
assert body["kind"] == "character"
@pytest.mark.asyncio
async def test_explicit_kind_overrides_prefix_parsing(client):
"""If caller passes explicit kind, don't re-parse the name —
colon and prefix stay literal."""
resp = await client.post(
"/api/tags", json={"name": "character:Saber", "kind": "general"}
)
assert resp.status_code == 201
body = await resp.get_json()
assert body["name"] == "character:Saber"
assert body["kind"] == "general"
@pytest.mark.asyncio
async def test_unknown_prefix_kept_literal(client):
"""`http:example` — `http` not in KNOWN_KINDS → kind=general, literal name."""
resp = await client.post("/api/tags", json={"name": "http://example.com"})
assert resp.status_code == 201
body = await resp.get_json()
assert body["name"] == "http://example.com"
assert body["kind"] == "general"
+32
View File
@@ -0,0 +1,32 @@
import pytest
from backend.app import create_app
from backend.app.celery_app import celery
pytestmark = pytest.mark.integration
@pytest.fixture(autouse=True)
def eager():
celery.conf.task_always_eager = True
yield
celery.conf.task_always_eager = False
@pytest.fixture
async def app():
return create_app()
@pytest.fixture
async def client(app):
async with app.test_client() as c:
yield c
@pytest.mark.asyncio
async def test_trigger_thumbnail_backfill(client):
r = await client.post("/api/thumbnails/backfill")
assert r.status_code == 202
body = await r.get_json()
assert "celery_task_id" in body
+42
View File
@@ -0,0 +1,42 @@
"""Tests for the single-color audit rule.
The rule downsamples + measures the fraction of pixels within `tolerance`
(Euclidean RGB distance) of the dominant color. Matches if that fraction
exceeds `threshold`. Single-color content is typically uploaded by
mistake (placeholder/error/preview images) and should be flagged.
"""
from PIL import Image
from backend.app.services.audits import single_color
def test_single_color_evaluate_true_for_uniform_image():
im = Image.new("RGB", (50, 50), (128, 64, 200))
assert single_color.evaluate(im, threshold=0.9, tolerance=10) is True
def test_single_color_evaluate_false_for_diverse_image():
# Half black, half white — no single color dominates.
im = Image.new("RGB", (50, 50), (0, 0, 0))
for x in range(25):
for y in range(50):
im.putpixel((x, y), (255, 255, 255))
assert single_color.evaluate(im, threshold=0.9, tolerance=10) is False
def test_single_color_evaluate_respects_tolerance_widening():
# Gradient image: pixels span 0..50 in R channel. Tight tolerance
# rejects (no concentration), wide tolerance accepts (all near 25).
im = Image.new("RGB", (50, 50), (0, 0, 0))
for x in range(50):
for y in range(50):
im.putpixel((x, y), (x, 0, 0))
assert single_color.evaluate(im, threshold=0.9, tolerance=5) is False
assert single_color.evaluate(im, threshold=0.9, tolerance=50) is True
def test_single_color_evaluate_handles_rgba_input():
# Alpha channel should be ignored — only RGB matters for the rule.
im = Image.new("RGBA", (50, 50), (100, 100, 100, 128))
assert single_color.evaluate(im, threshold=0.9, tolerance=10) is True
+46
View File
@@ -0,0 +1,46 @@
"""Tests for the transparency audit rule.
The rule mirrors `Importer._transparency_pct` semantics for retroactive
enforcement: returns True iff the fraction of fully-transparent pixels
exceeds the threshold. Animated images short-circuit to False to avoid
the multi-frame PIL decode that triggered SoftTimeLimitExceeded
2026-05-26 against animated WebPs.
"""
from PIL import Image
from backend.app.services.audits import transparency
def test_transparency_evaluate_true_when_fully_transparent():
im = Image.new("RGBA", (10, 10), (0, 0, 0, 0))
assert transparency.evaluate(im, threshold=0.5) is True
def test_transparency_evaluate_false_when_fully_opaque():
im = Image.new("RGBA", (10, 10), (200, 100, 50, 255))
assert transparency.evaluate(im, threshold=0.5) is False
def test_transparency_evaluate_respects_threshold_boundary():
# Half-transparent image: 50% alpha=0 pixels, 50% alpha=255.
im = Image.new("RGBA", (10, 10), (0, 0, 0, 0))
for x in range(5):
for y in range(10):
im.putpixel((x, y), (0, 0, 0, 255))
# 50% transparent. threshold=0.4 → True; threshold=0.6 → False.
assert transparency.evaluate(im, threshold=0.4) is True
assert transparency.evaluate(im, threshold=0.6) is False
def test_transparency_evaluate_false_for_rgb_image_without_alpha():
im = Image.new("RGB", (10, 10), (128, 128, 128))
assert transparency.evaluate(im, threshold=0.5) is False
def test_transparency_evaluate_false_for_animated_image():
im = Image.new("RGBA", (10, 10), (0, 0, 0, 0))
# Mark as animated (mimics PIL's WebP/GIF multi-frame attribute).
im.is_animated = True # type: ignore[attr-defined]
im.n_frames = 5 # type: ignore[attr-defined]
assert transparency.evaluate(im, threshold=0.5) is False
+242
View File
@@ -0,0 +1,242 @@
"""Thumbnail backfill: _thumb_is_valid helper + backfill_thumbnails planner."""
from pathlib import Path
import pytest
from backend.app.models import ImageRecord
from backend.app.tasks.thumbnail import _thumb_is_valid
pytestmark = pytest.mark.integration
def test_thumb_is_valid_jpeg(tmp_path):
p = tmp_path / "good.jpg"
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 100)
assert _thumb_is_valid(p) is True
def test_thumb_is_valid_png(tmp_path):
p = tmp_path / "good.png"
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 100)
assert _thumb_is_valid(p) is True
def test_thumb_is_valid_too_short(tmp_path):
p = tmp_path / "tiny"
p.write_bytes(b"\xff\xd8")
assert _thumb_is_valid(p) is False
def test_thumb_is_valid_wrong_magic(tmp_path):
p = tmp_path / "garbage"
p.write_bytes(b"\x00" * 12)
assert _thumb_is_valid(p) is False
def test_thumb_is_valid_missing_file(tmp_path):
assert _thumb_is_valid(tmp_path / "nope") is False
# --- backfill_thumbnails planner tests ------------------------------------
class _Ctx:
def __init__(self, s):
self.s = s
def __enter__(self):
return self.s
def __exit__(self, *a):
return False
def _sf(db_sync):
"""sessionmaker-like returning the test's bound session, matching the
pattern in tests/test_backfill_phash.py."""
class _SM:
def __call__(self):
return _Ctx(db_sync)
return _SM()
def _sha(prefix: str) -> str:
return f"{prefix}".ljust(64, "0")[:64]
def _rec(db_sync, path, *, sha, thumb_path=None, mime="image/jpeg"):
rec = ImageRecord(
path=str(path), sha256=sha, size_bytes=1, mime=mime,
width=64, height=64, origin="imported_filesystem",
integrity_status="unknown",
thumbnail_path=str(thumb_path) if thumb_path is not None else None,
)
db_sync.add(rec)
db_sync.flush()
return rec
def _write_jpeg(p: Path) -> Path:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"\xff\xd8\xff\xe0" + b"\x00" * 100)
return p
def _write_png(p: Path) -> Path:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 100)
return p
def _write_garbage(p: Path) -> Path:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(b"\x00" * 12)
return p
def test_backfill_null_path_enqueued(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src = tmp_path / "a.bin"
src.write_bytes(b"x")
rec = _rec(db_sync, src, sha=_sha("a"), thumb_path=None)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 0}
assert delayed == [rec.id]
def test_backfill_missing_file_clears_and_enqueues(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src = tmp_path / "b.bin"
src.write_bytes(b"x")
rec = _rec(
db_sync, src, sha=_sha("b"),
thumb_path=tmp_path / "thumbs" / "missing.jpg",
)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 1}
assert delayed == [rec.id]
assert db_sync.get(ImageRecord, rec.id).thumbnail_path is None
def test_backfill_valid_jpeg_skipped(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src = tmp_path / "c.bin"
src.write_bytes(b"x")
thumb = _write_jpeg(tmp_path / "thumbs" / "c.jpg")
rec = _rec(db_sync, src, sha=_sha("c"), thumb_path=thumb)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 0, "ok": 1, "regenerated": 0}
assert delayed == []
assert db_sync.get(ImageRecord, rec.id).thumbnail_path == str(thumb)
def test_backfill_valid_png_skipped(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src = tmp_path / "d.bin"
src.write_bytes(b"x")
thumb = _write_png(tmp_path / "thumbs" / "d.png")
_rec(db_sync, src, sha=_sha("d"), thumb_path=thumb)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 0, "ok": 1, "regenerated": 0}
assert delayed == []
def test_backfill_corrupt_magic_clears_and_enqueues(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src = tmp_path / "e.bin"
src.write_bytes(b"x")
thumb = _write_garbage(tmp_path / "thumbs" / "e.jpg")
rec = _rec(db_sync, src, sha=_sha("e"), thumb_path=thumb)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
db_sync.expire_all()
assert result == {"enqueued": 1, "ok": 0, "regenerated": 1}
assert delayed == [rec.id]
assert db_sync.get(ImageRecord, rec.id).thumbnail_path is None
def test_backfill_mixed_aggregate(db_sync, tmp_path, monkeypatch):
from backend.app.tasks import thumbnail as m
src_null = tmp_path / "src_null.bin"
src_null.write_bytes(b"x")
src_jpeg = tmp_path / "src_jpeg.bin"
src_jpeg.write_bytes(b"x")
src_png = tmp_path / "src_png.bin"
src_png.write_bytes(b"x")
src_missing = tmp_path / "src_missing.bin"
src_missing.write_bytes(b"x")
src_bad = tmp_path / "src_bad.bin"
src_bad.write_bytes(b"x")
jpeg = _write_jpeg(tmp_path / "thumbs" / "ok.jpg")
png = _write_png(tmp_path / "thumbs" / "ok.png")
bad = _write_garbage(tmp_path / "thumbs" / "bad.jpg")
r_null = _rec(db_sync, src_null, sha=_sha("aa"), thumb_path=None)
_rec(db_sync, src_jpeg, sha=_sha("bb"), thumb_path=jpeg)
_rec(db_sync, src_png, sha=_sha("cc"), thumb_path=png)
r_missing = _rec(
db_sync, src_missing, sha=_sha("dd"),
thumb_path=tmp_path / "thumbs" / "missing.jpg",
)
r_bad = _rec(db_sync, src_bad, sha=_sha("ee"), thumb_path=bad)
db_sync.commit()
monkeypatch.setattr(m, "_sync_session_factory", lambda: _sf(db_sync))
delayed: list[int] = []
monkeypatch.setattr(
m.generate_thumbnail, "delay", lambda image_id: delayed.append(image_id)
)
result = m.backfill_thumbnails()
assert result == {"enqueued": 3, "ok": 2, "regenerated": 2}
assert sorted(delayed) == sorted([r_null.id, r_missing.id, r_bad.id])

Some files were not shown because too many files have changed in this diff Show More