Compare commits

..

125 Commits

Author SHA1 Message Date
bvandeusen 38a5e7f332 Merge pull request 'feat(settings): tidy Cleanup tab into sectioned compact tiles (pass 1)' (#125) from dev into main
CI / backend-lint-and-test (push) Failing after 2s
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 19s
CI / integration (push) Successful in 3m19s
2026-06-17 23:49:21 -04:00
bvandeusen 57fe15c267 Merge pull request 'feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)' (#124) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m22s
2026-06-17 22:01:12 -04:00
bvandeusen eb3231ef10 Merge pull request 'fix(subscribestar): port gallery-dl date extraction (wrapped dates) + parse canary' (#123) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m19s
2026-06-17 21:15:43 -04:00
bvandeusen e9af459c0d Merge pull request 'feat(subscribestar): port gallery-dl doc + audio attachment extraction' (#122) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 9s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 7s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m15s
2026-06-17 16:20:12 -04:00
bvandeusen 6f02806aec Merge pull request 'fix(subscribestar): port gallery-dl content + preview-skip extraction (body bug)' (#121) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 12s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m25s
2026-06-17 16:10:56 -04:00
bvandeusen a1d19bd96a Merge pull request 'fix(subscribestar): mirror gallery-dl's full request profile (verify_subscriber gate)' (#120) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m24s
2026-06-17 15:32:28 -04:00
bvandeusen 26827ff38f Merge pull request 'fix(subscribestar): match gallery-dl's generic post delimiter (live-feed drift)' (#119) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m15s
2026-06-17 15:07:59 -04:00
bvandeusen 26dcfaf6c2 Merge pull request 'fix(subscribestar): initial feed GET is a navigation, not XHR (first-run drift)' (#118) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m14s
2026-06-17 14:30:02 -04:00
bvandeusen 9b1b0369cc Merge pull request 'SubscribeStar → native core ingester + native-ingest DRY pass (milestone #71)' (#117) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m20s
2026-06-17 12:48:29 -04:00
bvandeusen 18123fb9cb Merge pull request 'fix(external): recovery-sweep threshold + queue recording + split fetch timeouts (#883)' (#116) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m14s
2026-06-16 21:24:31 -04:00
bvandeusen 2e806f202f Merge pull request 'test(artist-dir): fix flaky uq_image_record_sha256 collision' (#115) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m17s
2026-06-16 20:34:21 -04:00
bvandeusen 18d5c05639 Merge pull request 'fix(ml): per-task async engine for recompute_centroid (#881)' (#114) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Failing after 3m19s
2026-06-16 20:24:37 -04:00
bvandeusen 11ddfc3876 Merge pull request 'fix(maint): resurface dedup/gated-purge results after navigate-away (#877)' (#113) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
Build images / build-web (push) Successful in 15s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m20s
2026-06-16 16:48:52 -04:00
bvandeusen 2b8ce86622 Merge pull request 'Gated Patreon posts: skip on ingest + cleanup tool (#874)' (#112) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m27s
2026-06-16 15:25:34 -04:00
bvandeusen 49bee77cdc Merge pull request 'Video tag quality: cadence sampling + min-frame aggregation + ML thread cap (#747)' (#111) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m15s
2026-06-16 14:08:40 -04:00
bvandeusen c209e3b37e Merge pull request 'Tier-1 video dedup: import-time + retroactive cleanup (#871)' (#110) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 27s
Build images / build-web (push) Successful in 26s
CI / integration (push) Successful in 3m16s
2026-06-16 08:55:38 -04:00
bvandeusen cffdd93418 Merge pull request 'Nested-archive extraction (#718) + post-first ingest (#67) + post-body canary (#862)' (#109) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m19s
2026-06-15 21:37:39 -04:00
bvandeusen fd84be40dd Merge pull request 'External-attach orphan fix (#859) + image-less post display' (#108) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 40s
CI / integration (push) Successful in 3m19s
2026-06-15 01:55:36 -04:00
bvandeusen 79f510d7f8 Merge pull request 'Merge dev → main: read post body from content_json_string (the empty-body fix) (#842)' (#107) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m24s
2026-06-15 00:19:45 -04:00
bvandeusen 59181069da Merge pull request 'Merge dev → main: per-post stdout diagnostics + post-field write DRY (#842/#753)' (#106) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 41s
Build images / build-web (push) Successful in 2m15s
Build images / build-ml (push) Successful in 2m55s
CI / integration (push) Successful in 3m11s
2026-06-14 23:45:39 -04:00
bvandeusen 428ecd8642 Merge pull request 'Merge dev → main: per-post body-capture diagnostics in the event UI (#842)' (#105) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m11s
2026-06-14 23:14:14 -04:00
bvandeusen ed1e04b831 Merge pull request 'Merge dev → main: recapture body-fetch fix + diagnostics (#842)' (#104) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m15s
2026-06-14 22:31:41 -04:00
bvandeusen f5156bd847 Merge pull request 'Merge dev → main: Recapture mode (#842) — re-grab post bodies/links + localize on-disk inline images' (#103) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 2m30s
Build images / build-ml (push) Successful in 3m27s
CI / integration (push) Successful in 3m29s
2026-06-14 21:21:49 -04:00
bvandeusen dfc3922d24 Merge pull request 'Merge dev → main: #830 rich post capture + external-host downloads (+ #768/#789/#739)' (#102) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 48s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m47s
CI / integration (push) Successful in 3m20s
2026-06-14 19:16:09 -04:00
bvandeusen 3eb08e926b Merge pull request 'fix(aliases): modal raw-key bug + alias visibility/management' (#101) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m13s
2026-06-12 14:02:00 -04:00
bvandeusen 9e81ced359 Merge pull request 'fix(images): percent-encode original-image URLs ('#' in paths 404'd)' (#100) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 36s
CI / integration (push) Successful in 3m15s
2026-06-12 00:44:56 -04:00
bvandeusen 11e9f5af60 Merge pull request 'fix(browse): tabs and search on one row' (#99) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 11s
CI / integration (push) Successful in 3m13s
2026-06-12 00:28:38 -04:00
bvandeusen 909fa37b15 Merge pull request 'Browse search + series numbering rework + kebab fix' (#98) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 2m50s
CI / integration (push) Successful in 3m16s
2026-06-12 00:14:31 -04:00
bvandeusen dfab8f65ff Merge pull request 'feat(series): flat sequence + cosmetic dividers + pending staging (#789)' (#97) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 2m13s
Build images / build-ml (push) Successful in 2m49s
CI / integration (push) Successful in 3m9s
2026-06-11 22:07:57 -04:00
bvandeusen 618f7cdc36 Merge pull request 'feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)' (#96) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 25s
Build images / build-web (push) Successful in 2m29s
Build images / build-ml (push) Successful in 3m0s
CI / integration (push) Successful in 3m13s
2026-06-11 19:32:05 -04:00
bvandeusen 028ea33a7c Merge pull request 'fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)' (#95) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 2m50s
Build images / build-ml (push) Successful in 2m58s
CI / integration (push) Successful in 3m15s
2026-06-11 09:22:22 -04:00
bvandeusen 444c1fb075 Merge pull request 'perf(migration): 0045 streams json_each (no materialize / no temp blowup)' (#94) from dev into main
Build images / sign-extension (push) Successful in 3s
Build images / build-web (push) Successful in 2m14s
Build images / build-ml (push) Successful in 2m36s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m8s
2026-06-10 22:07:16 -04:00
bvandeusen 26c68b0a75 Merge pull request 'fix(migration): 0045 guards json_each against scalar tagger_predictions' (#93) from dev into main
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m13s
Build images / sign-extension (push) Successful in 2s
Build images / build-web (push) Successful in 6s
Build images / build-ml (push) Successful in 2m40s
2026-06-10 20:33:27 -04:00
bvandeusen e75427b19a Merge pull request '#768 steps 1+2: normalized image_prediction table (read cutover)' (#92) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 27s
Build images / build-web (push) Successful in 2m4s
Build images / build-ml (push) Successful in 2m42s
CI / integration (push) Successful in 3m8s
2026-06-10 20:15:26 -04:00
bvandeusen 5447fab987 Merge pull request 'Activity search + RecoverySweep fix + tagger_predictions shrink (#762, #764) + backup polish' (#91) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 2m23s
Build images / build-ml (push) Successful in 2m51s
CI / integration (push) Successful in 3m10s
2026-06-10 14:33:35 -04:00
bvandeusen bad37e07b2 Merge pull request 'Browse hub, series rename, full-prediction dropdown + a DRY pass (7 sweeps)' (#90) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 27s
CI / frontend-build (push) Successful in 44s
Build images / build-ml (push) Successful in 3m2s
CI / integration (push) Successful in 3m15s
Build images / build-web (push) Successful in 2m17s
2026-06-10 00:24:01 -04:00
bvandeusen 2bfc9936a1 Merge pull request 'UI batch: tagging flow, series browse, fandom chips, nav' (#89) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 39s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m52s
CI / integration (push) Successful in 3m24s
2026-06-09 20:48:19 -04:00
bvandeusen 4c6406ee18 Merge pull request 'fix(posts): link duplicate items to every post + prune bare shells' (#88) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 28s
Build images / build-web (push) Successful in 2m19s
Build images / build-ml (push) Successful in 2m41s
CI / integration (push) Successful in 3m5s
2026-06-08 19:42:31 -04:00
bvandeusen bb47e80b3e Merge pull request 'fix(cleanup): unused-tags delete must use the same predicate as the preview' (#87) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m13s
2026-06-08 18:17:49 -04:00
bvandeusen dc1083b5e0 Merge pull request 'Unused-tag fandom fix + ML-worker logging/tuning + unified dropdown Enter' (#86) from dev into main
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m16s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 2m21s
2026-06-08 17:27:55 -04:00
bvandeusen e46893fefd Merge pull request 'Migration lock safety + remove the merge's full-table scan (the real 0040-hang fix)' (#85) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 25s
Build images / build-web (push) Successful in 2m3s
Build images / build-ml (push) Successful in 2m33s
CI / integration (push) Successful in 3m7s
2026-06-08 01:03:06 -04:00
bvandeusen 0666e15211 Merge pull request 'Series manage redesign (FC-6.4) + migration/normalize hardening + UX fixes' (#84) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 25s
Build images / build-web (push) Successful in 1m58s
Build images / build-ml (push) Successful in 2m28s
CI / integration (push) Successful in 3m6s
2026-06-07 21:17:57 -04:00
bvandeusen 747390631d Merge pull request 'FC-6 series authoring + backup/NFS hardening + UX fixes' (#83) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
Build images / build-web (push) Successful in 2m5s
Build images / build-ml (push) Successful in 2m43s
CI / integration (push) Successful in 3m6s
2026-06-07 19:31:04 -04:00
bvandeusen e0d2a20588 Merge pull request 'Modal focus/keyboard polish, Camie-in-autocomplete, re-extract self-resume' (#82) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 28s
Build images / build-web (push) Successful in 2m2s
Build images / build-ml (push) Successful in 2m32s
CI / integration (push) Successful in 3m9s
2026-06-07 12:17:43 -04:00
bvandeusen 1d84f67418 Merge pull request 'Modal: large centered spinner + kebab z-index fix' (#81) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 2m3s
Build images / build-ml (push) Successful in 2m35s
CI / integration (push) Successful in 3m7s
2026-06-07 10:57:00 -04:00
bvandeusen 91265df3d6 Merge pull request 'Maintenance-queue health + modal/tagging keyboard pass' (#80) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
Build images / build-web (push) Successful in 2m5s
Build images / build-ml (push) Successful in 2m26s
CI / integration (push) Successful in 3m1s
2026-06-07 10:31:01 -04:00
bvandeusen 11acdb0322 Merge pull request 'Patreon: a missing media file_name is a fallback, not API drift' (#79) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m2s
2026-06-06 23:14:25 -04:00
bvandeusen 2eb9fd5dd0 Merge pull request 'Patreon: enforce the backfill time-box mid-post (stop soft-limit overruns)' (#78) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m3s
2026-06-06 23:00:29 -04:00
bvandeusen 01e5ce1410 Merge pull request 'Patreon: resolve creator campaign from a single-post URL' (#77) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m4s
2026-06-06 21:38:33 -04:00
bvandeusen 3bb94674cf Merge pull request 'Patreon download concurrency cap + immediate backfill kickoff on create' (#76) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m1s
2026-06-06 21:27:22 -04:00
bvandeusen a75c602175 Merge pull request 'Subscriptions UX overhaul + Patreon /cw/ vanity fix' (#75) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 28s
CI / integration (push) Successful in 3m2s
2026-06-06 21:12:02 -04:00
bvandeusen ef8f4f7193 Merge pull request 'Tag-casing acronym fix, Patreon resolver hardening, archive diagnostics, post-card strip' (#74) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m1s
2026-06-06 19:59:32 -04:00
bvandeusen ec3d27b219 Merge pull request 'Tag-maintenance sweep + bug-fix batch: #699 #700 #701 #709 #711 #712 #713 #714' (#73) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m2s
2026-06-06 16:49:00 -04:00
bvandeusen 03bd3b2eda Merge pull request 'Native Patreon ingester + download-engine ownership (plans #697, #703–#708)' (#72) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 10s
CI / backend-lint-and-test (push) Successful in 25s
CI / frontend-build (push) Successful in 32s
CI / integration (push) Successful in 3m1s
2026-06-06 12:57:31 -04:00
bvandeusen 7395e77d75 Merge pull request 'Smarter backfill: time-boxed chunks, run-until-done (plan #693)' (#71) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 18s
CI / integration (push) Successful in 2m58s
2026-06-05 16:33:06 -04:00
bvandeusen 575d817919 Merge pull request '#70 dev→main: cursor-paged backfill + mobile modal fixes' from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / integration (push) Successful in 2m59s
2026-06-05 11:10:02 -04:00
bvandeusen 2a8f7cd8b6 Merge pull request '#69 dev→main: release v26.06.04.0' from dev into main
CI / lint (push) Successful in 2s
CI / backend-lint-and-test (push) Successful in 12s
CI / frontend-build (push) Successful in 19s
CI / integration (push) Successful in 3m2s
Build images / sign-extension (push) Has been skipped
Build images / build-ml (push) Successful in 6s
Build images / build-web (push) Successful in 6s
2026-06-04 23:16:12 -04:00
bvandeusen 83f8af8090 Merge pull request 'dev→main: surface near-duplicate (pHash) control + reorder import tab' (#68) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 6s
CI / backend-lint-and-test (push) Successful in 11s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 20s
CI / integration (push) Successful in 2m56s
2026-06-04 21:39:44 -04:00
bvandeusen 9a2617c1a2 Merge pull request 'dev→main: post-card redesign (images→modal, in-place text expand)' (#67) from dev into main
CI / lint (push) Failing after 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
CI / backend-lint-and-test (push) Successful in 11s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 24s
CI / integration (push) Successful in 2m57s
2026-06-04 17:32:50 -04:00
bvandeusen 81688815a0 Merge pull request 'dev→main: similar-search render fix + reset-content-tagging + scan persistence' (#66) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 22s
CI / integration (push) Successful in 2m58s
2026-06-04 16:59:52 -04:00
bvandeusen 773128c3bf Merge pull request 'dev→main: purpose-built mobile layout for subscriptions hub' (#65) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
CI / backend-lint-and-test (push) Successful in 14s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 38s
CI / integration (push) Successful in 2m56s
2026-06-04 13:43:56 -04:00
bvandeusen ce7b154ae9 Merge pull request 'dev→main: subscriptions table mobile card layout' (#64) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 14s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 22s
CI / integration (push) Successful in 2m57s
2026-06-04 12:56:58 -04:00
bvandeusen 9430a9d9c3 Merge pull request 'dev→main: gallery similarity search (Phase 3) + UI/mobile polish' (#63) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 21s
CI / integration (push) Successful in 2m56s
2026-06-04 11:21:26 -04:00
bvandeusen 23aee56ce3 Merge pull request 'dev→main: showcase cascade + filter styling + DB maintenance + gallery filter Phase 2 + showcase decode-gate + CI perf' (#62) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 11s
CI / frontend-build (push) Successful in 20s
CI / integration (push) Successful in 3m0s
2026-06-04 08:27:47 -04:00
bvandeusen 711abea567 Merge pull request 'Gallery speed + fandom editing + filters + pinned filter bar' (#61) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 11s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m29s
2026-06-04 00:07:19 -04:00
bvandeusen 844bb86802 Merge pull request 'fix(download): release DB connections across the gallery-dl subprocess' (#60) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m41s
CI / intcore (push) Successful in 8m24s
2026-06-03 22:11:36 -04:00
bvandeusen a8f6a464aa Merge pull request 'fix(download): salvage soft-time-limit kills + fix timeout ladder' (#59) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 21s
CI / frontend-build (push) Successful in 24s
Build images / build-web (push) Successful in 2m37s
Build images / build-ml (push) Successful in 3m21s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 7m47s
CI / intcore (push) Successful in 8m15s
2026-06-03 19:35:45 -04:00
bvandeusen ab9922ad2e Merge pull request 'feat(artist): "new since last visit" badge + banner' (#58) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 23s
Build images / build-web (push) Successful in 2m9s
Build images / build-ml (push) Successful in 2m53s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m9s
2026-06-03 16:20:54 -04:00
bvandeusen 0533807669 Merge pull request 'feat(ext): verify cookies in-browser before uploading (1.0.7)' (#57) from dev into main
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
extension / lint (push) Successful in 14s
Build images / sign-extension (push) Successful in 3m17s
Build images / build-ml (push) Successful in 3m28s
CI / intimp (push) Successful in 4m26s
Build images / build-web (push) Successful in 3m42s
CI / intapi (push) Successful in 9m32s
CI / intcore (push) Successful in 10m18s
2026-06-03 14:17:42 -04:00
bvandeusen 279dff3fb6 Merge pull request 'feat(ml): normalize Camie suggestion names to human-readable' (#56) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 2m55s
CI / intimp (push) Successful in 3m36s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m24s
2026-06-03 13:18:44 -04:00
bvandeusen 37e66cddc4 Merge pull request 'chore(modal): drop ?image=N soft-compat — pure overlay' (#55) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-ml (push) Successful in 2m46s
Build images / build-web (push) Successful in 2m36s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m12s
2026-06-02 19:35:04 -04:00
bvandeusen 9cf6b2d363 Merge pull request 'audit-g5 final + ML threshold default + kebab menu fix' (#54) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 27s
Build images / build-web (push) Successful in 3m0s
Build images / build-ml (push) Successful in 3m45s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 8m8s
CI / intcore (push) Successful in 8m34s
2026-06-02 19:09:49 -04:00
bvandeusen 6ef0fed41f Merge pull request 'audit-g5: architectural debt — 4 bundles (A/B/C/D)' (#53) from dev into main
CI / lint (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 18s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m51s
CI / intapi (push) Successful in 7m38s
CI / intcore (push) Successful in 8m52s
2026-06-02 18:07:25 -04:00
bvandeusen 89b48f8f35 Merge pull request 'audit-g4: status-enum miss batch' (#52) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 13s
Build images / build-web (push) Successful in 10s
CI / frontend-build (push) Successful in 35s
CI / intimp (push) Successful in 3m41s
CI / intapi (push) Successful in 7m25s
CI / intcore (push) Successful in 8m10s
2026-06-02 16:15:00 -04:00
bvandeusen d60e0b9494 Merge pull request 'audit-g3: lifecycle batch — recovery sweeps, retention, timeouts' (#51) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 36s
CI / intimp (push) Successful in 3m39s
CI / intapi (push) Successful in 7m39s
CI / intcore (push) Successful in 8m10s
2026-06-02 14:49:28 -04:00
bvandeusen 9c27a2d3c7 Merge pull request 'audit-g2: async race / state-leak fixes across eight stores' (#50) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 23s
CI / frontend-build (push) Successful in 37s
Build images / build-web (push) Successful in 13s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 8m15s
CI / intcore (push) Successful in 8m50s
2026-06-02 14:17:12 -04:00
bvandeusen 93e37681b7 Merge pull request 'audit-g1: six one-liner drift fixes from 2026-06-02 audit' (#49) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 10s
CI / intimp (push) Successful in 3m42s
CI / intapi (push) Successful in 7m28s
CI / intcore (push) Successful in 8m3s
2026-06-02 13:29:17 -04:00
bvandeusen 64ca858574 Merge pull request 'UX fixes: suggestion-accept chip refresh, showcase endless feed, non-media downloads' (#48) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Successful in 9s
CI / backend-lint-and-test (push) Successful in 12s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / intimp (push) Successful in 3m52s
CI / intapi (push) Successful in 7m45s
CI / intcore (push) Successful in 8m20s
2026-06-02 08:26:58 -04:00
bvandeusen 9d0c0b7da8 Merge pull request 'fix(thumbnails): surface backfill counts + tighten validity check' (#47) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 9s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 18s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m32s
CI / intcore (push) Successful in 8m4s
2026-06-01 22:34:38 -04:00
bvandeusen 8e4d252ae4 Merge pull request 'fix(downloads): enqueue thumbnail + ML per attached image' (#46) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 26s
CI / intimp (push) Successful in 3m40s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m46s
2026-06-01 21:52:42 -04:00
bvandeusen fdd3e01f56 Merge pull request 'ux(failing-sources): visible row separators + clearer hover' (#45) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 10s
Build images / build-web (push) Successful in 10s
CI / backend-lint-and-test (push) Successful in 20s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m44s
CI / intapi (push) Successful in 7m48s
CI / intcore (push) Successful in 8m16s
2026-06-01 21:09:28 -04:00
bvandeusen c82fb308b6 Merge pull request 'Post.source_id refactor + tick/backfill modes + PARTIAL classifier + Mux fix + UX' (#44) from dev into main
Build images / build-ml (push) Failing after 2s
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-web (push) Successful in 12s
CI / backend-lint-and-test (push) Successful in 19s
CI / frontend-build (push) Successful in 31s
CI / intimp (push) Successful in 3m47s
CI / intapi (push) Successful in 7m52s
CI / intcore (push) Successful in 8m16s
2026-06-01 20:44:23 -04:00
bvandeusen 8cf8d2ca4d Merge pull request 'Modal Esc/overflow polish, artist-scoped post scroll, failing-sources Logs button' (#43) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 2m45s
Build images / build-ml (push) Successful in 3m31s
CI / intimp (push) Successful in 3m48s
CI / intapi (push) Successful in 7m11s
CI / intcore (push) Successful in 7m47s
2026-06-01 12:10:11 -04:00
bvandeusen b1d58bc3b8 Merge pull request 'fix(ci): POSIX-safe SHORT_SHA in build.yml' (#42) from dev into main
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 2m17s
Build images / build-ml (push) Successful in 3m24s
CI / intimp (push) Successful in 3m22s
CI / intapi (push) Successful in 7m51s
CI / intcore (push) Successful in 8m5s
2026-06-01 08:03:17 -04:00
bvandeusen 65386f02a0 Merge pull request 'View modal batch: autofocus, suggestions UX, post-title click, retire copyright/artist' (#41) from dev into main
Build images / sign-extension (push) Successful in 2s
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 4s
Build images / build-web (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 15s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Successful in 3m46s
CI / intapi (push) Successful in 7m17s
CI / intcore (push) Successful in 8m2s
2026-06-01 07:01:42 -04:00
bvandeusen 667b05f14e Merge pull request 'Extension probe-and-add (v1.0.6) + per-commit image tags' (#40) from dev into main
CI / lint (push) Successful in 2s
Build images / build-ml (push) Failing after 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 22s
extension / lint (push) Successful in 8s
Build images / sign-extension (push) Successful in 1m51s
Build images / build-web (push) Failing after 4s
CI / intimp (push) Successful in 3m38s
CI / intapi (push) Successful in 7m20s
CI / intcore (push) Successful in 7m45s
2026-06-01 01:44:03 -04:00
bvandeusen 856e9104b4 Merge pull request 'Sidecar synthetic anchor cleanup + tier-gated classifier fix' (#39) from dev into main
CI / lint (push) Successful in 5s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 28s
CI / intimp (push) Successful in 3m54s
Build images / sign-extension (push) Has been skipped
Build images / build-web (push) Successful in 1m1s
Build images / build-ml (push) Successful in 1m21s
CI / intapi (push) Successful in 7m31s
CI / intcore (push) Successful in 8m0s
2026-06-01 00:16:58 -04:00
bvandeusen 0397642b21 Merge pull request 'Showcase cadence tuning + cooldown-aware bulk retry' (#38) from dev into main 2026-05-30 23:50:36 -04:00
bvandeusen 237575447d Merge pull request 'Thumbnail URL fix + archive daemon fix + batched initial loads' (#37) from dev into main 2026-05-30 22:01:43 -04:00
bvandeusen ed358757dc Merge pull request 'Most-overdue-first scheduling + rich timeout diagnostics' (#36) from dev into main 2026-05-30 14:30:41 -04:00
bvandeusen d181f4afb8 Merge pull request 'Downloads burst-prevention + maintenance-menu fix + gdl timeout' (#35) from dev into main 2026-05-30 11:43:18 -04:00
bvandeusen 2886fa4997 Merge pull request 'Tooltip !important fix — 104cac5 follow-up after Vite CSS reorder' (#34) from dev into main 2026-05-30 00:02:24 -04:00
bvandeusen f256f587ee Merge pull request 'UI batch + I1–I6 service passes + download-event recovery sweep' (#33) from dev into main 2026-05-29 22:46:16 -04:00
bvandeusen 384d8d5e50 Merge pull request 'Dashboard insights + project-wide DRY pass' (#32) from dev into main 2026-05-28 15:38:26 -04:00
bvandeusen 319e8c1d18 Merge pull request 'v26.05.28.0: downloads dashboard + task-resilience overhaul (timeouts, archive split, 3-layer poison-pill defense)' (#31) from dev into main 2026-05-28 00:45:00 -04:00
bvandeusen 9075d8eadd Merge pull request 'v26.05.27.2: subscribestar + HF cookie quirks, platforms package refactor, showcase IR-parity, secure-context audit' (#30) from dev into main 2026-05-27 21:34:02 -04:00
bvandeusen 88e53e5b86 Merge pull request 'v26.05.27.1: subscriptions hub + post-card merge + sidecar audit' (#29) from dev into main 2026-05-27 17:12:48 -04:00
bvandeusen 37e8b796a1 Merge pull request 'v26.05.27.0: PostCard redesign + IR-style tag suffix + drop meta/rating + extension v1.0.4 CSP fix' (#28) from dev into main 2026-05-27 11:31:18 -04:00
bvandeusen 4e82208926 Merge pull request 'v26.05.26.5 — extension CORS unblock + UI gap closes + CI workflow cleanup' (#27) from dev into main 2026-05-26 20:15:07 -04:00
bvandeusen 52fff00353 Merge pull request 'v26.05.26.4 — hotfix: migration 0022 pre-DELETE colliding ImageProvenance before UPDATE' (#26) from dev into main 2026-05-26 18:06:20 -04:00
bvandeusen c14338cbce Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main 2026-05-26 17:52:59 -04:00
bvandeusen 8c36dd28b0 Merge pull request 'v26.05.26.2 — hotfix: alembic 0022 Post-collision pre-merge + ci.yml cache continue-on-error' (#24) from dev into main 2026-05-26 16:50:43 -04:00
bvandeusen 88cfb3dd02 Merge pull request 'v26.05.26.1 — thumb backfill, modal redesign, recovery sweep race-safety, artist view redesign, extension fixes' (#23) from dev into main 2026-05-26 16:32:00 -04:00
bvandeusen 5d4f223b71 Merge pull request 'Release v26.05.25.7 — FC-Cleanup tab + UniqueViolation fix + error modal + extension install fix' (#22) from dev into main 2026-05-26 08:26:46 -04:00
bvandeusen 05090c6e85 Merge pull request 'Release v26.05.25.7 — animated-WebP worker fix + FC-Cleanup backend' (#21) from dev into main 2026-05-26 01:48:13 -04:00
bvandeusen 3a577d5ade Merge pull request 'fix(ext-ci): use browser_download_url + curl -f + ZIP magic check (XPI silently corrupt)' (#20) from dev into main 2026-05-26 00:43:02 -04:00
bvandeusen f4fe02e346 Merge pull request 'fix(ext-ci): drop actions/upload-artifact (Forgejo doesn't support v4+ GHES)' (#19) from dev into main 2026-05-25 23:33:40 -04:00
bvandeusen e766197d99 Merge pull request 'fix(ext-ci): jq→python + bump ext to 1.0.3 + rollback-on-upload-failure' (#18) from dev into main 2026-05-25 23:14:51 -04:00
bvandeusen 3872e1dda9 Merge pull request 'fix(ext-ci): web-ext v8 .cjs config workaround' (#17) from dev into main 2026-05-25 22:49:14 -04:00
bvandeusen 9814f3dbaf Merge pull request 'Release v26.05.25.5 — Extension publish refactor, deep-scan IR-parity, archive-import perf, artist Settings tab' (#16) from dev into main 2026-05-25 22:44:59 -04:00
bvandeusen b214460fdb Merge pull request 'Release v26.05.25.4 — importer ext sanitize fix, CI shard split, BrowserExtensionCard on Overview' (#15) from dev into main 2026-05-25 21:11:50 -04:00
bvandeusen ac55d0e8d8 Merge pull request 'fix(ext-ci): match AMO-renamed signed XPI' (#14) from dev into main 2026-05-25 18:22:50 -04:00
bvandeusen 89a89e0ded Merge pull request 'Release v26.05.25.3 — ML embedder SigLIP fix, import-UX, extension publish' (#13) from dev into main 2026-05-25 17:56:50 -04:00
bvandeusen 4e9aac2c05 Merge pull request 'v26.05.25.2: supersede + sidecar enrichment, scan toast feedback, CI uv + pip cache + durations' (#12) from dev into main 2026-05-25 14:30:25 -04:00
bvandeusen 2879ac6f2b Merge pull request 'v26.05.25.1: maintenance sweep + Camie v2 + corrupt-file handling + post-date gallery + clear-stuck escape hatch' (#11) from dev into main 2026-05-25 12:57:46 -04:00
bvandeusen b8dce6c483 Merge pull request 'FC-3h + FC-3k: backup first-class + admin destructive actions' (#10) from dev into main 2026-05-25 01:41:53 -04:00
bvandeusen d1c0b82a22 Merge pull request 'v26.05.24.3: FC-3i System Activity dashboard + migration backup-gate retired + modal Escape' (#9) from dev into main 2026-05-24 21:47:53 -04:00
bvandeusen 5526b8dc78 Merge pull request 'v26.05.24.2: IR Post/Provenance restore + modal artist fallback' (#8) from dev into main 2026-05-24 14:30:06 -04:00
bvandeusen 16eb7075c4 Merge pull request 'v26.05.24.1: FC-3g Firefox extension + worker resilience + UI/migration fixes' (#7) from dev into main 2026-05-24 12:52:31 -04:00
bvandeusen 885dcf64f3 Merge pull request 'v26.05.24.0: TopNav re-fix (flex 1 1 0 side cells)' (#6) from dev into main 2026-05-23 22:49:29 -04:00
bvandeusen f2f6b6d25e Merge pull request 'v26.05.23.3: dogfood UX polish + accurate active-batch stats' (#5) from dev into main 2026-05-23 22:05:59 -04:00
bvandeusen 0822240fde Merge pull request 'v26.05.23.2: serve /images + artist cleanup migrator' (#4) from dev into main 2026-05-23 12:19:16 -04:00
bvandeusen 27f7f3fd01 Merge pull request 'v26.05.23.1: migration durability + dogfood UX' (#3) from dev into main 2026-05-23 11:21:33 -04:00
bvandeusen c5bf564f53 Merge dev: v26.05.23.0 migration follow-ups (#2)
pg_dump + zstd in runtime image, lift Quart body cap to 1 GiB. See PR #2.
2026-05-22 22:37:06 -04:00
bvandeusen 602c7d275d Merge dev: FC-1 → FC-5 v1 build (#1)
First merge of `dev` into `main` for FabledCurator. Brings FC-1 (Foundation) through FC-5 (Migration tooling) onto `main`. See PR #1 body for the full stage rollup.
2026-05-22 14:15:45 -04:00
143 changed files with 1091 additions and 11295 deletions
-38
View File
@@ -329,41 +329,3 @@ jobs:
file: Dockerfile.ml file: Dockerfile.ml
push: true push: true
tags: ${{ steps.tag.outputs.tags }} tags: ${{ steps.tag.outputs.tags }}
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
# the GPU machine instead of building locally. Independent of web/ml (its own
# CUDA + onnxruntime-gpu image, context = agent/). Same tag cadence.
build-agent:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
- name: Determine tag
id: tag
run: |
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
else
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
fi
- name: Login to Forgejo registry
uses: docker/login-action@v3
with:
registry: git.fabledsword.com
username: ${{ github.actor }}
password: ${{ secrets.RELEASE_TOKEN }}
- name: Build and push agent image
uses: docker/build-push-action@v5
with:
context: agent
file: agent/Dockerfile
push: true
tags: ${{ steps.tag.outputs.tags }}
-27
View File
@@ -1,27 +0,0 @@
# FabledCurator GPU agent — runs on the desktop with the GPU.
# CUDA + cuDNN runtime so onnxruntime-gpu can use the card (it needs cuDNN 9 —
# the plain -runtime image lacks it: "libcudnn.so.9: cannot open shared object
# file"); ffmpeg for video frames.
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
ENV DEBIAN_FRONTEND=noninteractive PYTHONUNBUFFERED=1
RUN apt-get update \
&& apt-get install -y --no-install-recommends python3 python3-pip ffmpeg \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# torch from the CUDA-12.4 wheel index (matches the base image); its wheels
# bundle their own CUDA + cuDNN and coexist with onnxruntime-gpu. Installed
# first + separately so the GPU build of torch is deterministic and layer-cached.
RUN pip3 install --no-cache-dir torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
COPY fc_agent ./fc_agent
# imgutils ONNX models + the transformers SigLIP weights both cache here; mount
# a volume to persist them across restarts (the SigLIP download is ~3.5 GB once).
ENV HF_HOME=/models
EXPOSE 8770
# The control UI; the worker is started from it (or POST /start).
CMD ["uvicorn", "fc_agent.app:app", "--host", "0.0.0.0", "--port", "8770"]
-71
View File
@@ -1,71 +0,0 @@
# FabledCurator GPU agent
A desktop-GPU worker that embeds characters (CCIP) + figure crops for
FabledCurator. It talks to FC **only over HTTP** — it leases jobs, fetches image
pixels, runs the models on your GPU, and posts results back. Your FC database and
Redis stay private; the agent never touches them.
You run it when you want a burst and stop it to reclaim the card.
## 0. Host prerequisite — NVIDIA Container Toolkit
Docker needs the toolkit to hand the GPU to a container (else: *"could not select
device driver nvidia with capabilities [[gpu]]"*). On Arch/CachyOS:
```sh
sudo pacman -S nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# verify:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
## 1. Get a token
In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.
## 2. Pull (CI publishes it alongside the web/ml images)
```sh
docker pull git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
```
> Local build for development instead: `docker build -t fc-gpu-agent agent/`
## 3. Run (on the machine with the GPU)
```sh
docker run --rm --gpus all -p 8770:8770 \
-e FC_URL=http://curator.traefik.internal \
-e FC_TOKEN=<paste-the-token> \
-v fc-agent-models:/models \
git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
```
Then open <http://localhost:8770> — the control page. Click **Start** to begin
draining the queue; **Pause**/**Stop** to yield the GPU. The `-v fc-agent-models`
volume caches the downloaded ONNX models so restarts are fast.
Kick off a backfill from FC (**GPU agent card → Queue character embedding**), then
watch the queue counts on the control page (or FC's card) drain.
## Config (env)
| var | default | meaning |
|---|---|---|
| `FC_URL` | `http://localhost:8000` | FC base URL |
| `FC_TOKEN` | — | the bearer token (required) |
| `AGENT_ID` | `desktop-agent` | identifies this agent's leases |
| `BATCH_SIZE` | `4` | jobs leased per round (still processed one at a time) |
| `CCIP_MODEL` | imgutils default | CCIP model name |
| `DETECTOR_LEVEL` | `m` | person-detector size: `n` < `s` < `m` < `x` |
| `POLL_IDLE_SECONDS` | `10` | wait between empty leases |
## ⚠️ Verify on first run
This part can't be CI-tested (no GPU/models in CI), so confirm against your
installed `dghs-imgutils` (`pip show dghs-imgutils`) — see `fc_agent/models.py`:
- `imgutils.detect.detect_person(image, level=...)` returns
`[((x0,y0,x1,y1), label, score), ...]`.
- `imgutils.metrics.ccip_extract_feature(image, model=...)` returns a vector
(768-d for caformer). If you want the F1-0.94 variant, set
`CCIP_MODEL=ccip-caformer_b36-24` (verify the exact string in imgutils).
If FC's matcher under/over-fires, tune the cosine threshold in
`backend/app/services/ml/ccip.py` (`DEFAULT_SIM_THRESHOLD`) and use
`GET /api/ccip/overview` + `/api/ccip/images/<id>` to spot-check.
## CPU fallback
Swap `onnxruntime-gpu``onnxruntime` in `requirements.txt` and drop `--gpus all`
to grind it slowly on the server instead. Same agent, no card.
-53
View File
@@ -1,53 +0,0 @@
# FabledCurator GPU agent — desktop run via docker compose.
#
# Usage:
# 1. Generate a token: FC → Settings → Tagging → GPU agent → Generate token.
# 2. Create a .env next to this file:
# FC_URL=http://curator.traefik.internal
# FC_TOKEN=<paste-the-token>
# # optional: CCIP_MODEL=ccip-caformer_b36-24 (the F1-0.94 variant)
# 3. docker compose up -d (pulls the published image)
# 4. Open http://localhost:8770 → Start. Pause/Stop hands the GPU back.
# docker compose down to stop the container entirely.
#
# Surviving a curator redeploy (you're away, can't touch the agent):
# - A running agent rides out curator being unreachable on its own — it retries
# leasing with capped backoff and resumes when the server is back. In-flight
# work is handed back (not failed), so a redeploy never poisons good jobs.
# - AUTO_START=1 (below) also resumes the worker if the AGENT container itself
# restarts (host reboot / crash via `restart: unless-stopped`) — no click.
#
# Needs the NVIDIA Container Toolkit installed on the host for --gpus.
services:
fc-gpu-agent:
image: git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
pull_policy: always
ports:
- "8770:8770"
environment:
FC_URL: ${FC_URL:-http://curator.traefik.internal}
FC_TOKEN: ${FC_TOKEN:?set FC_TOKEN in .env (FC → GPU agent → Generate token)}
CCIP_MODEL: ${CCIP_MODEL:-}
DETECTOR_LEVEL: ${DETECTOR_LEVEL:-m}
BATCH_SIZE: ${BATCH_SIZE:-4}
# Resume the worker automatically on container start (survive a reboot /
# crash-restart while you're away). Set to 0 to require a manual Start.
AUTO_START: ${AUTO_START:-1}
# Crop embedder (SigLIP concept bag): float16 keeps VRAM low on a shared
# desktop GPU; the model itself is announced by the server.
SIGLIP_DTYPE: ${SIGLIP_DTYPE:-float16}
volumes:
# Persist the downloaded ONNX models so restarts are fast.
- fc-agent-models:/models
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
fc-agent-models:
View File
-134
View File
@@ -1,134 +0,0 @@
"""FastAPI control surface for the agent (served on localhost).
Start / stop the worker pool, tune the worker count live (trades desktop
responsiveness for throughput), and watch GPU load + progress + the server-side
queue. Config is env-seeded; the worker count is adjustable here on the fly.
"""
from fastapi import FastAPI, Request
from fastapi.responses import HTMLResponse, JSONResponse
from .config import Config
from .gpu import read_gpu
from .worker import Worker
cfg = Config.from_env()
worker = Worker(cfg)
app = FastAPI(title="FabledCurator GPU agent")
@app.on_event("startup")
def _maybe_autostart() -> None:
# With AUTO_START set, a container restart (host reboot, or `restart:
# unless-stopped` after a crash) resumes the worker on its own — the slots
# then ride out a still-down curator via lease backoff. Lets the agent
# survive a redeploy with nobody at the desktop to click Start.
if cfg.auto_start and cfg.token:
worker.start()
@app.get("/", response_class=HTMLResponse)
def index() -> str:
return _PAGE
@app.post("/start")
def start():
worker.start()
return JSONResponse(worker.status())
@app.post("/stop")
def stop():
worker.stop()
return JSONResponse(worker.status())
@app.post("/concurrency")
async def concurrency(request: Request):
body = await request.json()
worker.set_concurrency(int(body.get("value", 1)))
return JSONResponse(worker.status())
@app.get("/status")
def status():
s = worker.status()
s["fc_url"] = cfg.fc_url
s["configured"] = bool(cfg.token)
s["gpu"] = read_gpu()
try:
s["queue"] = worker.client.queue_status()
except Exception:
s["queue"] = None
return JSONResponse(s)
_PAGE = """<!doctype html><html><head><meta charset=utf-8>
<title>FabledCurator GPU agent</title>
<style>
body{font:14px system-ui;margin:2rem;max-width:680px;background:#14171a;color:#e8e8e8}
h1{font-size:18px} button{font:14px system-ui;padding:.5rem 1rem;border:0;border-radius:6px;
margin-right:.5rem;cursor:pointer;color:#fff} .start{background:#2e7d32}.stop{background:#b3261e}
.step{background:#33373b;padding:.4rem .7rem;font-weight:700}
.stat{display:inline-block;margin-right:1.5rem;vertical-align:top}
.n{font-size:22px;font-weight:700} code{background:#222;padding:2px 6px;border-radius:4px}
.q,.gpu{margin-top:1rem;color:#9aa} .bar{height:8px;border-radius:4px;background:#222;overflow:hidden;
max-width:320px;margin-top:4px} .bar>i{display:block;height:100%;background:#3f7d3f}
.row{margin:.8rem 0}
</style></head><body>
<h1>FabledCurator GPU agent</h1>
<p>FC: <code id=fc>—</code> · token <code id=cfg>—</code></p>
<div class=row>
<button class=start onclick=act('start')>Start</button>
<button class=stop onclick=act('stop')>Stop</button>
</div>
<div class=row>
workers
<button class=step onclick=setc(-1)></button>
<input id=conc type=number min=1 value=1
style="width:3.5rem;font:700 16px system-ui;text-align:center;background:#222;color:#e8e8e8;border:1px solid #444;border-radius:6px;padding:.3rem"
onchange="setv(this.value)">
<button class=step onclick=setc(1)>+</button>
<span class=cap style=color:#9aa>(more = overlap I/O, fill the GPU) max <b id=capn>8</b></span>
</div>
<div class=row>
<span class=stat><span class=n id=state>stopped</span><br>state</span>
<span class=stat><span class=n id=active>0</span><br>active now</span>
<span class=stat><span class=n id=done>0</span><br>processed</span>
<span class=stat><span class=n id=err>0</span><br>errors</span>
<span class=stat><span class=n id=wait>0</span><br>waited out</span>
</div>
<div id=banner style="display:none;margin:.6rem 0;padding:.5rem .8rem;border-radius:6px;background:#5a4a17;color:#ffe28a">
curator unreachable — holding work + retrying, will resume on its own (no restart needed)
</div>
<div class=gpu id=gpu>GPU — …</div>
<div class=bar><i id=gpubar style=width:0%></i></div>
<div class=q id=queue></div>
<script>
let CAP=8
async function act(p){await fetch('/'+p,{method:'POST'});refresh()}
function setc(d){ setv((parseInt(conc.value||'1'))+d) }
async function setv(v){
v=Math.max(1,Math.min(CAP,parseInt(v)||1)); conc.value=v
await fetch('/concurrency',{method:'POST',headers:{'Content-Type':'application/json'},
body:JSON.stringify({value:v})});refresh()
}
async function refresh(){
const s=await (await fetch('/status')).json()
CAP=s.max_concurrency||8; capn.textContent=CAP
state.textContent=s.state; active.textContent=s.active; done.textContent=s.processed
err.textContent=s.errors; fc.textContent=s.fc_url; wait.textContent=s.transient||0
// Running but the queue read failed → curator is unreachable; show we're
// riding it out rather than erroring.
banner.style.display=(s.state==='running' && !s.queue)?'block':'none'
if(document.activeElement!==conc) conc.value=s.concurrency
conc.max=CAP
cfg.textContent=s.configured?'set':'MISSING'
if(s.gpu){
gpu.textContent=`GPU — ${s.gpu.util_pct}% util · VRAM ${s.gpu.mem_used_mb}/${s.gpu.mem_total_mb} MB · ${s.gpu.temp_c}°C`
gpubar.style.width=Math.round(100*s.gpu.mem_used_mb/s.gpu.mem_total_mb)+'%'
} else { gpu.textContent='GPU — n/a (CPU fallback?)'; gpubar.style.width='0%' }
queue.textContent=s.queue?`queue — pending ${s.queue.pending} · in flight ${s.queue.leased} · done ${s.queue.done} · errored ${s.queue.error}`:'queue — unreachable'
}
refresh(); setInterval(refresh,3000)
</script></body></html>"""
-85
View File
@@ -1,85 +0,0 @@
"""HTTP client for the FabledCurator GPU-job API.
The agent's ONLY contact with FC — lease/submit/heartbeat/fail + fetch image
bytes, all over HTTP with the bearer token. No DB/Redis.
"""
import requests
from requests.adapters import HTTPAdapter
class FcClient:
def __init__(self, base_url: str, token: str, agent_id: str):
self.base = base_url.rstrip("/")
self.agent_id = agent_id
self.s = requests.Session()
self.s.headers["Authorization"] = f"Bearer {token}"
# Many worker threads share this Session; the default pool (10) would
# throttle them + spam "connection pool is full". Size it for the cap.
adapter = HTTPAdapter(pool_connections=64, pool_maxsize=64)
self.s.mount("http://", adapter)
self.s.mount("https://", adapter)
def lease(self, batch_size: int) -> list[dict]:
r = self.s.post(
f"{self.base}/api/gpu/jobs/lease",
json={"agent_id": self.agent_id, "batch_size": batch_size},
timeout=30,
)
r.raise_for_status()
return r.json().get("jobs", [])
def submit(self, job_id: int, regions: list[dict], replace_kinds: list[str]) -> dict:
r = self.s.post(
f"{self.base}/api/gpu/jobs/submit",
json={
"agent_id": self.agent_id, "job_id": job_id,
"regions": regions, "replace_kinds": replace_kinds,
},
timeout=120,
)
r.raise_for_status()
return r.json()
def heartbeat(self, job_ids: list[int]) -> None:
try:
self.s.post(
f"{self.base}/api/gpu/jobs/heartbeat",
json={"agent_id": self.agent_id, "job_ids": job_ids},
timeout=30,
)
except requests.RequestException:
pass
def fail(self, job_id: int, error: str) -> None:
try:
self.s.post(
f"{self.base}/api/gpu/jobs/fail",
json={"agent_id": self.agent_id, "job_id": job_id, "error": error},
timeout=30,
)
except requests.RequestException:
pass
def release(self, job_ids: list[int]) -> None:
# Graceful hand-back on stop so orphaned work is re-leased at once.
if not job_ids:
return
try:
self.s.post(
f"{self.base}/api/gpu/jobs/release",
json={"agent_id": self.agent_id, "job_ids": job_ids},
timeout=30,
)
except requests.RequestException:
pass
def fetch_image(self, image_url: str) -> bytes:
# image_url is a server-relative path ("/images/...").
r = self.s.get(f"{self.base}{image_url}", timeout=180)
r.raise_for_status()
return r.content
def queue_status(self) -> dict:
r = self.s.get(f"{self.base}/api/gpu/status", timeout=15)
r.raise_for_status()
return r.json()
-36
View File
@@ -1,36 +0,0 @@
"""Agent config, all from env (the control container is configured at run)."""
import os
from dataclasses import dataclass
@dataclass
class Config:
fc_url: str # base URL of the FabledCurator web service
token: str # the bearer token from Settings → Tagging → GPU agent
agent_id: str # identifies this agent's leases
batch_size: int # jobs a worker leases per round
concurrency: int # INITIAL parallel workers (tunable live from the UI)
ccip_model: str # imgutils CCIP model name ("" → imgutils default)
detector_level: str # imgutils person-detector level: n|s|m|x
poll_idle_seconds: float # wait between empty leases
embed_dtype: str # torch dtype for the crop embedder: float16|float32
embed_model_override: str # force a SigLIP-family model ("" → use the one
# the server announces in the lease)
auto_start: bool # start the worker pool on boot (so a container restart
# resumes processing without anyone clicking Start)
@classmethod
def from_env(cls) -> "Config":
return cls(
fc_url=os.environ.get("FC_URL", "http://localhost:8000").rstrip("/"),
token=os.environ.get("FC_TOKEN", ""),
agent_id=os.environ.get("AGENT_ID", "desktop-agent"),
batch_size=int(os.environ.get("BATCH_SIZE", "4")),
concurrency=int(os.environ.get("CONCURRENCY", "1")),
ccip_model=os.environ.get("CCIP_MODEL", ""),
detector_level=os.environ.get("DETECTOR_LEVEL", "m"),
poll_idle_seconds=float(os.environ.get("POLL_IDLE_SECONDS", "10")),
embed_dtype=os.environ.get("SIGLIP_DTYPE", "float16"),
embed_model_override=os.environ.get("EMBED_MODEL_NAME", ""),
auto_start=os.environ.get("AUTO_START", "").lower() in ("1", "true", "yes"),
)
-36
View File
@@ -1,36 +0,0 @@
"""Crop primitive — vendored from backend/app/services/ml/crops.py so the agent
is self-contained. Keep in sync if the floor logic changes."""
from PIL import Image
MIN_CROP_FRACTION = 0.10
MIN_CROP_PX = 64
def crop_region(
img: Image.Image,
bbox: tuple[float, float, float, float],
*,
pad: float = 0.0,
min_fraction: float = MIN_CROP_FRACTION,
min_px: int = MIN_CROP_PX,
) -> Image.Image | None:
"""Crop a NORMALIZED bbox (x, y, w, h in [0,1]); None if below the size
floor (max of a fraction-of-short-side and an absolute pixel floor)."""
iw, ih = img.size
x, y, w, h = bbox
px, py, pw, ph = x * iw, y * ih, w * iw, h * ih
if pad:
px -= pw * pad / 2.0
py -= ph * pad / 2.0
pw *= (1.0 + pad)
ph *= (1.0 + pad)
left = max(0, int(round(px)))
top = max(0, int(round(py)))
right = min(iw, int(round(px + pw)))
bottom = min(ih, int(round(py + ph)))
if right <= left or bottom <= top:
return None
floor = max(min_px, int(min_fraction * min(iw, ih)))
if min(right - left, bottom - top) < floor:
return None
return img.crop((left, top, right, bottom)).convert("RGB")
-69
View File
@@ -1,69 +0,0 @@
"""Crop EMBEDDER for the concept bag — model-agnostic (CLIP/SigLIP-family).
The server trains its per-concept heads in the embedding space of whatever model
its `embedder_model_version` names; a crop must be embedded with the SAME model
or its vector lands in a different coordinate system and every head misfires. So
the model identity (HF name + version) is ANNOUNCED BY THE SERVER in the lease —
nothing here is hardcoded to SigLIP. Whatever name the server sends is loaded via
transformers `get_image_features` (the CLIP/SigLIP-family image-tower call); a
non-CLIP backbone (e.g. a DINO encoder) would need its own pooling adapter.
torch on CUDA, fp16 by default to keep VRAM low on a shared desktop GPU — the
tiny fp16-vs-fp32 difference is negligible for the linear heads (cosine ~0.999).
A single inference lock serializes the forward pass: the pipeline is I/O-bound,
so the GPU isn't the bottleneck, and one model shared across worker threads is
safest behind a lock.
"""
import threading
import numpy as np
from PIL import Image
class CropEmbedder:
def __init__(self, model_name: str, dtype: str = "float16"):
self._name = model_name
self._dtype_name = dtype
self._model = None
self._processor = None
self._torch = None
self._device = None
self._dt = None
self._load_lock = threading.Lock()
self._infer_lock = threading.Lock()
@property
def model_name(self) -> str:
return self._name
def load(self) -> None:
if self._model is not None:
return
with self._load_lock:
if self._model is not None:
return
import torch
from transformers import AutoImageProcessor, AutoModel
self._torch = torch
self._device = "cuda" if torch.cuda.is_available() else "cpu"
dt = getattr(torch, self._dtype_name, torch.float16)
if self._device == "cpu":
dt = torch.float32 # fp16 matmul is unsupported/slow on CPU
self._dt = dt
self._processor = AutoImageProcessor.from_pretrained(self._name)
model = AutoModel.from_pretrained(self._name, torch_dtype=dt)
model.eval().to(self._device)
self._model = model
def embed(self, image: Image.Image) -> list[float]:
"""A crop → its embedding as a plain float list, ready to POST."""
self.load()
torch = self._torch
enc = self._processor(images=image, return_tensors="pt")
pixel_values = enc["pixel_values"].to(self._device, self._dt)
with self._infer_lock, torch.no_grad():
out = self._model.get_image_features(pixel_values=pixel_values)
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
vec = pooled[0].float().cpu().numpy().astype(np.float32).reshape(-1)
return vec.tolist()
-30
View File
@@ -1,30 +0,0 @@
"""GPU load readout via nvidia-smi (present in the container thanks to the
NVIDIA Container Toolkit's `utility` capability). Returns None if unavailable —
the UI just shows n/a (e.g. CPU-fallback run)."""
import subprocess
def read_gpu() -> dict | None:
try:
out = subprocess.run(
[
"nvidia-smi",
"--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
"--format=csv,noheader,nounits",
],
capture_output=True, text=True, timeout=5, check=True,
).stdout.strip().splitlines()
except (OSError, subprocess.SubprocessError):
return None
if not out:
return None
parts = [p.strip() for p in out[0].split(",")]
try:
return {
"util_pct": int(float(parts[0])),
"mem_used_mb": int(float(parts[1])),
"mem_total_mb": int(float(parts[2])),
"temp_c": int(float(parts[3])),
}
except (ValueError, IndexError):
return None
-63
View File
@@ -1,63 +0,0 @@
"""Image + video handling. Stills load directly; videos are sampled into frames
(ffmpeg) at the cadence FC sends — so a video becomes a bag of per-frame
instances, each with a timestamp."""
import io
import os
import subprocess
import tempfile
from PIL import Image
def is_video(mime: str) -> bool:
return bool(mime) and (mime.startswith("video/") or mime in {"image/gif"})
def to_rgb(img: Image.Image) -> Image.Image:
"""RGB, flattening any transparency onto white first. A naive convert('RGB')
on a palette-with-transparency image (common for character PNGs on a clear
background) lets PIL guess the transparent pixels — usually black artifacts
that bleed into the crop + the embedding (and the "should be converted to
RGBA" warning). Compositing over white gives a clean, consistent background."""
if img.mode in ("RGBA", "LA", "PA") or (
img.mode == "P" and "transparency" in img.info
):
img = img.convert("RGBA")
bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
return Image.alpha_composite(bg, img).convert("RGB")
return img.convert("RGB")
def load_image(data: bytes) -> Image.Image:
return to_rgb(Image.open(io.BytesIO(data)))
def sample_frames(
data: bytes, interval_seconds: float, max_frames: int
) -> list[tuple[float, Image.Image]]:
"""Extract up to max_frames frames at one-every-interval_seconds via ffmpeg.
Returns [(timestamp_seconds, frame)]. Empty on failure (caller falls back)."""
interval = max(0.5, float(interval_seconds or 4.0))
cap = max(1, int(max_frames or 64))
with tempfile.TemporaryDirectory() as tmp:
src = os.path.join(tmp, "in")
with open(src, "wb") as fh:
fh.write(data)
pattern = os.path.join(tmp, "f_%05d.jpg")
try:
subprocess.run(
[
"ffmpeg", "-nostdin", "-loglevel", "error", "-i", src,
"-vf", f"fps=1/{interval}", "-frames:v", str(cap),
"-q:v", "3", pattern,
],
check=True, timeout=600,
)
except (subprocess.SubprocessError, FileNotFoundError):
return []
out: list[tuple[float, Image.Image]] = []
names = sorted(n for n in os.listdir(tmp) if n.startswith("f_"))
for i, name in enumerate(names[:cap]):
with Image.open(os.path.join(tmp, name)) as im:
out.append((round(i * interval, 2), to_rgb(im)))
return out
-39
View File
@@ -1,39 +0,0 @@
"""imgutils model wrappers — the figure DETECTOR + the CCIP EMBEDDER.
⚠️ VERIFY ON FIRST RUN: the exact imgutils function names/signatures + the CCIP
model string can drift between dghs-imgutils releases. These are the two seams to
check against your installed version (`pip show dghs-imgutils`):
- detect_person(image, level=...) -> [((x0,y0,x1,y1), label, score), ...]
- ccip_extract_feature(image, model=...) -> a vector (768-d for caformer)
imgutils auto-downloads the ONNX models from HuggingFace on first use; GPU is
used when onnxruntime-gpu is installed.
"""
import numpy as np
from PIL import Image
def detect_figures(image: Image.Image, level: str = "m") -> list[tuple[tuple, float | None]]:
"""Person/figure bounding boxes, NORMALIZED (x, y, w, h in [0,1]) + score.
Returns [] if detection finds nothing (caller falls back to whole-image)."""
from imgutils.detect import detect_person
iw, ih = image.size
out = []
for (x0, y0, x1, y1), _label, score in detect_person(image, level=level):
out.append((
(x0 / iw, y0 / ih, (x1 - x0) / iw, (y1 - y0) / ih),
float(score),
))
return out
def ccip_vector(image: Image.Image, model: str | None = None) -> list[float]:
"""The CCIP identity embedding of a (cropped) character image, as a plain
float list ready to POST."""
from imgutils.metrics import ccip_extract_feature
feat = (
ccip_extract_feature(image, model=model)
if model else ccip_extract_feature(image)
)
return np.asarray(feat, dtype=np.float32).reshape(-1).tolist()
-274
View File
@@ -1,274 +0,0 @@
"""The lease → fetch → detect+embed → submit loop, run by a pool of worker
slots whose count is tunable live from the UI.
Each slot is an independent loop (its own leases; the server's SKIP-LOCKED lease
keeps them from colliding). More slots = more GPU load + throughput; the model is
loaded once and shared, so slots add concurrent inference, not N× model VRAM.
That's the dial the operator turns to trade desktop responsiveness for speed.
Stop (or shrinking the pool) RELEASES a slot's still-leased jobs immediately so
orphaned work is re-picked at once rather than waiting out the lease.
"""
import threading
import requests
from . import media, models
from .client import FcClient
from .config import Config
from .crops import crop_region
# Cap on the lease-retry backoff: when curator is unreachable (e.g. you redeploy
# it while away), each slot retries leasing with exponential backoff up to this
# many seconds, then resumes within this window once the server is back — no
# restart needed.
MAX_BACKOFF_SECONDS = 60.0
def _is_transient(exc: "requests.RequestException") -> bool:
"""A server/transport problem (wait it out) vs a job-specific fault (fail it).
No response → connection refused/timeout → curator is down → transient. With
a response: 5xx, auth (401/403, e.g. a token blip on redeploy), 408/409/429
(timeout / our lease reclaimed / rate-limited) are all 'not this job's fault'.
A specific 4xx like 404 (image gone) / 400 IS the job's fault → fail it."""
resp = getattr(exc, "response", None)
if resp is None:
return True
return resp.status_code >= 500 or resp.status_code in (401, 403, 408, 409, 429)
# Generous cap: the pipeline is usually I/O-bound (downloading + decoding images
# over HTTP), so the GPU stays underused until many workers overlap that I/O.
# Push it up while watching the GPU util + VRAM in the UI.
MAX_CONCURRENCY = 32
# Fallbacks only — the server ANNOUNCES the embedding model (name + version) in
# the lease so the agent stays model-agnostic and in lock-step with the space
# the heads were trained in. These cover an older server that doesn't send them.
DEFAULT_EMBED_MODEL = "google/siglip-so400m-patch14-384"
DEFAULT_EMBED_VERSION = "siglip-so400m-patch14-384"
class _Slot:
"""One worker loop. `inflight` = jobs leased but not yet processed, so a
graceful stop can hand them back."""
__slots__ = ("stop", "inflight")
def __init__(self):
self.stop = threading.Event()
self.inflight: list[int] = []
class Worker:
def __init__(self, cfg: Config):
self.cfg = cfg
self.client = FcClient(cfg.fc_url, cfg.token, cfg.agent_id)
self._lock = threading.Lock()
self._running = False
self._target = max(1, min(MAX_CONCURRENCY, cfg.concurrency))
self._slots: list[_Slot] = []
self.processed = 0
self.errors = 0
self.transient = 0 # jobs handed back due to a server outage (NOT
# failed) — the "waiting out curator" counter
self._active = 0 # slots currently mid-image
# The crop embedder (SigLIP-family) is built lazily on the first job that
# needs it, from the model the server announces — one shared instance.
self._embedder = None
self._embedder_lock = threading.Lock()
# --- control -----------------------------------------------------------
def start(self):
with self._lock:
self._running = True
self._reconcile_locked()
def stop(self):
with self._lock:
self._running = False
slots, self._slots = self._slots, []
for s in slots:
s.stop.set() # each slot releases its inflight on exit
def set_concurrency(self, n: int):
with self._lock:
self._target = max(1, min(MAX_CONCURRENCY, int(n)))
if self._running:
self._reconcile_locked()
def _reconcile_locked(self):
while len(self._slots) < self._target:
slot = _Slot()
self._slots.append(slot)
threading.Thread(target=self._loop, args=(slot,), daemon=True).start()
while len(self._slots) > self._target:
self._slots.pop().stop.set()
def status(self) -> dict:
with self._lock:
return {
"state": "running" if self._running else "stopped",
"concurrency": self._target,
"max_concurrency": MAX_CONCURRENCY,
"workers": len(self._slots),
"active": self._active,
"processed": self.processed,
"errors": self.errors,
"transient": self.transient,
}
def _bump(self, *, processed=0, errors=0, active=0, transient=0):
with self._lock:
self.processed += processed
self.errors += errors
self.transient += transient
self._active += active
# --- per-slot loop -----------------------------------------------------
def _loop(self, slot: _Slot):
backoff = self.cfg.poll_idle_seconds
while not slot.stop.is_set() and self._running:
try:
jobs = self.client.lease(self.cfg.batch_size)
backoff = self.cfg.poll_idle_seconds # server answered → reset
except Exception:
# curator unreachable (redeploy, network drop): wait it out with
# exponential backoff, capped — resume on our own when it returns.
self._interruptible_sleep(slot, backoff)
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
continue
if not jobs:
self._interruptible_sleep(slot, self.cfg.poll_idle_seconds)
continue
slot.inflight = [j["job_id"] for j in jobs]
for job in jobs:
if slot.stop.is_set() or not self._running:
break
ok = self._process(job)
slot.inflight = [i for i in slot.inflight if i != job["job_id"]]
if not ok:
# Server went away mid-batch: hand the rest back (best effort)
# and back off instead of hammering a recovering server or
# burning the jobs' attempt budgets on fail().
if slot.inflight:
self.client.release(slot.inflight)
slot.inflight = []
self._interruptible_sleep(slot, backoff)
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
break
if slot.inflight:
self.client.heartbeat(slot.inflight)
# Graceful hand-back of anything leased but not processed.
if slot.inflight:
self.client.release(slot.inflight)
slot.inflight = []
def _interruptible_sleep(self, slot: _Slot, seconds: float):
"""Sleep, but wake immediately if the slot is told to stop — so a Stop or
a pool-shrink doesn't hang for a full backoff window."""
slot.stop.wait(timeout=seconds)
def _ensure_embedder(self, model_name: str):
if self._embedder is not None:
return self._embedder
with self._embedder_lock:
if self._embedder is None:
from .embedder import CropEmbedder
self._embedder = CropEmbedder(model_name, self.cfg.embed_dtype)
return self._embedder
def _process(self, job: dict) -> bool:
"""Process one job. Returns True when handled (completed, or hard-failed
because the job itself is bad) and False on a TRANSPORT error (curator
unreachable / 5xx / our lease was reclaimed mid-flight) — which is not
the job's fault, so the caller backs off and the job is left to be
re-leased rather than fail()ed into its attempt budget."""
self._bump(active=1)
try:
data = self.client.fetch_image(job["image_url"])
if media.is_video(job.get("mime", "")):
frames = media.sample_frames(
data, job.get("frame_interval_seconds", 4.0),
job.get("max_frames", 64),
) or [(None, media.load_image(data))]
else:
frames = [(None, media.load_image(data))]
# task picks what to produce per crop:
# 'siglip' (backfill existing images) → concept (SigLIP) regions
# ONLY, so it never churns their figure/CCIP regions or the
# character-reference cache.
# 'ccip' / 'both' (a new image's first pass) → figure (CCIP) AND
# concept (SigLIP) in one go, off the same crop.
task = job.get("task") or "ccip"
want_ccip = task in ("ccip", "both")
want_siglip = task in ("ccip", "siglip", "both")
replace_kinds = (
["concept"] if task == "siglip" else ["figure", "face", "concept"]
)
embed_version = job.get("embed_version") or DEFAULT_EMBED_VERSION
embedder = None
if want_siglip:
model_name = (
self.cfg.embed_model_override
or job.get("embed_model_name")
or DEFAULT_EMBED_MODEL
)
embedder = self._ensure_embedder(model_name)
regions = []
ccip_ev = self.cfg.ccip_model or "ccip-default"
dv = f"person-{self.cfg.detector_level}"
for t, frame in frames:
figs = models.detect_figures(frame, self.cfg.detector_level)
if not figs:
figs = [((0.0, 0.0, 1.0, 1.0), None)] # whole-frame fallback
for bbox, score in figs:
crop = crop_region(frame, bbox)
if crop is None:
continue
if want_ccip:
regions.append({
"kind": "figure",
"bbox": list(bbox),
"frame_time": t,
"score": score,
"ccip_embedding": models.ccip_vector(
crop, self.cfg.ccip_model or None
),
"embedding_version": ccip_ev,
"detector_version": dv,
})
if want_siglip:
regions.append({
"kind": "concept",
"bbox": list(bbox),
"frame_time": t,
"score": score,
"siglip_embedding": embedder.embed(crop),
"embedding_version": embed_version,
"detector_version": dv,
})
self.client.submit(job["job_id"], regions, replace_kinds)
self._bump(processed=1)
return True
except requests.RequestException as exc:
if _is_transient(exc):
# curator down/redeploying, a 5xx, or our lease was reclaimed
# while we worked. NOT the job's fault — hand it back (best
# effort; no-ops if the server is still down, then the server's
# orphan-recovery reclaims it) and signal the loop to wait.
self._bump(transient=1)
self.client.release([job["job_id"]])
return False
# A job-specific HTTP fault (404 image gone, 400) → fail it so it
# doesn't re-lease forever.
self._bump(errors=1)
self.client.fail(job["job_id"], str(exc)[:500])
return True
except Exception as exc: # noqa: BLE001 — a genuine job fault: report it
self._bump(errors=1)
self.client.fail(job["job_id"], str(exc)[:500])
return True
finally:
self._bump(active=-1)
-15
View File
@@ -1,15 +0,0 @@
# CCIP + figure detection (ONNX models, auto-downloaded from HuggingFace).
dghs-imgutils>=0.4
# GPU inference for the ONNX models. Swap to onnxruntime (CPU) for a slow
# server-side fallback run.
onnxruntime-gpu
# The crop EMBEDDER (concept bag). torch is installed separately in the
# Dockerfile from the CUDA-12.4 wheel index so the GPU build is deterministic;
# transformers loads whatever SigLIP-family model the server announces.
transformers>=4.45
# Control surface + HTTP.
fastapi
uvicorn[standard]
requests
pillow
numpy
@@ -1,55 +0,0 @@
"""image_provenance: from_attachment_id (which archive an image was extracted from)
Milestone #87. When an image is pulled out of a .zip/.rar, record WHICH archive
PostAttachment it came from, so the provenance UI can show the single archive a
file lives inside instead of every attachment on the post. Nullable FK with
ON DELETE SET NULL — a loose (non-archive) download leaves it NULL, and deleting
the archive attachment forgets the linkage without destroying the (image, post)
provenance edge. Existing rows are NULL until the reextract backfill stamps them.
Revision ID: 0055
Revises: 0054
Create Date: 2026-06-22
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0055"
down_revision: Union[str, None] = "0054"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"image_provenance",
sa.Column("from_attachment_id", sa.Integer(), nullable=True),
)
op.create_index(
"ix_image_provenance_from_attachment_id",
"image_provenance",
["from_attachment_id"],
)
op.create_foreign_key(
"fk_image_provenance_from_attachment",
"image_provenance",
"post_attachment",
["from_attachment_id"],
["id"],
ondelete="SET NULL",
)
def downgrade() -> None:
op.drop_constraint(
"fk_image_provenance_from_attachment",
"image_provenance",
type_="foreignkey",
)
op.drop_index(
"ix_image_provenance_from_attachment_id",
table_name="image_provenance",
)
op.drop_column("image_provenance", "from_attachment_id")
-43
View File
@@ -1,43 +0,0 @@
"""tag_eval_run: persisted head-vs-centroid tagging eval runs (#1130)
Milestone #114 slice 1. A long ml-queue eval whose full report must SURVIVE
navigation, so the run + report live in a row the admin card rehydrates from
(mirrors library_audit_run). running -> ready / error.
Revision ID: 0056
Revises: 0055
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0056"
down_revision: Union[str, None] = "0055"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"tag_eval_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("params", JSONB(), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False, server_default="running"),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("report", JSONB(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_tag_eval_run_status", "tag_eval_run", ["status"])
def downgrade() -> None:
op.drop_index("ix_tag_eval_run_status", table_name="tag_eval_run")
op.drop_table("tag_eval_run")
@@ -1,40 +0,0 @@
"""tag_positive_confirmation: operator-affirmed correct positives (#1130)
Mirror of tag_suggestion_rejection. "Keep" on a doubted positive records here so
the eval's doubts list stops resurfacing confirmed-correct images every run.
Revision ID: 0057
Revises: 0056
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0057"
down_revision: Union[str, None] = "0056"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"tag_positive_confirmation",
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True,
),
sa.Column(
"confirmed_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
def downgrade() -> None:
op.drop_table("tag_positive_confirmation")
-95
View File
@@ -1,95 +0,0 @@
"""tag_head + head_training_run: production heads that learn from tags (#114)
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its
production form. tag_head stores one logistic-regression head per concept (the
new suggestion source, replacing Camie + centroid); head_training_run tracks the
batch that (re)trains them. Adds two head-training tunables to ml_settings.
Revision ID: 0058
Revises: 0057
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0058"
down_revision: Union[str, None] = "0057"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_HEAD_DIM = 1152
def upgrade() -> None:
op.create_table(
"tag_head",
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column("embedding_version", sa.String(length=128), nullable=False),
sa.Column("weights", Vector(_HEAD_DIM), nullable=False),
sa.Column("bias", sa.Float(), nullable=False),
sa.Column("suggest_threshold", sa.Float(), nullable=False),
sa.Column("auto_apply_threshold", sa.Float(), nullable=True),
sa.Column("n_pos", sa.Integer(), nullable=False),
sa.Column("n_neg", sa.Integer(), nullable=False),
sa.Column("ap", sa.Float(), nullable=False),
sa.Column("precision_cv", sa.Float(), nullable=False),
sa.Column("recall", sa.Float(), nullable=False),
sa.Column(
"trained_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("metrics", JSONB(), nullable=True),
)
op.create_table(
"head_training_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("params", JSONB(), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("n_trained", sa.Integer(), nullable=True),
sa.Column("n_skipped", sa.Integer(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index(
"ix_head_training_run_status", "head_training_run", ["status"],
)
# Head-training tunables on the ml_settings singleton.
op.add_column(
"ml_settings",
sa.Column(
"head_min_positives", sa.Integer(), nullable=False,
server_default="8",
),
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_precision", sa.Float(), nullable=False,
server_default="0.97",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "head_auto_apply_precision")
op.drop_column("ml_settings", "head_min_positives")
op.drop_index("ix_head_training_run_status", table_name="head_training_run")
op.drop_table("head_training_run")
op.drop_table("tag_head")
-70
View File
@@ -1,70 +0,0 @@
"""head_auto_apply_run + earned-auto-apply settings (#114)
A graduated head can apply its tag without a human, gated by a master switch +
a support floor. head_auto_apply_run tracks each sweep / dry-run preview.
Revision ID: 0059
Revises: 0058
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0059"
down_revision: Union[str, None] = "0058"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"head_auto_apply_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"dry_run", sa.Boolean(), nullable=False, server_default=sa.false()
),
sa.Column("params", JSONB(), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("n_applied", sa.Integer(), nullable=True),
sa.Column("report", JSONB(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index(
"ix_head_auto_apply_run_status", "head_auto_apply_run", ["status"],
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(), # opt-out: on by default (operator-asked)
),
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_min_positives", sa.Integer(), nullable=False,
server_default="30",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "head_auto_apply_min_positives")
op.drop_column("ml_settings", "head_auto_apply_enabled")
op.drop_index(
"ix_head_auto_apply_run_status", table_name="head_auto_apply_run"
)
op.drop_table("head_auto_apply_run")
-74
View File
@@ -1,74 +0,0 @@
"""head_metric + head_metrics_snapshot: auto-apply observability (#114)
Running misfire/under-fire counters per concept (captured at correction time,
since image_tag.source is lost on delete) + a daily per-concept time-series so
the operator can tune the precision target + support floor from real data.
Revision ID: 0060
Revises: 0059
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0060"
down_revision: Union[str, None] = "0059"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"head_metric",
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_table(
"head_metrics_snapshot",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"),
),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column(
"snapshot_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("n_auto_applied", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("ap", sa.Float(), nullable=True),
sa.Column("precision_cv", sa.Float(), nullable=True),
sa.Column("recall", sa.Float(), nullable=True),
sa.Column("n_pos", sa.Integer(), nullable=True),
)
op.create_index(
"ix_head_metrics_snapshot_tag_id", "head_metrics_snapshot", ["tag_id"],
)
op.create_index(
"ix_head_metrics_snapshot_snapshot_at", "head_metrics_snapshot",
["snapshot_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_head_metrics_snapshot_snapshot_at", table_name="head_metrics_snapshot"
)
op.drop_index(
"ix_head_metrics_snapshot_tag_id", table_name="head_metrics_snapshot"
)
op.drop_table("head_metrics_snapshot")
op.drop_table("head_metric")
-59
View File
@@ -1,59 +0,0 @@
"""image_region: detected/proposed regions + their crop embeddings (#114)
Storage backbone of the crop pipeline. A region = normalized bbox + the crop's
embedding (CCIP for face/figure → character id; SigLIP for concept regions →
head bag-of-embeddings). Also serves as grounded-tag bbox provenance.
Revision ID: 0061
Revises: 0060
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
revision: str = "0061"
down_revision: Union[str, None] = "0060"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_CCIP_DIM = 768
_SIGLIP_DIM = 1152
def upgrade() -> None:
op.create_table(
"image_region",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
),
sa.Column("kind", sa.String(length=16), nullable=False),
# Video/animated: source frame timestamp (seconds); NULL for stills.
sa.Column("frame_time", sa.Float(), nullable=True),
sa.Column("rx", sa.Float(), nullable=False),
sa.Column("ry", sa.Float(), nullable=False),
sa.Column("rw", sa.Float(), nullable=False),
sa.Column("rh", sa.Float(), nullable=False),
sa.Column("score", sa.Float(), nullable=True),
sa.Column("detector_version", sa.String(length=64), nullable=True),
sa.Column("crop_version", sa.String(length=64), nullable=True),
sa.Column("embedding_version", sa.String(length=128), nullable=True),
sa.Column("ccip_embedding", Vector(_CCIP_DIM), nullable=True),
sa.Column("siglip_embedding", Vector(_SIGLIP_DIM), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_index(
"ix_image_region_image_record_id", "image_region", ["image_record_id"],
)
def downgrade() -> None:
op.drop_index("ix_image_region_image_record_id", table_name="image_region")
op.drop_table("image_region")
-55
View File
@@ -1,55 +0,0 @@
"""gpu_job: the HTTP-leased GPU work queue for the desktop agent (#114)
The agent stays HTTP-only — the server enqueues per-(image, task) jobs here and
the agent leases/submits over the web API; Redis/Postgres stay private.
Revision ID: 0062
Revises: 0061
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0062"
down_revision: Union[str, None] = "0061"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"gpu_job",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
),
sa.Column("task", sa.String(length=32), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("lease_token", sa.String(length=64), nullable=True),
sa.Column("leased_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_index("ix_gpu_job_image_record_id", "gpu_job", ["image_record_id"])
op.create_index("ix_gpu_job_status", "gpu_job", ["status"])
def downgrade() -> None:
op.drop_index("ix_gpu_job_status", table_name="gpu_job")
op.drop_index("ix_gpu_job_image_record_id", table_name="gpu_job")
op.drop_table("gpu_job")
@@ -1,33 +0,0 @@
"""ml_settings.ccip_match_threshold — tunable CCIP character-match cut (#114)
The v1 matcher used a flat 0.75 cosine; live data showed that over-fires (a
high-reference character matched a scatter of images). 0.85 keeps the confident
single-character matches and drops the noise. Tunable from the GPU agent card.
Revision ID: 0063
Revises: 0062
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0063"
down_revision: Union[str, None] = "0062"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"ccip_match_threshold", sa.Float(), nullable=False,
server_default="0.85",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "ccip_match_threshold")
-42
View File
@@ -1,42 +0,0 @@
"""ml_settings: CCIP auto-apply switch + threshold (#114)
Confident CCIP character matches auto-tag (source='ccip_auto') on a daily sweep,
so identity tags keep flowing without pressing a button. ON by default (opt-out,
like head auto-apply); the high threshold (0.92, above the 0.85 suggest cut) +
single-character references keep it safe, and every auto-tag is reversible.
Revision ID: 0064
Revises: 0063
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0064"
down_revision: Union[str, None] = "0063"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"ccip_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(),
),
)
op.add_column(
"ml_settings",
sa.Column(
"ccip_auto_apply_threshold", sa.Float(), nullable=False,
server_default="0.92",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "ccip_auto_apply_threshold")
op.drop_column("ml_settings", "ccip_auto_apply_enabled")
-8
View File
@@ -20,14 +20,11 @@ def all_blueprints() -> list[Blueprint]:
from .artist import artist_bp from .artist import artist_bp
from .artists import artists_bp from .artists import artists_bp
from .attachments import attachments_bp from .attachments import attachments_bp
from .ccip import ccip_bp
from .cleanup import cleanup_bp from .cleanup import cleanup_bp
from .credentials import credentials_bp from .credentials import credentials_bp
from .downloads import downloads_bp from .downloads import downloads_bp
from .extension import extension_bp from .extension import extension_bp
from .gallery import gallery_bp from .gallery import gallery_bp
from .gpu import gpu_bp
from .heads import heads_bp
from .import_admin import import_admin_bp from .import_admin import import_admin_bp
from .ml_admin import ml_admin_bp from .ml_admin import ml_admin_bp
from .platforms import platforms_bp from .platforms import platforms_bp
@@ -39,7 +36,6 @@ def all_blueprints() -> list[Blueprint]:
from .suggestions import suggestions_bp from .suggestions import suggestions_bp
from .system_activity import system_activity_bp from .system_activity import system_activity_bp
from .system_backup import system_backup_bp from .system_backup import system_backup_bp
from .tag_eval import tag_eval_bp
from .tags import tags_bp from .tags import tags_bp
from .thumbnails import thumbnails_bp from .thumbnails import thumbnails_bp
return [ return [
@@ -60,10 +56,6 @@ def all_blueprints() -> list[Blueprint]:
suggestions_bp, suggestions_bp,
allowlist_bp, allowlist_bp,
aliases_bp, aliases_bp,
tag_eval_bp,
heads_bp,
gpu_bp,
ccip_bp,
ml_admin_bp, ml_admin_bp,
thumbnails_bp, thumbnails_bp,
sources_bp, sources_bp,
+49 -59
View File
@@ -39,31 +39,6 @@ def _bulk_image_confirm_token(image_ids: list[int]) -> str:
return digest[:8] return digest[:8]
async def _run_dry_run_op(service_fn, **service_kwargs):
"""Shared body for the Tier-A dry-run/apply endpoints: read the `dry_run`
flag, run the cleanup_service predicate under `run_sync`, and return its
result dict. The SAME `service_fn` drives both preview and apply (the flag
just toggles), so a handler physically can't let its preview diverge from
its delete (rule 93). Default False preserves the existing contract — the UI
always passes `dry_run` explicitly (true to preview, false to apply). Extra
service kwargs (e.g. `source_id`) pass straight through."""
body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: service_fn(sync_sess, dry_run=dry_run, **service_kwargs)
)
return jsonify(result)
def _queued(async_result):
"""Standard 202 for an operator-triggered maintenance task: hand the UI the
Celery task id so it can tail /maintenance/task-result (or the activity
dashboard) for the summary. (trigger_vacuum stays bespoke — the UI doesn't
poll it, so it returns no task id.)"""
return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"]) @admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"])
async def artist_cascade_delete(slug: str): async def artist_cascade_delete(slug: str):
body = await request.get_json(silent=True) or {} body = await request.get_json(silent=True) or {}
@@ -171,30 +146,6 @@ async def tag_merge(dest_id: int):
if not isinstance(source_id, int) or source_id == dest_id: if not isinstance(source_id, int) or source_id == dest_id:
return _bad("invalid_source_id", detail="source_id must be int and differ from dest") return _bad("invalid_source_id", detail="source_id must be int and differ from dest")
# dry_run: non-mutating preview (counts + sample) so the operator can
# confirm the target before the irreversible merge (#8, rule 93 parity).
if body.get("dry_run"):
async with get_session() as session:
try:
p = await TagService(session).merge_preview(
source_id=source_id, target_id=dest_id,
)
except TagValidationError as exc:
return _bad("tag_not_found", status=404, detail=str(exc))
return jsonify({
"preview": {
"source_id": p.source_id, "source_name": p.source_name,
"target_id": p.target_id, "target_name": p.target_name,
"compatible": p.compatible,
"images_moving": p.images_moving,
"images_already_on_target": p.images_already_on_target,
"source_total": p.source_total,
"series_pages": p.series_pages,
"will_alias": p.will_alias,
"sample_thumbnails": p.sample_thumbnails,
},
})
async with get_session() as session: async with get_session() as session:
try: try:
result = await TagService(session).merge( result = await TagService(session).merge(
@@ -242,7 +193,16 @@ async def tags_prune_unused():
re-call with dry_run=false.""" re-call with dry_run=false."""
from ..services.cleanup_service import prune_unused_tags from ..services.cleanup_service import prune_unused_tags
return await _run_dry_run_op(prune_unused_tags) body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: prune_unused_tags(
sync_sess, dry_run=dry_run,
)
)
return jsonify(result)
@admin_bp.route("/posts/prune-bare", methods=["POST"]) @admin_bp.route("/posts/prune-bare", methods=["POST"])
@@ -254,7 +214,16 @@ async def posts_prune_bare():
prune itself, so the preview can't diverge from the delete.""" prune itself, so the preview can't diverge from the delete."""
from ..services.cleanup_service import prune_bare_posts from ..services.cleanup_service import prune_bare_posts
return await _run_dry_run_op(prune_bare_posts) body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: prune_bare_posts(
sync_sess, dry_run=dry_run,
)
)
return jsonify(result)
@admin_bp.route("/posts/reconcile-duplicates", methods=["POST"]) @admin_bp.route("/posts/reconcile-duplicates", methods=["POST"])
@@ -268,13 +237,20 @@ async def posts_reconcile_duplicates():
from ..services.cleanup_service import reconcile_duplicate_posts from ..services.cleanup_service import reconcile_duplicate_posts
body = await request.get_json(silent=True) or {} body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
raw_source = body.get("source_id") raw_source = body.get("source_id")
try: try:
source_id = int(raw_source) if raw_source is not None else None source_id = int(raw_source) if raw_source is not None else None
except (TypeError, ValueError): except (TypeError, ValueError):
return _bad("invalid_source_id", detail="source_id must be an integer") return _bad("invalid_source_id", detail="source_id must be an integer")
return await _run_dry_run_op(reconcile_duplicate_posts, source_id=source_id) async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: reconcile_duplicate_posts(
sync_sess, source_id=source_id, dry_run=dry_run,
)
)
return jsonify(result)
@admin_bp.route("/tags/purge-legacy", methods=["POST"]) @admin_bp.route("/tags/purge-legacy", methods=["POST"])
@@ -287,7 +263,14 @@ async def tags_purge_legacy():
operator confirms with dry_run=false.""" operator confirms with dry_run=false."""
from ..services.cleanup_service import purge_legacy_tags from ..services.cleanup_service import purge_legacy_tags
return await _run_dry_run_op(purge_legacy_tags) body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
)
return jsonify(result)
@admin_bp.route("/tags/reset-content", methods=["POST"]) @admin_bp.route("/tags/reset-content", methods=["POST"])
@@ -301,7 +284,14 @@ async def tags_reset_content():
Irreversible except via DB backup restore.""" Irreversible except via DB backup restore."""
from ..services.cleanup_service import reset_content_tagging from ..services.cleanup_service import reset_content_tagging
return await _run_dry_run_op(reset_content_tagging) body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", False))
async with get_session() as session:
result = await session.run_sync(
lambda sync_sess: reset_content_tagging(sync_sess, dry_run=dry_run)
)
return jsonify(result)
@admin_bp.route("/tags/normalize", methods=["POST"]) @admin_bp.route("/tags/normalize", methods=["POST"])
@@ -327,7 +317,7 @@ async def tags_normalize():
from ..tasks.admin import normalize_tags_task from ..tasks.admin import normalize_tags_task
async_result = normalize_tags_task.delay() async_result = normalize_tags_task.delay()
return _queued(async_result) return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/db-stats", methods=["GET"]) @admin_bp.route("/maintenance/db-stats", methods=["GET"])
@@ -384,7 +374,7 @@ async def trigger_reextract_archives():
from ..tasks.admin import reextract_archive_attachments_task from ..tasks.admin import reextract_archive_attachments_task
async_result = reextract_archive_attachments_task.delay() async_result = reextract_archive_attachments_task.delay()
return _queued(async_result) return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/prune-missing-files", methods=["POST"]) @admin_bp.route("/maintenance/prune-missing-files", methods=["POST"])
@@ -397,7 +387,7 @@ async def trigger_prune_missing_files():
from ..tasks.admin import prune_missing_file_records_task from ..tasks.admin import prune_missing_file_records_task
async_result = prune_missing_file_records_task.delay() async_result = prune_missing_file_records_task.delay()
return _queued(async_result) return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"]) @admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
@@ -413,7 +403,7 @@ async def trigger_dedup_videos():
body = await request.get_json(silent=True) or {} body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
async_result = dedup_videos_task.delay(dry_run=dry_run) async_result = dedup_videos_task.delay(dry_run=dry_run)
return _queued(async_result) return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/purge-gated-previews", methods=["POST"]) @admin_bp.route("/maintenance/purge-gated-previews", methods=["POST"])
@@ -429,7 +419,7 @@ async def trigger_purge_gated_previews():
body = await request.get_json(silent=True) or {} body = await request.get_json(silent=True) or {}
dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview dry_run = bool(body.get("dry_run", True)) # default to the SAFE preview
async_result = purge_gated_previews_task.delay(dry_run=dry_run) async_result = purge_gated_previews_task.delay(dry_run=dry_run)
return _queued(async_result) return jsonify({"task_id": async_result.id, "status": "queued"}), 202
@admin_bp.route("/maintenance/task-result/<task_id>", methods=["GET"]) @admin_bp.route("/maintenance/task-result/<task_id>", methods=["GET"])
-25
View File
@@ -20,37 +20,12 @@ async def list_allowlist():
"tag_name": r.tag_name, "tag_name": r.tag_name,
"tag_kind": r.tag_kind, "tag_kind": r.tag_kind,
"min_confidence": r.min_confidence, "min_confidence": r.min_confidence,
"applied_count": r.applied_count,
"coverage_count": r.coverage_count,
} }
for r in rows for r in rows
] ]
) )
@allowlist_bp.route("/tags/<int:tag_id>/allowlist/coverage", methods=["GET"])
async def coverage(tag_id: int):
"""Live "at threshold T, a sweep would cover ~N images" projection for the
allowlist tuning dashboard. Defaults to the tag's stored threshold."""
raw = request.args.get("threshold")
async with get_session() as session:
svc = AllowlistService(session)
if raw is not None:
try:
threshold = float(raw)
except ValueError:
return jsonify({"error": "threshold must be a float"}), 400
if not (0 < threshold <= 1):
return jsonify({"error": "threshold must be in (0, 1]"}), 400
else:
row = await session.get(TagAllowlist, tag_id)
if row is None:
return jsonify({"error": "not on allowlist"}), 404
threshold = row.min_confidence
count = await svc.coverage(tag_id, threshold)
return jsonify({"count": count, "threshold": threshold})
@allowlist_bp.route("/tags/<int:tag_id>/allowlist", methods=["GET"]) @allowlist_bp.route("/tags/<int:tag_id>/allowlist", methods=["GET"])
async def get_one(tag_id: int): async def get_one(tag_id: int):
async with get_session() as session: async with get_session() as session:
-124
View File
@@ -1,124 +0,0 @@
"""CCIP / region observability API (#114) — read-only, analysis-shaped.
So the work can be checked through an API as the agent fills in vectors: overall
coverage (regions by kind, how many images have figure CCIP vectors, which
characters have enough reference examples to match on) + a per-image drill-down
(its regions + the CCIP character matches it would get). Mirrors the heads
metrics endpoint; no GPU, just reads what's stored.
"""
from quart import Blueprint, jsonify
from sqlalchemy import distinct, func, select
from ..extensions import get_session
from ..models import ImageRegion, Tag, TagKind
from ..models.tag import image_tag
from ..services.ml.ccip import match_image
ccip_bp = Blueprint("ccip", __name__, url_prefix="/api/ccip")
_FIGURE_KINDS = ("face", "figure")
@ccip_bp.route("/overview", methods=["GET"])
async def overview():
async with get_session() as session:
by_kind = dict(
(
await session.execute(
select(ImageRegion.kind, func.count()).group_by(ImageRegion.kind)
)
).all()
)
images_with_figure_ccip = (
await session.execute(
select(func.count(distinct(ImageRegion.image_record_id)))
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
.where(ImageRegion.ccip_embedding.is_not(None))
)
).scalar_one()
# Concept-crop (SigLIP bag) coverage — how far the back-catalogue embed
# has progressed, so the max-over-bag scorer's reach is checkable.
images_with_concept_siglip = (
await session.execute(
select(func.count(distinct(ImageRegion.image_record_id)))
.where(ImageRegion.kind == "concept")
.where(ImageRegion.siglip_embedding.is_not(None))
)
).scalar_one()
# Per-character reference counts (no vectors loaded) — which characters
# have enough examples to match on.
ref_rows = (
await session.execute(
select(image_tag.c.tag_id, Tag.name, func.count())
.select_from(ImageRegion)
.join(
image_tag,
image_tag.c.image_record_id == ImageRegion.image_record_id,
)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
.where(ImageRegion.ccip_embedding.is_not(None))
.group_by(image_tag.c.tag_id, Tag.name)
.order_by(func.count().desc())
)
).all()
versions = [
v for (v,) in (
await session.execute(
select(distinct(ImageRegion.embedding_version))
)
).all() if v
]
auto_applied = (
await session.execute(
select(func.count()).select_from(image_tag).where(
image_tag.c.source == "ccip_auto"
)
)
).scalar_one()
return jsonify({
"regions_by_kind": by_kind,
"images_with_figure_ccip": images_with_figure_ccip,
"images_with_concept_siglip": images_with_concept_siglip,
"characters_with_references": len(ref_rows),
"character_references": [
{"tag_id": t, "name": n, "n_refs": c} for (t, n, c) in ref_rows
],
"embedding_versions": versions,
"auto_applied": auto_applied,
})
@ccip_bp.route("/images/<int:image_id>", methods=["GET"])
async def image_detail(image_id: int):
"""An image's stored regions + the CCIP character matches it would get —
for spot-checking the agent's output + the matcher."""
async with get_session() as session:
regions = (
await session.execute(
select(ImageRegion)
.where(ImageRegion.image_record_id == image_id)
.order_by(ImageRegion.id)
)
).scalars().all()
matches = await match_image(session, image_id)
return jsonify({
"image_id": image_id,
"regions": [
{
"id": r.id,
"kind": r.kind,
"bbox": [r.rx, r.ry, r.rw, r.rh],
"frame_time": r.frame_time,
"score": r.score,
"detector_version": r.detector_version,
"embedding_version": r.embedding_version,
"has_ccip": r.ccip_embedding is not None,
"has_siglip": r.siglip_embedding is not None,
}
for r in regions
],
"ccip_matches": matches,
})
+7 -23
View File
@@ -37,30 +37,16 @@ def _parse_filters():
"""Parse the composable gallery filters from query args, returning """Parse the composable gallery filters from query args, returning
``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates. ``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates.
The structured tag filter (#6) is AND-of-OR plus exclusions: `tag_id` accepts a single id or a comma-separated list (AND); `media` is
- `tag_id` accepts a single id or a comma-separated list — all ANDed image|video; `sort` is newest|oldest; `platform` selects one platform
(the include common case; back-compat). (or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are boolean
- `tag_or` is REPEATABLE; each instance is a comma-separated OR-group, and flags; `date_from`/`date_to` are inclusive calendar-day bounds (date_to is
the image must match at least one tag from EACH group (groups ANDed). widened by a day so the whole day is covered by the service's half-open
- `tag_not` is a comma-separated exclude list (image must carry none). `< date_to`)."""
`media` is image|video; `sort` is newest|oldest; `platform` selects one
platform (or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are
boolean flags; `date_from`/`date_to` are inclusive calendar-day bounds
(date_to is widened by a day so the whole day is covered by the service's
half-open `< date_to`)."""
tag_raw = request.args.get("tag_id") tag_raw = request.args.get("tag_id")
tag_ids = ( tag_ids = (
[int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None [int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None
) or None ) or None
tag_or_groups = [
grp for raw in request.args.getlist("tag_or")
if (grp := [int(x) for x in raw.split(",") if x.strip()])
] or None
not_raw = request.args.get("tag_not")
tag_exclude = (
[int(x) for x in not_raw.split(",") if x.strip()] if not_raw else None
) or None
post_id_raw = request.args.get("post_id") post_id_raw = request.args.get("post_id")
post_id = int(post_id_raw) if post_id_raw else None post_id = int(post_id_raw) if post_id_raw else None
artist_id_raw = request.args.get("artist_id") artist_id_raw = request.args.get("artist_id")
@@ -78,9 +64,7 @@ def _parse_filters():
date_to += timedelta(days=1) # inclusive of the date_to calendar day date_to += timedelta(days=1) # inclusive of the date_to calendar day
filters = { filters = {
"tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id, "tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id,
"media_type": media_type, "media_type": media_type, "platform": platform,
"tag_or_groups": tag_or_groups, "tag_exclude": tag_exclude,
"platform": platform,
"untagged": untagged, "no_artist": no_artist, "untagged": untagged, "no_artist": no_artist,
"date_from": date_from, "date_to": date_to, "date_from": date_from, "date_to": date_to,
} }
-220
View File
@@ -1,220 +0,0 @@
"""GPU-job API (#114): the HTTP surface the desktop agent pulls work from.
The agent stays HTTP-only — it leases jobs, fetches image pixels via the normal
FC image URLs, and submits embeddings/regions back, all over this API. Redis and
Postgres are never exposed. The agent endpoints are gated by a bearer token
(Authorization: Bearer <token>) stored in AppSetting; the admin endpoints
(token / backfill / status) ride the browser session like the rest of FC's
homelab admin.
"""
import secrets
from quart import Blueprint, jsonify, request
from sqlalchemy import func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..extensions import get_session
from ..models import AppSetting, GpuJob, ImageRecord, MLSettings
from ..services.gallery_service import image_url
from ..services.ml.embedder import MODEL_NAME as EMBED_MODEL_NAME
from ..services.ml.gpu_jobs import GpuJobService
from ..services.ml.regions import RegionService
gpu_bp = Blueprint("gpu", __name__, url_prefix="/api/gpu")
_TOKEN_KEY = "gpu_agent_token"
def _bearer() -> str | None:
h = request.headers.get("Authorization", "")
return h[7:].strip() if h.startswith("Bearer ") else None
async def _agent_authed(session) -> bool:
supplied = _bearer()
if not supplied:
return False
stored = (
await session.execute(
select(AppSetting.value).where(AppSetting.key == _TOKEN_KEY)
)
).scalar_one_or_none()
return stored is not None and secrets.compare_digest(supplied, stored)
# --- Admin (browser): token + backfill + status -------------------------
@gpu_bp.route("/token", methods=["GET"])
async def get_token():
async with get_session() as session:
tok = (
await session.execute(
select(AppSetting.value).where(AppSetting.key == _TOKEN_KEY)
)
).scalar_one_or_none()
return jsonify({"token": tok, "configured": tok is not None})
@gpu_bp.route("/token/rotate", methods=["POST"])
async def rotate_token():
token = secrets.token_urlsafe(32)
async with get_session() as session:
await session.execute(
pg_insert(AppSetting)
.values(key=_TOKEN_KEY, value=token)
.on_conflict_do_update(index_elements=["key"], set_={"value": token})
)
await session.commit()
return jsonify({"token": token})
@gpu_bp.route("/status", methods=["GET"])
async def status():
async with get_session() as session:
rows = (
await session.execute(
select(GpuJob.status, func.count()).group_by(GpuJob.status)
)
).all()
counts = dict(rows)
return jsonify({
"pending": counts.get("pending", 0),
"leased": counts.get("leased", 0),
"done": counts.get("done", 0),
"error": counts.get("error", 0),
})
@gpu_bp.route("/backfill", methods=["POST"])
async def backfill():
"""Enqueue a job for every image that doesn't already have one for `task`."""
body = await request.get_json(silent=True) or {}
task = str(body.get("task") or "ccip")
from ..tasks.ml import enqueue_gpu_backfill
r = enqueue_gpu_backfill.delay(task)
return jsonify({"celery_task_id": r.id, "task": task}), 202
# --- Agent (bearer token): lease / submit / heartbeat / fail ------------
@gpu_bp.route("/jobs/lease", methods=["POST"])
async def lease():
body = await request.get_json(silent=True) or {}
agent_id = str(body.get("agent_id") or "agent")
try:
batch = min(max(int(body.get("batch_size", 8)), 1), 64)
except (TypeError, ValueError):
batch = 8
async with get_session() as session:
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
ml = (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
# image rows for url/mime in one shot
ids = [j.image_record_id for j in jobs]
imgs = {
i.id: i for i in (
await session.execute(
select(ImageRecord).where(ImageRecord.id.in_(ids))
)
).scalars()
} if ids else {}
await session.commit()
out = []
for j in jobs:
img = imgs.get(j.image_record_id)
if img is None:
continue
out.append({
"job_id": j.id,
"image_id": j.image_record_id,
"task": j.task,
"mime": img.mime,
"image_url": image_url(img.path),
# For video/animated: the agent samples at this cadence.
"frame_interval_seconds": ml.video_frame_interval_seconds,
"max_frames": ml.video_max_frames,
# The embedding model the agent must use for concept crops, so
# its region vectors land in the SAME space the heads trained in.
# Server-announced → the agent stays model-agnostic; a swap is a
# server setting + a re-embed migration, never an agent change.
"embed_model_name": EMBED_MODEL_NAME,
"embed_version": ml.embedder_model_version,
})
return jsonify({"jobs": out})
@gpu_bp.route("/jobs/heartbeat", methods=["POST"])
async def heartbeat():
body = await request.get_json(silent=True) or {}
agent_id = str(body.get("agent_id") or "agent")
job_ids = [int(x) for x in (body.get("job_ids") or [])]
async with get_session() as session:
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
n = await GpuJobService(session).heartbeat(agent_id, job_ids)
await session.commit()
return jsonify({"extended": n})
@gpu_bp.route("/jobs/submit", methods=["POST"])
async def submit():
"""Store a job's regions + close it. regions: [{kind, bbox:[x,y,w,h],
frame_time?, score?, *_version?, ccip_embedding?, siglip_embedding?}].
replace_kinds defaults to the kinds present in the submitted regions."""
body = await request.get_json(silent=True) or {}
agent_id = str(body.get("agent_id") or "agent")
job_id = body.get("job_id")
regions = body.get("regions") or []
if job_id is None:
return jsonify({"error": "job_id required"}), 400
kinds = body.get("replace_kinds") or sorted({r["kind"] for r in regions})
async with get_session() as session:
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
job = await session.get(GpuJob, int(job_id))
if job is None or job.status != "leased" or job.lease_token != agent_id:
return jsonify({"error": "lease_invalid"}), 409
if kinds:
await RegionService(session).replace_regions(
job.image_record_id, kinds, regions
)
await GpuJobService(session).complete(agent_id, int(job_id))
await session.commit()
return jsonify({"ok": True, "stored": len(regions)})
@gpu_bp.route("/jobs/fail", methods=["POST"])
async def fail():
body = await request.get_json(silent=True) or {}
agent_id = str(body.get("agent_id") or "agent")
job_id = body.get("job_id")
if job_id is None:
return jsonify({"error": "job_id required"}), 400
async with get_session() as session:
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
ok = await GpuJobService(session).fail(
agent_id, int(job_id), str(body.get("error") or "")
)
await session.commit()
return jsonify({"ok": ok})
@gpu_bp.route("/jobs/release", methods=["POST"])
async def release():
"""Graceful stop: the agent hands its still-leased jobs back to pending so
they're picked up immediately instead of waiting out the lease."""
body = await request.get_json(silent=True) or {}
agent_id = str(body.get("agent_id") or "agent")
job_ids = [int(x) for x in (body.get("job_ids") or [])]
async with get_session() as session:
if not await _agent_authed(session):
return jsonify({"error": "unauthorized"}), 401
n = await GpuJobService(session).release(agent_id, job_ids)
await session.commit()
return jsonify({"released": n})
-285
View File
@@ -1,285 +0,0 @@
"""Heads API (#114): train + inspect the per-concept heads that power
suggestions (replacing Camie + centroid).
POST /api/heads/train — (re)train all eligible heads (one run at a time).
GET /api/heads — status: head count, last-trained, running run, the
per-concept head table (strength + auto-apply ready),
and recent training runs. The card rehydrates from
here so status survives navigation.
"""
from quart import Blueprint, jsonify, request
from sqlalchemy import desc, func, select
from ..extensions import get_session
from ..models import (
HeadAutoApplyRun,
HeadMetric,
HeadMetricsSnapshot,
HeadTrainingRun,
Tag,
TagHead,
)
from ..models.tag import image_tag
from ..services.ml.heads import (
HeadAutoApplyAlreadyRunning,
HeadAutoApplyDisabled,
HeadTrainingAlreadyRunning,
start_head_auto_apply_run,
start_head_training_run,
)
heads_bp = Blueprint("heads", __name__, url_prefix="/api/heads")
def _serialize_run(run: HeadTrainingRun) -> dict:
return {
"id": run.id,
"params": run.params,
"status": run.status,
"started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"n_trained": run.n_trained,
"n_skipped": run.n_skipped,
"error": run.error,
}
@heads_bp.route("/train", methods=["POST"])
async def train():
body = await request.get_json(silent=True) or {}
params = body.get("params") or body or {}
async with get_session() as session:
try:
run_id = await session.run_sync(
lambda s: start_head_training_run(s, params)
)
except HeadTrainingAlreadyRunning as running:
return jsonify({
"error": "training_already_running",
"running_id": int(running.args[0]),
}), 409
await session.commit()
return jsonify({"run_id": run_id, "status": "running"}), 202
@heads_bp.route("", methods=["GET"])
async def status():
async with get_session() as session:
count, last_trained = (
await session.execute(
select(func.count(), func.max(TagHead.trained_at))
)
).one()
graduated = (
await session.execute(
select(func.count()).where(
TagHead.auto_apply_threshold.is_not(None)
)
)
).scalar_one()
running = (
await session.execute(
select(HeadTrainingRun.id)
.where(HeadTrainingRun.status == "running")
.order_by(HeadTrainingRun.id.desc())
.limit(1)
)
).scalar_one_or_none()
runs = (
await session.execute(
select(HeadTrainingRun)
.order_by(HeadTrainingRun.id.desc())
.limit(10)
)
).scalars().all()
# The per-concept table: strongest first, capped for the admin card.
head_rows = (
await session.execute(
select(
TagHead.tag_id, Tag.name, Tag.kind,
TagHead.n_pos, TagHead.n_neg, TagHead.ap,
TagHead.precision_cv, TagHead.recall,
TagHead.auto_apply_threshold, TagHead.trained_at,
)
.join(Tag, Tag.id == TagHead.tag_id)
.order_by(desc(TagHead.ap))
.limit(500)
)
).all()
heads = [
{
"tag_id": r.tag_id,
"name": r.name,
"category": r.kind.value if hasattr(r.kind, "value") else str(r.kind),
"n_pos": r.n_pos,
"n_neg": r.n_neg,
"ap": r.ap,
"precision": r.precision_cv,
"recall": r.recall,
"auto_apply": r.auto_apply_threshold is not None,
"trained_at": r.trained_at.isoformat() if r.trained_at else None,
}
for r in head_rows
]
return jsonify({
"head_count": count,
"graduated_count": graduated,
"last_trained_at": last_trained.isoformat() if last_trained else None,
"running_id": running,
"runs": [_serialize_run(r) for r in runs],
"heads": heads,
})
def _serialize_apply_run(run: HeadAutoApplyRun) -> dict:
return {
"id": run.id,
"dry_run": run.dry_run,
"status": run.status,
"started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"n_applied": run.n_applied,
"report": run.report,
"error": run.error,
}
@heads_bp.route("/auto-apply", methods=["POST"])
async def auto_apply():
"""Trigger an earned-auto-apply sweep. {dry_run:true} previews (writes
nothing); a real sweep needs head_auto_apply_enabled on."""
body = await request.get_json(silent=True) or {}
params = {"dry_run": bool(body.get("dry_run", False))}
async with get_session() as session:
try:
run_id = await session.run_sync(
lambda s: start_head_auto_apply_run(s, params)
)
except HeadAutoApplyAlreadyRunning as running:
return jsonify({
"error": "auto_apply_already_running",
"running_id": int(running.args[0]),
}), 409
except HeadAutoApplyDisabled:
return jsonify({"error": "auto_apply_disabled"}), 400
await session.commit()
return jsonify({"run_id": run_id, "status": "running"}), 202
@heads_bp.route("/auto-apply", methods=["GET"])
async def auto_apply_status():
async with get_session() as session:
running = (
await session.execute(
select(HeadAutoApplyRun.id)
.where(HeadAutoApplyRun.status == "running")
.order_by(HeadAutoApplyRun.id.desc())
.limit(1)
)
).scalar_one_or_none()
runs = (
await session.execute(
select(HeadAutoApplyRun)
.order_by(HeadAutoApplyRun.id.desc())
.limit(10)
)
).scalars().all()
return jsonify({
"running_id": running,
"runs": [_serialize_apply_run(r) for r in runs],
})
@heads_bp.route("/metrics", methods=["GET"])
async def metrics():
"""Auto-apply observability: per-concept current counts (volume, misfires,
under-fires, realized misfire rate, head quality) + the daily time-series so
the operator can tune the precision target + support floor from real data."""
async with get_session() as session:
head_rows = (
await session.execute(
select(
TagHead.tag_id, Tag.name, TagHead.ap, TagHead.precision_cv,
TagHead.recall, TagHead.auto_apply_threshold, TagHead.n_pos,
).join(Tag, Tag.id == TagHead.tag_id)
)
).all()
heads = {r.tag_id: r for r in head_rows}
metric_rows = (
await session.execute(
select(
HeadMetric.tag_id, HeadMetric.n_misfires, HeadMetric.n_underfires
)
)
).all()
mets = {r.tag_id: r for r in metric_rows}
applied = dict(
(
await session.execute(
select(image_tag.c.tag_id, func.count())
.where(image_tag.c.source == "head_auto")
.group_by(image_tag.c.tag_id)
)
).all()
)
names = {r.tag_id: r.name for r in head_rows}
# Names for metric-only tags (head pruned but corrections recorded).
missing = [t for t in mets if t not in names]
if missing:
for tid, nm in (
await session.execute(
select(Tag.id, Tag.name).where(Tag.id.in_(missing))
)
).all():
names[tid] = nm
concepts = []
for tid in set(heads) | set(mets):
h = heads.get(tid)
m = mets.get(tid)
n_applied = applied.get(tid, 0)
n_mis = m.n_misfires if m else 0
denom = n_applied + n_mis
concepts.append({
"tag_id": tid,
"name": names.get(tid, str(tid)),
"n_auto_applied": n_applied,
"n_misfires": n_mis,
"n_underfires": m.n_underfires if m else 0,
# Of everything this head ever auto-applied, the fraction you
# removed — the misfire rate (null until something fired).
"misfire_rate": round(n_mis / denom, 4) if denom else None,
"ap": h.ap if h else None,
"precision_cv": h.precision_cv if h else None,
"recall": h.recall if h else None,
"auto_apply": bool(h and h.auto_apply_threshold is not None),
"n_pos": h.n_pos if h else None,
})
concepts.sort(key=lambda c: (c["n_misfires"], c["n_auto_applied"]), reverse=True)
snaps = (
await session.execute(
select(HeadMetricsSnapshot)
.order_by(HeadMetricsSnapshot.snapshot_at.desc())
.limit(1000)
)
).scalars().all()
return jsonify({
"concepts": concepts,
"snapshots": [
{
"tag_id": s.tag_id,
"name": s.name,
"snapshot_at": s.snapshot_at.isoformat() if s.snapshot_at else None,
"n_auto_applied": s.n_auto_applied,
"n_misfires": s.n_misfires,
"n_underfires": s.n_underfires,
"ap": s.ap,
"precision_cv": s.precision_cv,
"recall": s.recall,
"n_pos": s.n_pos,
}
for s in snaps
],
})
-25
View File
@@ -17,13 +17,6 @@ _EDITABLE = (
"video_frame_interval_seconds", "video_frame_interval_seconds",
"video_max_frames", "video_max_frames",
"video_min_tag_frames", "video_min_tag_frames",
"head_min_positives",
"head_auto_apply_precision",
"head_auto_apply_enabled",
"head_auto_apply_min_positives",
"ccip_match_threshold",
"ccip_auto_apply_enabled",
"ccip_auto_apply_threshold",
) )
@@ -47,13 +40,6 @@ async def get_settings():
"video_min_tag_frames": s.video_min_tag_frames, "video_min_tag_frames": s.video_min_tag_frames,
"tagger_model_version": s.tagger_model_version, "tagger_model_version": s.tagger_model_version,
"embedder_model_version": s.embedder_model_version, "embedder_model_version": s.embedder_model_version,
"head_min_positives": s.head_min_positives,
"head_auto_apply_precision": s.head_auto_apply_precision,
"head_auto_apply_enabled": s.head_auto_apply_enabled,
"head_auto_apply_min_positives": s.head_auto_apply_min_positives,
"ccip_match_threshold": s.ccip_match_threshold,
"ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
"ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
} }
) )
@@ -114,17 +100,6 @@ def _validate(p: dict) -> str | None:
return "video_min_tag_frames must be >= 1" return "video_min_tag_frames must be >= 1"
if p["video_min_tag_frames"] > p["video_max_frames"]: if p["video_min_tag_frames"] > p["video_max_frames"]:
return "video_min_tag_frames cannot exceed video_max_frames" return "video_min_tag_frames cannot exceed video_max_frames"
# Head training (#114).
if int(p["head_min_positives"]) < 1:
return "head_min_positives must be >= 1"
if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999):
return "head_auto_apply_precision must be between 0.5 and 0.999"
if int(p["head_auto_apply_min_positives"]) < 1:
return "head_auto_apply_min_positives must be >= 1"
if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999):
return "ccip_match_threshold must be between 0.5 and 0.999"
if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999):
return "ccip_auto_apply_threshold must be between 0.5 and 0.999"
return None return None
+6 -51
View File
@@ -3,31 +3,12 @@
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
from ..extensions import get_session from ..extensions import get_session
from ..models import Tag, TagAllowlist
from ..services.ml.allowlist import AllowlistService from ..services.ml.allowlist import AllowlistService
from ..services.ml.suggestions import SuggestionService from ..services.ml.suggestions import SuggestionService
suggestions_bp = Blueprint("suggestions", __name__, url_prefix="/api") suggestions_bp = Blueprint("suggestions", __name__, url_prefix="/api")
async def _accept_payload(session, svc, newly_added: bool, tag_id: int) -> dict:
"""Shape the accept/alias response. When accepting newly allowlists a tag,
include the coverage PROJECTION (at the tag's threshold) so the UI can show
a non-blocking "auto-applying to ~N images" toast — the actual apply runs
async via apply_allowlist_tags, so this is an estimate, not a post-hoc
count (#7)."""
payload = {"allowlisted": newly_added}
if newly_added:
tag = await session.get(Tag, tag_id)
row = await session.get(TagAllowlist, tag_id)
payload["tag_id"] = tag_id
payload["tag_name"] = tag.name if tag is not None else None
payload["projected_count"] = await svc.coverage(
tag_id, row.min_confidence if row is not None else 0.90,
)
return payload
@suggestions_bp.route("/images/<int:image_id>/suggestions", methods=["GET"]) @suggestions_bp.route("/images/<int:image_id>/suggestions", methods=["GET"])
async def get_suggestions(image_id: int): async def get_suggestions(image_id: int):
# ?min=<float> overrides the configured per-category thresholds so the typed # ?min=<float> overrides the configured per-category thresholds so the typed
@@ -61,10 +42,6 @@ async def get_suggestions(image_id: int):
# modal's "Treat as alias"/"Remove alias" affordances. # modal's "Treat as alias"/"Remove alias" affordances.
"raw_name": s.raw_name, "raw_name": s.raw_name,
"via_alias": s.via_alias, "via_alias": s.via_alias,
# operator dismissed this tag for this image — surfaced
# (not dropped) so the rail can show it rejected + offer
# one-click un-reject.
"rejected": s.rejected,
} }
for s in items for s in items
] ]
@@ -83,15 +60,13 @@ async def accept_suggestion(image_id: int):
return jsonify({"error": "tag_id required"}), 400 return jsonify({"error": "tag_id required"}), 400
tag_id = body["tag_id"] tag_id = body["tag_id"]
async with get_session() as session: async with get_session() as session:
svc = AllowlistService(session) newly_added = await AllowlistService(session).accept(image_id, tag_id)
newly_added = await svc.accept(image_id, tag_id)
payload = await _accept_payload(session, svc, newly_added, tag_id)
await session.commit() await session.commit()
if newly_added: if newly_added:
from ..tasks.ml import apply_allowlist_tags from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=tag_id) apply_allowlist_tags.delay(tag_id=tag_id)
return jsonify(payload) return "", 204
@suggestions_bp.route( @suggestions_bp.route(
@@ -102,24 +77,19 @@ async def alias_suggestion(image_id: int):
required = {"alias_string", "alias_category", "canonical_tag_id"} required = {"alias_string", "alias_category", "canonical_tag_id"}
if not body or not required.issubset(body): if not body or not required.issubset(body):
return jsonify({"error": f"required: {sorted(required)}"}), 400 return jsonify({"error": f"required: {sorted(required)}"}), 400
canonical_tag_id = body["canonical_tag_id"]
async with get_session() as session: async with get_session() as session:
svc = AllowlistService(session) newly_added = await AllowlistService(session).add_alias_and_accept(
newly_added = await svc.add_alias_and_accept(
image_id, image_id,
body["alias_string"], body["alias_string"],
body["alias_category"], body["alias_category"],
canonical_tag_id, body["canonical_tag_id"],
)
payload = await _accept_payload(
session, svc, newly_added, canonical_tag_id,
) )
await session.commit() await session.commit()
if newly_added: if newly_added:
from ..tasks.ml import apply_allowlist_tags from ..tasks.ml import apply_allowlist_tags
apply_allowlist_tags.delay(tag_id=canonical_tag_id) apply_allowlist_tags.delay(tag_id=body["canonical_tag_id"])
return jsonify(payload) return "", 204
@suggestions_bp.route( @suggestions_bp.route(
@@ -135,21 +105,6 @@ async def dismiss_suggestion(image_id: int):
return "", 204 return "", 204
@suggestions_bp.route(
"/images/<int:image_id>/suggestions/undismiss", methods=["POST"]
)
async def undismiss_suggestion(image_id: int):
"""Reverse a per-image dismissal (reject-recovery). Idempotent — undoing a
tag that isn't rejected is a no-op delete."""
body = await request.get_json()
if not body or "tag_id" not in body:
return jsonify({"error": "tag_id required"}), 400
async with get_session() as session:
await AllowlistService(session).undismiss(image_id, body["tag_id"])
await session.commit()
return "", 204
@suggestions_bp.route("/suggestions/bulk", methods=["POST"]) @suggestions_bp.route("/suggestions/bulk", methods=["POST"])
async def bulk_suggestions(): async def bulk_suggestions():
body = await request.get_json() body = await request.get_json()
-70
View File
@@ -1,70 +0,0 @@
"""Tag-eval API (#1130): trigger + revisit the head-vs-centroid eval.
The run + full report live in the tag_eval_run row, so the admin card rehydrates
from GET (history / detail) on mount — the report survives navigation rather than
living in transient frontend state.
"""
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..extensions import get_session
from ..models import TagEvalRun
from ..services.ml.tag_eval import EvalAlreadyRunning, start_tag_eval_run
tag_eval_bp = Blueprint("tag_eval", __name__, url_prefix="/api/tag-eval")
def _serialize(run: TagEvalRun, *, include_report: bool) -> dict:
out = {
"id": run.id,
"params": run.params,
"status": run.status,
"started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
"error": run.error,
}
if include_report:
out["report"] = run.report
return out
@tag_eval_bp.route("", methods=["POST"])
async def create():
body = await request.get_json(silent=True) or {}
params = body.get("params") or body or {}
async with get_session() as session:
try:
run_id = await session.run_sync(
lambda s: start_tag_eval_run(s, params)
)
except EvalAlreadyRunning as running:
return jsonify({
"error": "eval_already_running",
"running_id": int(running.args[0]),
}), 409
await session.commit()
return jsonify({"run_id": run_id, "status": "running"}), 202
@tag_eval_bp.route("", methods=["GET"])
async def history():
try:
limit = min(int(request.args.get("limit", "20")), 100)
except ValueError:
return jsonify({"error": "invalid_limit"}), 400
async with get_session() as session:
rows = (await session.execute(
select(TagEvalRun).order_by(TagEvalRun.id.desc()).limit(limit)
)).scalars().all()
# List is light — no full report (the detail endpoint carries it).
return jsonify({"runs": [_serialize(r, include_report=False) for r in rows]})
@tag_eval_bp.route("/<int:run_id>", methods=["GET"])
async def detail(run_id: int):
async with get_session() as session:
run = await session.get(TagEvalRun, run_id)
if run is None:
return jsonify({"error": "not_found"}), 404
return jsonify(_serialize(run, include_report=True))
+1 -16
View File
@@ -2,11 +2,10 @@
from quart import Blueprint, jsonify, request from quart import Blueprint, jsonify, request
from sqlalchemy import exists, select from sqlalchemy import exists, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
from ..extensions import get_session from ..extensions import get_session
from ..models import Tag, TagKind, TagPositiveConfirmation from ..models import Tag, TagKind
from ..models.tag_allowlist import TagAllowlist from ..models.tag_allowlist import TagAllowlist
from ..services.bulk_tag_service import BulkTagService from ..services.bulk_tag_service import BulkTagService
from ..services.ml.aliases import AliasService from ..services.ml.aliases import AliasService
@@ -184,20 +183,6 @@ async def remove_tag_from_image(image_id: int, tag_id: int):
return "", 204 return "", 204
@tags_bp.route("/images/<int:image_id>/tags/<int:tag_id>/confirm", methods=["POST"])
async def confirm_tag_on_image(image_id: int, tag_id: int):
"""Operator affirmed an applied tag is correct ("keep" on a doubted positive).
Idempotent; recorded so the eval's doubts list stops resurfacing it (#1130)."""
async with get_session() as session:
await session.execute(
pg_insert(TagPositiveConfirmation)
.values(image_record_id=image_id, tag_id=tag_id)
.on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
)
await session.commit()
return "", 204
@tags_bp.route("/tags/<int:tag_id>", methods=["GET"]) @tags_bp.route("/tags/<int:tag_id>", methods=["GET"])
async def get_tag(tag_id: int): async def get_tag(tag_id: int):
"""Resolve a single tag (used by the gallery to label its active """Resolve a single tag (used by the gallery to label its active
-68
View File
@@ -61,33 +61,7 @@ def make_celery() -> Celery:
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent. # Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
task_acks_late=True, task_acks_late=True,
worker_prefetch_multiplier=1, worker_prefetch_multiplier=1,
# Broker resilience (2026-06-24): a swarm overlay-network blip after a
# redeploy left Redis healthy but transiently unreachable, and a worker
# starting in that window crash-looped on the initial broker connect
# (kombu OperationalError) instead of waiting it out — needing a manual
# Redis reset to recover. Retry the broker FOREVER (None) on startup and
# at runtime so a transient outage self-heals when routing returns,
# rather than the worker exiting.
broker_connection_retry_on_startup=True, broker_connection_retry_on_startup=True,
broker_connection_retry=True,
broker_connection_max_retries=None,
# Redis-transport socket options (apply to the BROKER connection): a
# short connect timeout + TCP keepalive so a dead/blocked socket is
# noticed and retried, and a periodic health check that proactively
# reconnects a live worker through a network hiccup.
broker_transport_options={
"socket_connect_timeout": 5,
"socket_timeout": 30,
"socket_keepalive": True,
"retry_on_timeout": True,
"health_check_interval": 30,
},
# Same hardening for the Redis RESULT backend (separate connection pool).
redis_socket_connect_timeout=5,
redis_socket_timeout=30,
redis_socket_keepalive=True,
redis_retry_on_timeout=True,
redis_backend_health_check_interval=30,
beat_schedule={ beat_schedule={
"recover-interrupted-tasks": { "recover-interrupted-tasks": {
"task": "backend.app.tasks.maintenance.recover_interrupted_tasks", "task": "backend.app.tasks.maintenance.recover_interrupted_tasks",
@@ -109,36 +83,6 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.ml.apply_allowlist_tags", "task": "backend.app.tasks.ml.apply_allowlist_tags",
"schedule": 86400.0, "schedule": 86400.0,
}, },
"train-heads-nightly": {
"task": "backend.app.tasks.ml.scheduled_train_heads",
"schedule": 86400.0, # passive cadence; manual retrain stays available
},
"apply-head-tags-daily": {
"task": "backend.app.tasks.ml.scheduled_apply_head_tags",
"schedule": 86400.0, # no-op unless head_auto_apply_enabled
},
"recover-orphaned-gpu-jobs": {
"task": "backend.app.tasks.ml.recover_orphaned_gpu_jobs",
"schedule": 60.0, # quick pickup of work a dead agent orphaned
},
"enqueue-ccip-backfill-hourly": {
"task": "backend.app.tasks.ml.enqueue_gpu_backfill",
"schedule": 3600.0, # auto-feed new images (+ retry errored) so
"args": ("ccip",), # the queue keeps moving without the button
},
"enqueue-siglip-backfill-daily": {
"task": "backend.app.tasks.ml.enqueue_gpu_backfill",
"schedule": 86400.0, # drain the concept-crop back-catalogue +
"args": ("siglip",), # retry failed embeds, no button needed
},
"ccip-auto-apply-daily": {
"task": "backend.app.tasks.ml.scheduled_ccip_auto_apply",
"schedule": 86400.0, # no-op unless ccip_auto_apply_enabled
},
"snapshot-head-metrics-daily": {
"task": "backend.app.tasks.maintenance.snapshot_head_metrics",
"schedule": 86400.0,
},
"integrity-verify-weekly": { "integrity-verify-weekly": {
"task": "backend.app.tasks.maintenance.verify_integrity", "task": "backend.app.tasks.maintenance.verify_integrity",
"schedule": 604800.0, # weekly "schedule": 604800.0, # weekly
@@ -186,18 +130,6 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs", "task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
"schedule": 300.0, "schedule": 300.0,
}, },
"recover-stalled-tag-eval-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_tag_eval_runs",
"schedule": 300.0,
},
"recover-stalled-head-training-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_head_training_runs",
"schedule": 300.0,
},
"recover-stalled-head-auto-apply-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs",
"schedule": 300.0,
},
"recover-stalled-import-batches": { "recover-stalled-import-batches": {
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches", "task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
"schedule": 300.0, "schedule": 300.0,
-18
View File
@@ -8,15 +8,9 @@ from .base import Base
from .credential import Credential from .credential import Credential
from .download_event import DownloadEvent from .download_event import DownloadEvent
from .external_link import ExternalLink from .external_link import ExternalLink
from .gpu_job import GpuJob
from .head_auto_apply_run import HeadAutoApplyRun
from .head_metric import HeadMetric
from .head_metrics_snapshot import HeadMetricsSnapshot
from .head_training_run import HeadTrainingRun
from .image_prediction import ImagePrediction from .image_prediction import ImagePrediction
from .image_provenance import ImageProvenance from .image_provenance import ImageProvenance
from .image_record import ImageRecord from .image_record import ImageRecord
from .image_region import ImageRegion
from .import_batch import ImportBatch from .import_batch import ImportBatch
from .import_settings import ImportSettings from .import_settings import ImportSettings
from .import_task import ImportTask from .import_task import ImportTask
@@ -35,9 +29,6 @@ from .subscribestar_seen_media import SubscribeStarSeenMedia
from .tag import Tag, TagKind, image_tag from .tag import Tag, TagKind, image_tag
from .tag_alias import TagAlias from .tag_alias import TagAlias
from .tag_allowlist import TagAllowlist from .tag_allowlist import TagAllowlist
from .tag_eval_run import TagEvalRun
from .tag_head import TagHead
from .tag_positive_confirmation import TagPositiveConfirmation
from .tag_reference_embedding import TagReferenceEmbedding from .tag_reference_embedding import TagReferenceEmbedding
from .tag_suggestion_rejection import TagSuggestionRejection from .tag_suggestion_rejection import TagSuggestionRejection
from .task_run import TaskRun from .task_run import TaskRun
@@ -62,27 +53,18 @@ __all__ = [
"ImageRecord", "ImageRecord",
"ImagePrediction", "ImagePrediction",
"ImageProvenance", "ImageProvenance",
"ImageRegion",
"Tag", "Tag",
"TagKind", "TagKind",
"image_tag", "image_tag",
"DownloadEvent", "DownloadEvent",
"ExternalLink", "ExternalLink",
"GpuJob",
"ImportBatch", "ImportBatch",
"ImportTask", "ImportTask",
"ImportSettings", "ImportSettings",
"LibraryAuditRun", "LibraryAuditRun",
"MLSettings", "MLSettings",
"HeadAutoApplyRun",
"HeadMetric",
"HeadMetricsSnapshot",
"HeadTrainingRun",
"TagAlias", "TagAlias",
"TagAllowlist", "TagAllowlist",
"TagEvalRun",
"TagHead",
"TagPositiveConfirmation",
"TagReferenceEmbedding", "TagReferenceEmbedding",
"TagSuggestionRejection", "TagSuggestionRejection",
"TaskRun", "TaskRun",
-50
View File
@@ -1,50 +0,0 @@
"""GpuJob — a unit of GPU work the desktop agent pulls over HTTP (#114).
The durable work list that lets the agent stay HTTP-only: the server enqueues a
job per (image, task) — e.g. detect figures + CCIP-embed — and the agent LEASES a
batch, computes on its GPU, then SUBMITS results, all over the already-exposed web
API. Redis/Postgres stay private. A lease has an expiry; the lease query itself
re-claims expired leases (agent died / stopped mid-batch), so the queue is
self-healing without a separate sweep. One job is per ITEM; the agent fans a
VIDEO out into per-frame instances internally (see image_region.frame_time).
State: pending → leased → done | error (a failure under the attempt cap returns to
pending for another agent).
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, Text, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class GpuJob(Base):
__tablename__ = "gpu_job"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), index=True
)
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
task: Mapped[str] = mapped_column(String(32), nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True
)
# pending | leased | done | error
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
leased_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
lease_expires_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
-46
View File
@@ -1,46 +0,0 @@
"""HeadAutoApplyRun — persisted lifecycle of an earned-auto-apply sweep (#114).
A graduated head can apply its tag to images it scores above the head's
auto-apply threshold, without a human. This row tracks one such sweep (or a
dry-run PREVIEW of it) so the result survives navigation and the admin card can
show what fired / what would fire. Mirrors HeadTrainingRun. State machine:
running → ready / error. The `report` JSONB holds per-concept counts
(applied / projected / scanned).
"""
from datetime import datetime
from typing import Any
from sqlalchemy import Boolean, DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadAutoApplyRun(Base):
__tablename__ = "head_auto_apply_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
# dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing
# (preview/apply parity, rule 93).
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True
)
# running | ready | error
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
# Total tags applied across all heads this sweep (0 for a clean dry-run).
n_applied: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Per-concept breakdown: [{tag_id, name, applied, scanned, threshold}, ...].
report: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
-32
View File
@@ -1,32 +0,0 @@
"""HeadMetric — running correction counters per concept (#114 observability).
Earned auto-apply fires graduated heads; to TUNE it we need to know how often a
head's auto-applied tag was wrong (the operator removed it = a MISFIRE) and how
often the operator had to add a tag a head exists for by hand (an UNDER-FIRE,
the head missed it). image_tag.source is lost when a row is deleted, so these
are captured as durable cumulative counters at correction time — they survive
head retrain/prune (keyed by tag, not by the head row). The daily snapshot reads
them into the time-series.
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadMetric(Base):
__tablename__ = "head_metric"
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
)
# An auto-applied (source='head_auto') tag the operator later REMOVED.
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# A tag with a head that the operator added by HAND (the head missed it).
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
@@ -1,38 +0,0 @@
"""HeadMetricsSnapshot — a daily per-concept time-series point (#114).
The "amount of change over time" reporting the operator asked for: once a day,
record each concept's auto-applied VOLUME (current head_auto tags), cumulative
misfires/under-fires, and the head's measured quality. Plotting these rows over
time shows whether auto-apply is landing better/worse and whether tagging more is
sharpening a concept — the signal for tuning the precision target + support floor.
"""
from datetime import datetime
from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadMetricsSnapshot(Base):
__tablename__ = "head_metrics_snapshot"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), index=True
)
# Denormalized so a snapshot stays readable even if the tag is later renamed.
name: Mapped[str] = mapped_column(String(255), nullable=False)
snapshot_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
)
# Current count of source='head_auto' applications still standing.
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
# The head's measured quality at snapshot time (null if no head exists).
ap: Mapped[float | None] = mapped_column(Float, nullable=True)
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
recall: Mapped[float | None] = mapped_column(Float, nullable=True)
n_pos: Mapped[int | None] = mapped_column(Integer, nullable=True)
-44
View File
@@ -1,44 +0,0 @@
"""HeadTrainingRun — persisted lifecycle of a head-training batch (#114).
Mirrors TagEvalRun so the run SURVIVES navigation and the admin card can show
live + historical status instead of holding it in transient frontend state.
Training is idempotent (it upserts tag_head rows), so a SIGKILL'd run is harmless
— a maintenance recovery sweep flips a stalled `running` row to `error`, and the
next run re-trains. State machine: running → ready / error.
"""
from datetime import datetime
from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class HeadTrainingRun(Base):
__tablename__ = "head_training_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
# Training parameters: {min_positives, neg_ratio, precision_target, ...}.
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True
)
# running | ready | error
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
# How many concepts got a (re)trained head vs were skipped (too few labels).
n_trained: Mapped[int | None] = mapped_column(Integer, nullable=True)
n_skipped: Mapped[int | None] = mapped_column(Integer, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Last time the task made progress — the recovery sweep tells a live run from
# a SIGKILL'd one by this (mirrors TagEvalRun).
last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
-10
View File
@@ -41,16 +41,6 @@ class ImageProvenance(Base):
source_id: Mapped[int | None] = mapped_column( source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
) )
# The archive PostAttachment this image was extracted FROM, when it came
# out of a .zip/.rar rather than as a loose file (milestone #87). Lets the
# provenance UI show the exact archive a file lives inside instead of every
# attachment on the post. NULL for loose downloads and pre-backfill rows.
# SET NULL so deleting the archive attachment never destroys the (image,
# post) edge — it just forgets which archive it came from.
from_attachment_id: Mapped[int | None] = mapped_column(
ForeignKey("post_attachment.id", ondelete="SET NULL"),
nullable=True, index=True,
)
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True) captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
captured_at: Mapped[datetime] = mapped_column( captured_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
-62
View File
@@ -1,62 +0,0 @@
"""ImageRegion — a detected/proposed sub-region of an image + its crop embedding.
The storage backbone of the crop pipeline (#114). A region is a normalized bbox
plus the embedding of its crop:
- kind='face' / 'figure' → embedded by CCIP for cross-artist character identity.
- kind='concept' → embedded by SigLIP, a localized instance for a concept head's
bag-of-embeddings (a concept is "present if ANY instance matches").
One row carries the embedding appropriate to its kind (the other is null). The
bbox doubles as grounded-tag provenance (hover a tag → highlight its region; a
wrong box is a precise negative). The GPU agent writes these via the job API;
the few-shot character matcher + bag scorer read them — both server-side, no GPU.
"""
from datetime import datetime
from pgvector.sqlalchemy import Vector
from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
CCIP_DIM = 768 # deepghs/imgutils CCIP character embedding
SIGLIP_DIM = 1152 # matches image_record.siglip_embedding
class ImageRegion(Base):
__tablename__ = "image_region"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), index=True
)
# 'frame' (a whole video frame → SigLIP bag) | 'face' | 'figure' (→ CCIP
# character id) | 'concept' (→ SigLIP head bag).
kind: Mapped[str] = mapped_column(String(16), nullable=False)
# For video/animated media: the source frame's timestamp in SECONDS. NULL for
# static images. Lets a video be a BAG of per-frame instances (fixes the
# mean-embedding muddle) + grounds a tag to "appears at 0:42".
frame_time: Mapped[float | None] = mapped_column(Float, nullable=True)
# Normalized bbox in [0,1]: top-left (rx, ry) + size (rw, rh). Named rx/ry/…
# rather than x/y/by to dodge SQL keyword ambiguity ('by').
rx: Mapped[float] = mapped_column(Float, nullable=False)
ry: Mapped[float] = mapped_column(Float, nullable=False)
rw: Mapped[float] = mapped_column(Float, nullable=False)
rh: Mapped[float] = mapped_column(Float, nullable=False)
# Proposer/detector confidence (null for deterministic proposers).
score: Mapped[float | None] = mapped_column(Float, nullable=True)
# Version stamps so a re-detect / re-crop / re-embed can be gated (compute
# once; only redo when the producing model version changes).
detector_version: Mapped[str | None] = mapped_column(String(64), nullable=True)
crop_version: Mapped[str | None] = mapped_column(String(64), nullable=True)
embedding_version: Mapped[str | None] = mapped_column(String(128), nullable=True)
# Exactly one is set, per kind.
ccip_embedding: Mapped[list[float] | None] = mapped_column(
Vector(CCIP_DIM), nullable=True
)
siglip_embedding: Mapped[list[float] | None] = mapped_column(
Vector(SIGLIP_DIM), nullable=True
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+1 -47
View File
@@ -2,15 +2,7 @@
from datetime import datetime from datetime import datetime
from sqlalchemy import ( from sqlalchemy import CheckConstraint, DateTime, Float, Integer, String, func
Boolean,
CheckConstraint,
DateTime,
Float,
Integer,
String,
func,
)
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -63,44 +55,6 @@ class MLSettings(Base):
video_min_tag_frames: Mapped[int] = mapped_column( video_min_tag_frames: Mapped[int] = mapped_column(
Integer, nullable=False, default=3 Integer, nullable=False, default=3
) )
# Tagging-v2 head training (#114). The head is the suggestion source that
# LEARNS from the operator's tags (replacing Camie + centroid). A concept
# needs >= head_min_positives labelled images before a head is trained;
# head_auto_apply_precision is the precision bar a head must clear (at some
# operating point) to "graduate" into earned auto-apply. Operator-tunable.
head_min_positives: Mapped[int] = mapped_column(
Integer, nullable=False, default=8
)
head_auto_apply_precision: Mapped[float] = mapped_column(
Float, nullable=False, default=0.97
)
# Earned auto-apply (#114). A graduated head fires (tags images without a
# human) when this master switch is on AND the head has at least
# head_auto_apply_min_positives clean labels — so a precise-looking but
# under-supported low-N head can't spray tags across the library. ON by
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
# measured-precision gates keep it safe, and every auto-tag is reversible.
head_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True
)
head_auto_apply_min_positives: Mapped[int] = mapped_column(
Integer, nullable=False, default=30
)
# CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
# over-fired (high-reference characters matched a scatter of images); 0.85
# keeps the confident single-character matches. Tunable from the agent card.
ccip_match_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.85
)
# CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
# above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
# single-character references + the high bar keep it safe, every tag reversible.
ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True
)
ccip_auto_apply_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.92
)
tagger_model_version: Mapped[str] = mapped_column( tagger_model_version: Mapped[str] = mapped_column(
String(128), nullable=False, default="camie-tagger-v2" String(128), nullable=False, default="camie-tagger-v2"
) )
-45
View File
@@ -1,45 +0,0 @@
"""TagEvalRun — persisted lifecycle of a head-vs-centroid tagging eval (#1130).
Mirrors LibraryAuditRun so the result SURVIVES navigation: the run + its full
report live in this row, and the admin card rehydrates from it on mount instead
of holding the report in transient frontend state. State machine:
running → ready / error. The async ml-queue task writes `report` (JSONB) when
done; a maintenance recovery sweep flips a stalled `running` row to `error`.
"""
from datetime import datetime
from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class TagEvalRun(Base):
__tablename__ = "tag_eval_run"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
# The eval parameters: {concepts: [...], curve_points: [...], neg_ratio,
# cv_folds, ...} — echoed back so the report is self-describing.
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True,
)
# running | ready | error
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(),
)
finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
# The full result: per-concept metrics (head vs centroid), learning-curve
# points, and example image ids. Null until the task finishes.
report: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)
error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Last time the task made progress — the recovery sweep tells a live run
# from a SIGKILL'd one by this (mirrors LibraryAuditRun).
last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
-77
View File
@@ -1,77 +0,0 @@
"""TagHead — a small per-concept classifier trained on the operator's tags.
Milestone #114, tagging-v2: the production form of the head the eval (#1130)
proved. One row per concept (general or character) that has enough labelled
positives. The head is a logistic-regression boundary over the FROZEN SigLIP
embedding (L2-normalized), trained on the operator's positives + negatives
(rejections + sampled unlabeled). It REPLACES the Camie prediction + per-tag
centroid as the suggestion source — and unlike them it LEARNS: every accept /
reject re-trains it sharper.
Scoring (suggestion path, API worker, NO numpy): p = sigmoid(weights · x̂ + bias)
where x̂ is the L2-normalized image embedding. Surface as a suggestion when
p >= suggest_threshold; auto-apply only once auto_apply_threshold is set (the
head "graduated" — a precision-targeted operating point was achievable). The
thresholds come from CROSS-VALIDATED out-of-fold scores so they're honest, not
in-sample-optimistic; the deployable weights are fit on all data.
"""
from datetime import datetime
from typing import Any
from pgvector.sqlalchemy import Vector
from sqlalchemy import (
DateTime,
Float,
ForeignKey,
Integer,
String,
func,
)
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
# Matches image_record.siglip_embedding's dimensionality — the head operates in
# the same space. A model-version change re-embeds AND retrains (embedding_version
# guards staleness).
HEAD_DIM = 1152
class TagHead(Base):
__tablename__ = "tag_head"
# One head per concept tag; cascade so deleting a tag retires its head.
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
)
# The embedding the head was trained against (image_record's
# embedder_model_version). A mismatch with the current embedder means the
# head is stale and must be retrained, not scored.
embedding_version: Mapped[str] = mapped_column(String(128), nullable=False)
# Logistic-regression coefficients over the L2-normalized embedding, stored
# as a pgvector for compactness + a future in-DB dot-product path. NOT a
# similarity target, just a serialized weight vector.
weights: Mapped[list[float]] = mapped_column(Vector(HEAD_DIM), nullable=False)
bias: Mapped[float] = mapped_column(Float, nullable=False)
# Probability cutoff for SURFACING as a suggestion (F1-best on CV scores).
suggest_threshold: Mapped[float] = mapped_column(Float, nullable=False)
# Probability cutoff for EARNED auto-apply: the operating point that holds
# precision >= the configured target while maximizing recall. NULL = the head
# hasn't graduated (can't auto-apply without a human yet).
auto_apply_threshold: Mapped[float | None] = mapped_column(Float, nullable=True)
# Training-set sizes + cross-validated quality, surfaced in the admin card so
# the operator can see which concepts are strong / need more tags.
n_pos: Mapped[int] = mapped_column(Integer, nullable=False)
n_neg: Mapped[int] = mapped_column(Integer, nullable=False)
ap: Mapped[float] = mapped_column(Float, nullable=False)
# 'precision' is a SQL reserved word → store as precision_cv (the
# cross-validated precision at the suggest operating point).
precision_cv: Mapped[float] = mapped_column(Float, nullable=False)
recall: Mapped[float] = mapped_column(Float, nullable=False)
trained_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
# Extra detail (auto-apply operating point, F1, etc.) — non-load-bearing.
metrics: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)
@@ -1,28 +0,0 @@
"""TagPositiveConfirmation — operator affirmed an applied tag is correct.
The mirror of TagSuggestionRejection (#1130). When the operator "keeps" a
positive the head doubts (low-scoring), record it so the eval's doubts list
stops resurfacing the same confirmed-correct images every run. Does not change
training (it's already a positive) — purely a "I've reviewed this" marker.
"""
from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class TagPositiveConfirmation(Base):
__tablename__ = "tag_positive_confirmation"
image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
)
tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True
)
confirmed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
)
+5 -39
View File
@@ -18,7 +18,7 @@ from pathlib import Path
from typing import Any from typing import Any
from sqlalchemy import delete, func, or_, select, update from sqlalchemy import delete, func, or_, select, update
from sqlalchemy.orm import Session, aliased from sqlalchemy.orm import Session
from ..models import ( from ..models import (
Artist, Artist,
@@ -151,22 +151,11 @@ def project_bulk_image_delete(
def count_tag_associations(session: Session, *, tag_id: int) -> int: def count_tag_associations(session: Session, *, tag_id: int) -> int:
"""Images affected by deleting this tag — the Tier-B blast-radius prompt. """COUNT(*) FROM image_tag WHERE tag_id=?. For Tier-B prompt."""
Mirrors the gallery/directory membership predicate: images carrying the tag
DIRECTLY, plus — when it's a fandom — images carrying one of its characters
(member.fandom_id == tag_id). DISTINCT so each image counts once. Without
the character leg a fandom would report 0 here yet its delete still strips
the fandom off every character, badly understating the prompt."""
member = aliased(Tag)
return session.execute( return session.execute(
select(func.count(image_tag.c.image_record_id.distinct())).where( select(func.count())
or_( .select_from(image_tag)
image_tag.c.tag_id == tag_id, .where(image_tag.c.tag_id == tag_id)
image_tag.c.tag_id.in_(
select(member.id).where(member.fandom_id == tag_id)
),
)
)
).scalar_one() ).scalar_one()
@@ -584,29 +573,6 @@ def _repoint_post_links(session: Session, loser_id: int, keeper_id: int) -> None
dup_imgs = select(ImageProvenance.image_record_id).where( dup_imgs = select(ImageProvenance.image_record_id).where(
ImageProvenance.post_id == keeper_id ImageProvenance.post_id == keeper_id
) )
# Before dropping the colliding loser rows, carry their from_attachment_id
# (which archive the file came out of, milestone #87) onto the keeper's
# surviving row when the keeper didn't record one. For the gallery-dl→native
# case this very milestone targets, the keeper is the native stub (no
# archive) and the loser is the gallery-dl row that extracted the member, so
# a blind delete would silently lose the containing-archive linkage.
for img_id, att_id in session.execute(
select(ImageProvenance.image_record_id, ImageProvenance.from_attachment_id)
.where(
ImageProvenance.post_id == loser_id,
ImageProvenance.image_record_id.in_(dup_imgs),
ImageProvenance.from_attachment_id.is_not(None),
)
).all():
session.execute(
update(ImageProvenance)
.where(
ImageProvenance.post_id == keeper_id,
ImageProvenance.image_record_id == img_id,
ImageProvenance.from_attachment_id.is_(None),
)
.values(from_attachment_id=att_id)
)
session.execute( session.execute(
delete(ImageProvenance).where( delete(ImageProvenance).where(
ImageProvenance.post_id == loser_id, ImageProvenance.post_id == loser_id,
+18 -65
View File
@@ -25,13 +25,7 @@ from sqlalchemy.orm import aliased
from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
from ..models.tag import image_tag from ..models.tag import image_tag
from .pagination import decode_cursor, encode_cursor from .pagination import decode_cursor, encode_cursor
from .tag_query import ( from .tag_query import fandom_join_alias, serialize_tag, tag_columns
fandom_join_alias,
image_in_any_tag_scope,
image_in_tag_scope,
serialize_tag,
tag_columns,
)
# Reserved `platform` filter value selecting images with NO platformed # Reserved `platform` filter value selecting images with NO platformed
# provenance (filesystem imports). Returned by facets() as a null-valued # provenance (filesystem imports). Returned by facets() as a null-valued
@@ -142,15 +136,11 @@ def image_url(path: str) -> str:
return f"/images/{quote(rel, safe='/')}" return f"/images/{quote(rel, safe='/')}"
def _require_single_filter( def _require_single_filter(tag_ids, post_id, artist_id) -> None:
tag_ids, post_id, artist_id, tag_or_groups=None, tag_exclude=None,
) -> None:
"""post_id is the post-detail view — it can't combine with the """post_id is the post-detail view — it can't combine with the
composable filters. tag_ids / tag_or_groups / tag_exclude + artist_id composable filters. tag_ids + artist_id (+ media_type) compose freely
(+ media_type) compose freely (AND).""" (AND)."""
if post_id is not None and ( if post_id is not None and (tag_ids or artist_id is not None):
tag_ids or artist_id is not None or tag_or_groups or tag_exclude
):
raise ValueError( raise ValueError(
"post_id cannot be combined with tag or artist filters" "post_id cannot be combined with tag or artist filters"
) )
@@ -158,7 +148,6 @@ def _require_single_filter(
def _apply_scope( def _apply_scope(
stmt, *, tag_ids, post_id, artist_id, media_type, stmt, *, tag_ids, post_id, artist_id, media_type,
tag_or_groups=None, tag_exclude=None,
platform=None, untagged=False, no_artist=False, platform=None, untagged=False, no_artist=False,
date_from=None, date_to=None, date_from=None, date_to=None,
): ):
@@ -169,14 +158,7 @@ def _apply_scope(
be present on `stmt` (the artist/platform paths alias Post/Source inside be present on `stmt` (the artist/platform paths alias Post/Source inside
their own EXISTS). their own EXISTS).
Tag filtering is one structured model (#6): AND-of-OR plus exclusions. - tag_ids: image must carry ALL of them — one correlated EXISTS per tag.
- tag_ids: image must carry ALL of them — one correlated EXISTS per tag
(the AND-of-singletons "include" common case; light editor + back-compat).
- tag_or_groups: list of OR-groups; the image must carry AT LEAST ONE tag
from EACH group — one EXISTS(tag_id IN group) per group, AND'd across
groups. (advanced editor)
- tag_exclude: image must carry NONE of these — a single NOT EXISTS(tag_id
IN exclude). (light "exclude" chips + advanced NOT)
- post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded - post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded
by _require_single_filter). by _require_single_filter).
- media_type: 'image' | 'video' narrows by mime prefix. - media_type: 'image' | 'video' narrows by mime prefix.
@@ -186,17 +168,13 @@ def _apply_scope(
- no_artist: ImageRecord.artist_id IS NULL. - no_artist: ImageRecord.artist_id IS NULL.
- date_from / date_to: half-open [from, to) bounds on effective_date. - date_from / date_to: half-open [from, to) bounds on effective_date.
""" """
# Every tag clause goes through image_in_tag_scope/_any: a fandom tag also
# matches images carrying any of its characters (Tag.fandom_id). Include,
# OR-group, and exclude are all symmetric on that membership.
for tid in tag_ids or []: for tid in tag_ids or []:
stmt = stmt.where(image_in_tag_scope(tid)) stmt = stmt.where(
for group in tag_or_groups or []: exists().where(
if not group: image_tag.c.image_record_id == ImageRecord.id,
continue # an empty OR-group would match nothing; treat as absent image_tag.c.tag_id == tid,
stmt = stmt.where(image_in_any_tag_scope(group)) )
if tag_exclude: )
stmt = stmt.where(~image_in_any_tag_scope(tag_exclude))
prov = _provenance_clause(post_id, artist_id) prov = _provenance_clause(post_id, artist_id)
if prov is not None: if prov is not None:
stmt = stmt.where(prov) stmt = stmt.where(prov)
@@ -318,8 +296,6 @@ class GalleryService:
artist_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, media_type: str | None = None,
sort: str = "newest", sort: str = "newest",
tag_or_groups: list[list[int]] | None = None,
tag_exclude: list[int] | None = None,
platform: str | None = None, platform: str | None = None,
untagged: bool = False, untagged: bool = False,
no_artist: bool = False, no_artist: bool = False,
@@ -328,9 +304,7 @@ class GalleryService:
) -> GalleryPage: ) -> GalleryPage:
if limit < 1 or limit > 200: if limit < 1 or limit > 200:
raise ValueError("limit must be between 1 and 200") raise ValueError("limit must be between 1 and 200")
_require_single_filter( _require_single_filter(tag_ids, post_id, artist_id)
tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
)
eff = _effective_date_col() eff = _effective_date_col()
stmt = select(ImageRecord, Post.post_date, eff.label("eff")) stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
@@ -338,7 +312,6 @@ class GalleryService:
stmt = _apply_scope( stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id, stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type, artist_id=artist_id, media_type=media_type,
tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
platform=platform, untagged=untagged, no_artist=no_artist, platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to, date_from=date_from, date_to=date_to,
) )
@@ -386,8 +359,6 @@ class GalleryService:
post_id: int | None = None, post_id: int | None = None,
artist_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, media_type: str | None = None,
tag_or_groups: list[list[int]] | None = None,
tag_exclude: list[int] | None = None,
platform: str | None = None, platform: str | None = None,
untagged: bool = False, untagged: bool = False,
no_artist: bool = False, no_artist: bool = False,
@@ -401,13 +372,10 @@ class GalleryService:
year_col, month_col, func.count(ImageRecord.id).label("cnt") year_col, month_col, func.count(ImageRecord.id).label("cnt")
) )
stmt = _outer_join_primary_post(stmt) stmt = _outer_join_primary_post(stmt)
_require_single_filter( _require_single_filter(tag_ids, post_id, artist_id)
tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
)
stmt = _apply_scope( stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id, stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type, artist_id=artist_id, media_type=media_type,
tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
platform=platform, untagged=untagged, no_artist=no_artist, platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to, date_from=date_from, date_to=date_to,
) )
@@ -419,8 +387,6 @@ class GalleryService:
self, year: int, month: int, tag_ids: list[int] | None = None, self, year: int, month: int, tag_ids: list[int] | None = None,
post_id: int | None = None, artist_id: int | None = None, post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, sort: str = "newest", media_type: str | None = None, sort: str = "newest",
tag_or_groups: list[list[int]] | None = None,
tag_exclude: list[int] | None = None,
platform: str | None = None, untagged: bool = False, platform: str | None = None, untagged: bool = False,
no_artist: bool = False, date_from: datetime | None = None, no_artist: bool = False, date_from: datetime | None = None,
date_to: datetime | None = None, date_to: datetime | None = None,
@@ -437,13 +403,10 @@ class GalleryService:
extract("month", eff) == month, extract("month", eff) == month,
) )
stmt = _outer_join_primary_post(stmt) stmt = _outer_join_primary_post(stmt)
_require_single_filter( _require_single_filter(tag_ids, post_id, artist_id)
tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
)
stmt = _apply_scope( stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=post_id, stmt, tag_ids=tag_ids, post_id=post_id,
artist_id=artist_id, media_type=media_type, artist_id=artist_id, media_type=media_type,
tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
platform=platform, untagged=untagged, no_artist=no_artist, platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to, date_from=date_from, date_to=date_to,
) )
@@ -464,10 +427,7 @@ class GalleryService:
async def facets( async def facets(
self, *, tag_ids: list[int] | None = None, self, *, tag_ids: list[int] | None = None,
post_id: int | None = None, artist_id: int | None = None, post_id: int | None = None, artist_id: int | None = None,
media_type: str | None = None, media_type: str | None = None, platform: str | None = None,
tag_or_groups: list[list[int]] | None = None,
tag_exclude: list[int] | None = None,
platform: str | None = None,
untagged: bool = False, no_artist: bool = False, untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None, date_from: datetime | None = None, date_to: datetime | None = None,
) -> GalleryFacets: ) -> GalleryFacets:
@@ -477,13 +437,10 @@ class GalleryService:
No outer join is needed — every clause is a correlated EXISTS or a No outer join is needed — every clause is a correlated EXISTS or a
column predicate on ImageRecord. column predicate on ImageRecord.
""" """
_require_single_filter( _require_single_filter(tag_ids, post_id, artist_id)
tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
)
common = { common = {
"tag_ids": tag_ids, "post_id": post_id, "tag_ids": tag_ids, "post_id": post_id,
"artist_id": artist_id, "media_type": media_type, "artist_id": artist_id, "media_type": media_type,
"tag_or_groups": tag_or_groups, "tag_exclude": tag_exclude,
} }
# total — the full active filter (the headline result count). # total — the full active filter (the headline result count).
@@ -558,10 +515,7 @@ class GalleryService:
async def similar( async def similar(
self, image_id: int, limit: int = 100, *, self, image_id: int, limit: int = 100, *,
tag_ids: list[int] | None = None, artist_id: int | None = None, tag_ids: list[int] | None = None, artist_id: int | None = None,
media_type: str | None = None, media_type: str | None = None, platform: str | None = None,
tag_or_groups: list[list[int]] | None = None,
tag_exclude: list[int] | None = None,
platform: str | None = None,
untagged: bool = False, no_artist: bool = False, untagged: bool = False, no_artist: bool = False,
date_from: datetime | None = None, date_to: datetime | None = None, date_from: datetime | None = None, date_to: datetime | None = None,
) -> list[GalleryImage] | None: ) -> list[GalleryImage] | None:
@@ -593,7 +547,6 @@ class GalleryService:
stmt = _apply_scope( stmt = _apply_scope(
stmt, tag_ids=tag_ids, post_id=None, stmt, tag_ids=tag_ids, post_id=None,
artist_id=artist_id, media_type=media_type, artist_id=artist_id, media_type=media_type,
tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
platform=platform, untagged=untagged, no_artist=no_artist, platform=platform, untagged=untagged, no_artist=no_artist,
date_from=date_from, date_to=date_to, date_from=date_from, date_to=date_to,
) )
+1 -58
View File
@@ -17,7 +17,7 @@ from enum import StrEnum
from pathlib import Path from pathlib import Path
from PIL import Image from PIL import Image
from sqlalchemy import select, update from sqlalchemy import select
from sqlalchemy.exc import IntegrityError from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -506,12 +506,6 @@ class Importer:
artist_use = artist if artist is not None else self._resolve_artist(source) artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use) post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = [] member_ids: list[int] = []
# Every member image touched (new + superseded + deduped), so the
# from_attachment_id stamp below covers files that already existed in the
# library and were merely re-linked to this post — those matter most
# (the HR copy a bundle re-ships). Separate from member_ids, which is
# the NEWLY-imported subset feeding the ImportResult contract.
member_record_ids: set[int] = set()
# Per-outcome tally so the "no images" reason names the ACTUAL cause # Per-outcome tally so the "no images" reason names the ACTUAL cause
# (#718): nested-archive packs, all-deduped (benign), unsupported formats, # (#718): nested-archive packs, all-deduped (benign), unsupported formats,
# or failed/corrupt members — instead of one catch-all string. # or failed/corrupt members — instead of one catch-all string.
@@ -522,21 +516,11 @@ class Importer:
self._collect_archive_members( self._collect_archive_members(
source, attribution=source, source_row=source_row, source, attribution=source, source_row=source_row,
depth=0, member_ids=member_ids, counts=counts, depth=0, member_ids=member_ids, counts=counts,
member_record_ids=member_record_ids,
) )
# Preserve the archive itself (links to the same Post/Artist). # Preserve the archive itself (links to the same Post/Artist).
self._capture_attachment( self._capture_attachment(
source, post=post, artist=artist_use, resolved=True source, post=post, artist=artist_use, resolved=True
) )
# Stamp each member's provenance row for THIS post with the archive it
# came out of (milestone #87). Done as a post-pass rather than threaded
# through _import_media/_apply_sidecar so the many dedup/supersede
# branches stay untouched. NULL-only so a re-extract never re-stamps and
# the backfill (reextract task → this same path) is idempotent. Nested
# members link to this OUTER archive — the only one stored as a blob.
self._stamp_member_archive(
post.id if post is not None else None, source, member_record_ids,
)
if member_ids: if member_ids:
return ImportResult( return ImportResult(
status="imported", image_id=member_ids[0], status="imported", image_id=member_ids[0],
@@ -571,7 +555,6 @@ class Importer:
self, archive_path: Path, *, attribution: Path, self, archive_path: Path, *, attribution: Path,
source_row: Source | None, depth: int, source_row: Source | None, depth: int,
member_ids: list[int], counts: dict, member_ids: list[int], counts: dict,
member_record_ids: set[int],
) -> None: ) -> None:
"""Extract `archive_path` and import its image/video members, RECURSING """Extract `archive_path` and import its image/video members, RECURSING
into nested archives (#718). Members attribute to `attribution` — the into nested archives (#718). Members attribute to `attribution` — the
@@ -607,7 +590,6 @@ class Importer:
member_path, attribution=attribution, member_path, attribution=attribution,
source_row=source_row, depth=depth + 1, source_row=source_row, depth=depth + 1,
member_ids=member_ids, counts=counts, member_ids=member_ids, counts=counts,
member_record_ids=member_record_ids,
) )
continue continue
counts["media"] += 1 counts["media"] += 1
@@ -619,16 +601,10 @@ class Importer:
) )
if res.status in ("imported", "superseded") and res.image_id: if res.status in ("imported", "superseded") and res.image_id:
member_ids.append(res.image_id) member_ids.append(res.image_id)
member_record_ids.add(res.image_id)
elif res.status == "skipped" and res.skip_reason in ( elif res.status == "skipped" and res.skip_reason in (
SkipReason.duplicate_hash, SkipReason.duplicate_phash SkipReason.duplicate_hash, SkipReason.duplicate_phash
): ):
counts["deduped"] += 1 counts["deduped"] += 1
# A deduped member still links provenance to this post
# (enrich-on-duplicate); record it so its archive origin
# gets stamped too.
if res.image_id:
member_record_ids.add(res.image_id)
else: else:
counts["failed"] += 1 counts["failed"] += 1
except Exception as exc: # noqa: BLE001 — defensive per level; keep going except Exception as exc: # noqa: BLE001 — defensive per level; keep going
@@ -637,39 +613,6 @@ class Importer:
archive_path.name, depth, exc, archive_path.name, depth, exc,
) )
def _stamp_member_archive(
self, post_id: int | None, archive_source: Path, member_record_ids: set[int],
) -> None:
"""Record which archive each extracted member came from (milestone #87).
Resolves the archive's own PostAttachment (by post + sha — it was just
captured) and stamps from_attachment_id on every member's provenance row
FOR THIS POST. NULL-only, so re-extracting the same archive (the backfill
path) never overwrites and stays idempotent. No-op when the archive isn't
post-attached (filesystem import with no post) or yielded no members.
"""
if post_id is None or not member_record_ids:
return
sha = _sha256_of(archive_source)
att_id = self.session.execute(
select(PostAttachment.id).where(
PostAttachment.post_id == post_id,
PostAttachment.sha256 == sha,
)
).scalar_one_or_none()
if att_id is None:
return
self.session.execute(
update(ImageProvenance)
.where(
ImageProvenance.image_record_id.in_(member_record_ids),
ImageProvenance.post_id == post_id,
ImageProvenance.from_attachment_id.is_(None),
)
.values(from_attachment_id=att_id)
)
self.session.commit()
@staticmethod @staticmethod
def _video_aspect_matches(w, h, cw, ch) -> bool: def _video_aspect_matches(w, h, cw, ch) -> bool:
"""True when two (w,h) pairs share an aspect ratio within tolerance. """True when two (w,h) pairs share an aspect ratio within tolerance.
+5 -79
View File
@@ -5,18 +5,11 @@ image_tag AND to tag_allowlist; per-image removal/dismiss writes a rejection.
from collections.abc import Sequence from collections.abc import Sequence
from dataclasses import dataclass from dataclasses import dataclass
from sqlalchemy import and_, delete, distinct, func, or_, select from sqlalchemy import delete, select
from sqlalchemy.dialects.postgresql import insert from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ...models import ( from ...models import MLSettings, Tag, TagAllowlist, TagSuggestionRejection
ImagePrediction,
MLSettings,
Tag,
TagAlias,
TagAllowlist,
TagSuggestionRejection,
)
from ...models.tag import image_tag from ...models.tag import image_tag
from .aliases import AliasService from .aliases import AliasService
@@ -27,8 +20,6 @@ class AllowlistRow:
tag_name: str tag_name: str
tag_kind: str tag_kind: str
min_confidence: float min_confidence: float
applied_count: int # image_tag rows currently carrying this tag
coverage_count: int # images a sweep WOULD cover at min_confidence
class AllowlistService: class AllowlistService:
@@ -89,12 +80,6 @@ class AllowlistService:
) )
await self.session.execute(stmt) await self.session.execute(stmt)
async def undismiss(self, image_id: int, tag_id: int) -> None:
"""Undo a per-image dismissal — drop the TagSuggestionRejection so the
suggestion reverts to a live (un-rejected) state. Backs the rail's
one-click reject-recovery (operator-asked 2026-06-27)."""
await self._clear_rejection(image_id, tag_id)
async def reject_applied_tag(self, image_id: int, tag_id: int) -> None: async def reject_applied_tag(self, image_id: int, tag_id: int) -> None:
"""Operator removed an applied tag from an image. Remove the """Operator removed an applied tag from an image. Remove the
image_tag row AND record a rejection so the allowlist won't image_tag row AND record a rejection so the allowlist won't
@@ -131,44 +116,6 @@ class AllowlistService:
delete(TagAllowlist).where(TagAllowlist.tag_id == tag_id) delete(TagAllowlist).where(TagAllowlist.tag_id == tag_id)
) )
async def _coverage_match(self, tag: Tag):
"""The predicate over image_prediction rows that resolve to `tag`,
mirroring tasks.ml._confidence_for_tag's resolution: a prediction whose
raw_name equals the tag name (any category), OR an alias maps
(raw_name, category) -> this tag. Returns a SQLAlchemy boolean clause.
"""
alias_rows = (
await self.session.execute(
select(TagAlias.alias_string, TagAlias.alias_category).where(
TagAlias.canonical_tag_id == tag.id
)
)
).all()
name_clause = ImagePrediction.raw_name == tag.name
alias_clauses = [
and_(
ImagePrediction.raw_name == a,
ImagePrediction.category == c,
)
for a, c in alias_rows
]
return or_(name_clause, *alias_clauses) if alias_clauses else name_clause
async def coverage(self, tag_id: int, threshold: float) -> int:
"""How many distinct images a sweep WOULD cover for this tag at
`threshold`: images with a resolving prediction scoring >= threshold.
The gross candidate pool (NOT minus already-applied/rejected) — it's
the tuning signal for "lower the threshold and ~N more images qualify".
"""
tag = await self.session.get(Tag, tag_id)
if tag is None:
return 0
match = await self._coverage_match(tag)
stmt = select(
func.count(distinct(ImagePrediction.image_record_id))
).where(ImagePrediction.score >= threshold, match)
return (await self.session.execute(stmt)).scalar_one()
async def list_all(self) -> Sequence[AllowlistRow]: async def list_all(self) -> Sequence[AllowlistRow]:
stmt = ( stmt = (
select( select(
@@ -181,33 +128,12 @@ class AllowlistService:
.order_by(Tag.name.asc()) .order_by(Tag.name.asc())
) )
rows = (await self.session.execute(stmt)).all() rows = (await self.session.execute(stmt)).all()
tag_ids = [r[0] for r in rows] return [
# Applied counts in ONE grouped query (vs N per-row counts).
applied: dict[int, int] = {}
if tag_ids:
applied = dict(
(
await self.session.execute(
select(image_tag.c.tag_id, func.count())
.where(image_tag.c.tag_id.in_(tag_ids))
.group_by(image_tag.c.tag_id)
)
).all()
)
result = []
for r in rows:
# Coverage is per-tag (alias set differs); allowlist is small.
cov = await self.coverage(r[0], r[3])
result.append(
AllowlistRow( AllowlistRow(
tag_id=r[0], tag_id=r[0],
tag_name=r[1], tag_name=r[1],
tag_kind=r[2].value if hasattr(r[2], "value") else str(r[2]), tag_kind=r[2].value if hasattr(r[2], "value") else str(r[2]),
min_confidence=r[3], min_confidence=r[3],
applied_count=applied.get(r[0], 0),
coverage_count=cov,
) )
) for r in rows
return result ]
-180
View File
@@ -1,180 +0,0 @@
"""CCIP few-shot character matcher (#114) — server-side, numpy on stored vectors.
CCIP is a FROZEN identity embedding; we don't train it. Instead the operator's
tagged characters become reference prototypes: a character tag's references are
the CCIP vectors of figure/face regions on images carrying that tag. To suggest
characters for a new image, we compare its figure-region CCIP vectors to every
character's references (multi-prototype: best match over a character's examples)
and surface the ones that clear a similarity threshold. No GPU here — the agent
already produced the vectors; this is cosine matching on what's stored.
v1 uses cosine similarity on the raw CCIP vectors with a tunable threshold; the
exact CCIP difference metric/threshold gets validated against the model during
the hands-on eval. numpy is imported lazily (API worker has it via pgvector).
"""
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import ImageRegion, MLSettings, Tag, TagKind
from ...models.tag import image_tag
# Cosine-similarity floor to call a figure the same character. The live setting
# (ml_settings.ccip_match_threshold) drives it; this is only the fallback when no
# threshold is supplied AND no settings row exists.
DEFAULT_SIM_THRESHOLD = 0.85
_FIGURE_KINDS = ("face", "figure")
async def _settings_threshold(session: AsyncSession) -> float:
val = (
await session.execute(
select(MLSettings.ccip_match_threshold).where(MLSettings.id == 1)
)
).scalar_one_or_none()
return float(val) if val is not None else DEFAULT_SIM_THRESHOLD
def _l2norm(mat, np):
n = np.linalg.norm(mat, axis=1, keepdims=True)
n[n == 0] = 1.0
return mat / n
# Single-shot cache of the (expensive) reference load, keyed on a cheap
# signature that changes exactly when references could: a character tag added/
# removed (n_char_tags) or a figure embedded (max/ n of ccip regions). Shared by
# the live matcher (every modal open) and the auto-apply sweep.
_REF_CACHE: dict = {"sig": None, "refs": None}
def _single_character_images():
"""Subquery of image ids carrying EXACTLY ONE character tag. References come
only from these — on a multi-character image the tag is image-level, so every
figure would otherwise pollute each character's prototype set (a 2-character
image tagged 'Velma' would make Daphne's figure a Velma reference)."""
return (
select(image_tag.c.image_record_id)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.group_by(image_tag.c.image_record_id)
.having(func.count() == 1)
)
async def _ref_signature(session: AsyncSession) -> tuple:
n_tags = (
await session.execute(
select(func.count())
.select_from(image_tag)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
)
).scalar_one()
n_regs, max_id = (
await session.execute(
select(func.count(), func.max(ImageRegion.id)).where(
ImageRegion.kind.in_(_FIGURE_KINDS),
ImageRegion.ccip_embedding.is_not(None),
)
)
).one()
return (n_tags, n_regs, max_id)
async def character_references(session: AsyncSession) -> dict[int, list]:
"""Per character-tag CCIP reference vectors: figure/face-region CCIP
embeddings on UNAMBIGUOUS (single-character) images carrying that tag.
Multi-prototype — several vectors per character. Cached on a cheap signature."""
sig = await _ref_signature(session)
if _REF_CACHE["sig"] == sig and _REF_CACHE["refs"] is not None:
return _REF_CACHE["refs"]
rows = (
await session.execute(
select(image_tag.c.tag_id, ImageRegion.ccip_embedding)
.select_from(ImageRegion)
.join(
image_tag,
image_tag.c.image_record_id == ImageRegion.image_record_id,
)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.where(ImageRegion.kind.in_(_FIGURE_KINDS))
.where(ImageRegion.ccip_embedding.is_not(None))
.where(ImageRegion.image_record_id.in_(_single_character_images()))
)
).all()
refs: dict[int, list] = {}
for tag_id, vec in rows:
refs.setdefault(tag_id, []).append(vec)
_REF_CACHE.update(sig=sig, refs=refs)
return refs
async def _tag_names(session: AsyncSession, tag_ids: list[int]) -> dict[int, str]:
if not tag_ids:
return {}
return dict(
(
await session.execute(
select(Tag.id, Tag.name).where(Tag.id.in_(tag_ids))
)
).all()
)
async def match_image(
session: AsyncSession, image_id: int, threshold: float | None = None
) -> list[dict]:
"""Character suggestions for one image from its figure-region CCIP vectors:
[{tag_id, name, category:'character', score, source:'ccip'}], ranked.
Already-applied character tags are excluded. Empty if the image has no figure
CCIP vectors or no character references exist yet. threshold defaults to the
live ml_settings.ccip_match_threshold."""
import numpy as np
if threshold is None:
threshold = await _settings_threshold(session)
qvecs = (
await session.execute(
select(ImageRegion.ccip_embedding).where(
ImageRegion.image_record_id == image_id,
ImageRegion.kind.in_(_FIGURE_KINDS),
ImageRegion.ccip_embedding.is_not(None),
)
)
).scalars().all()
if not qvecs:
return []
refs = await character_references(session)
if not refs:
return []
applied = set(
(
await session.execute(
select(image_tag.c.tag_id).where(
image_tag.c.image_record_id == image_id
)
)
).scalars()
)
names = await _tag_names(session, [t for t in refs if t not in applied])
Q = _l2norm(np.vstack([np.asarray(v, dtype=np.float32) for v in qvecs]), np)
out = []
for tag_id, vecs in refs.items():
if tag_id in applied:
continue
R = _l2norm(np.vstack([np.asarray(v, dtype=np.float32) for v in vecs]), np)
best = float((Q @ R.T).max()) # best (query figure, reference) cosine
if best >= threshold:
out.append({
"tag_id": tag_id,
"name": names.get(tag_id, str(tag_id)),
"category": "character",
"score": round(best, 4),
"source": "ccip",
})
out.sort(key=lambda d: d["score"], reverse=True)
return out
-73
View File
@@ -1,73 +0,0 @@
"""Shared crop primitive for the region/crop pipeline (#114).
One model- and transport-agnostic function sits at the trunk of both crop jobs:
- CCIP characters: a face/figure detector proposes regions → crop → CCIP-embed.
- SigLIP concepts: head-guided / saliency proposes regions → crop → SigLIP-embed.
Only the PROPOSER (where to crop) and the EMBEDDER (what to run) differ; the crop
itself — including the lower-bound size floor below which a region is too small to
embed reliably — is identical, so it lives here and both jobs call it.
The actual detector + embedders run in the GPU agent; this is pure Pillow so it's
importable + testable anywhere (and the agent imports it for the crop step).
"""
from __future__ import annotations
from PIL import Image
# Size floor: a region must be at least this big on its SHORTER edge to be worth
# embedding — a smaller crop is a blurry upscale carrying little real signal, and
# unbounded tiny crops would explode the bag. Expressed as BOTH a fraction of the
# image's short side and an absolute pixel floor; the larger of the two wins.
MIN_CROP_FRACTION = 0.10
MIN_CROP_PX = 64
def _to_pixels(bbox: tuple[float, float, float, float], w: int, h: int):
"""Normalized (x, y, w, h) in [0,1] → pixel (x, y, w, h)."""
x, y, bw, bh = bbox
return x * w, y * h, bw * w, bh * h
def crop_region(
img: Image.Image,
bbox: tuple[float, float, float, float],
*,
pad: float = 0.0,
min_fraction: float = MIN_CROP_FRACTION,
min_px: int = MIN_CROP_PX,
out_size: int | None = None,
) -> Image.Image | None:
"""Crop a NORMALIZED bbox (x, y, w, h in [0,1]) from img.
- pad: grow the box by this fraction on each side (e.g. 0.15 = +15% context),
clamped to the image bounds.
- Returns None when the resulting region is below the size floor (too small to
embed reliably) — the caller skips embedding it.
- out_size: if given, resize the crop to out_size×out_size; otherwise return
the raw crop and let the embedder do its own preprocessing.
"""
iw, ih = img.size
px, py, pw, ph = _to_pixels(bbox, iw, ih)
if pad:
px -= pw * pad / 2.0
py -= ph * pad / 2.0
pw *= (1.0 + pad)
ph *= (1.0 + pad)
left = max(0, int(round(px)))
top = max(0, int(round(py)))
right = min(iw, int(round(px + pw)))
bottom = min(ih, int(round(py + ph)))
if right <= left or bottom <= top:
return None
floor = max(min_px, int(min_fraction * min(iw, ih)))
if min(right - left, bottom - top) < floor:
return None
crop = img.crop((left, top, right, bottom)).convert("RGB")
if out_size:
crop = crop.resize((out_size, out_size))
return crop
-177
View File
@@ -1,177 +0,0 @@
"""GPU-job queue engine (#114): enqueue / lease / heartbeat / complete / fail
/ release / recover_orphaned.
Backs the HTTP API the desktop agent pulls work from. The lease claims pending
OR expired-leased jobs with FOR UPDATE SKIP LOCKED, so concurrent agents/workers
never grab the same job. Orphan recovery is three-layered: a graceful agent stop
calls release() to hand its in-flight jobs back instantly; a hard crash is caught
by recover_orphaned() (a 60s beat sweep) which resets expired leases to pending;
and the lease itself reclaims expired leases as a final backstop. Result-writing
(regions) is done by the API handler via RegionService; complete() just closes.
"""
from datetime import UTC, datetime, timedelta
from sqlalchemy import and_, or_, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import GpuJob
# Lease window. Kept comfortably above any single job (a capped-frame video embed
# is tens of seconds) so a live, heartbeating worker is never falsely expired,
# but short enough that a hard crash recovers fast once the sweep fires.
DEFAULT_LEASE_TTL = 180 # seconds an agent holds a job before it can be re-leased
DEFAULT_BATCH = 8
MAX_ATTEMPTS = 3
class GpuJobService:
def __init__(self, session: AsyncSession):
self.session = session
async def enqueue(self, image_id: int, task: str) -> GpuJob | None:
"""Queue a (image, task) job. Idempotent: returns None if one is already
pending/leased for the same pair (no duplicate work)."""
dup = (
await self.session.execute(
select(GpuJob.id).where(
GpuJob.image_record_id == image_id,
GpuJob.task == task,
GpuJob.status.in_(["pending", "leased"]),
)
)
).first()
if dup:
return None
job = GpuJob(image_record_id=image_id, task=task, status="pending")
self.session.add(job)
await self.session.flush()
return job
async def lease(
self, token: str, batch_size: int = DEFAULT_BATCH, ttl: int = DEFAULT_LEASE_TTL
) -> list[GpuJob]:
"""Claim up to batch_size pending (or expired-leased) jobs for `token`."""
now = datetime.now(UTC)
picked = (
await self.session.execute(
select(GpuJob.id)
.where(
or_(
GpuJob.status == "pending",
and_(
GpuJob.status == "leased",
GpuJob.lease_expires_at < now,
),
)
)
.order_by(GpuJob.id)
.limit(batch_size)
.with_for_update(skip_locked=True)
)
).scalars().all()
if not picked:
return []
await self.session.execute(
update(GpuJob)
.where(GpuJob.id.in_(picked))
.values(
status="leased", lease_token=token, leased_at=now,
lease_expires_at=now + timedelta(seconds=ttl),
attempts=GpuJob.attempts + 1, updated_at=now,
)
)
# populate_existing: overwrite identity-map copies with the post-UPDATE
# values so the returned jobs reflect the new lease/attempts, not stale
# pre-lease state.
return list(
(
await self.session.execute(
select(GpuJob)
.where(GpuJob.id.in_(picked))
.order_by(GpuJob.id)
.execution_options(populate_existing=True)
)
).scalars()
)
async def heartbeat(
self, token: str, job_ids: list[int], ttl: int = DEFAULT_LEASE_TTL
) -> int:
"""Extend the lease on the agent's in-flight jobs. Returns rows touched."""
now = datetime.now(UTC)
res = await self.session.execute(
update(GpuJob)
.where(
GpuJob.id.in_(job_ids),
GpuJob.lease_token == token,
GpuJob.status == "leased",
)
.values(lease_expires_at=now + timedelta(seconds=ttl), updated_at=now)
)
return res.rowcount or 0
async def complete(self, token: str, job_id: int) -> bool:
"""Close a leased job (after its results were stored). False if the job
isn't leased by this token (a stale/expired submit)."""
job = await self.session.get(GpuJob, job_id)
if job is None or job.status != "leased" or job.lease_token != token:
return False
job.status = "done"
job.lease_token = None
job.lease_expires_at = None
job.error = None
job.updated_at = datetime.now(UTC)
return True
async def fail(self, token: str, job_id: int, error: str) -> bool:
"""Report a failure: re-queue (pending) until MAX_ATTEMPTS, then 'error'."""
job = await self.session.get(GpuJob, job_id)
if job is None or job.lease_token != token:
return False
if job.attempts >= MAX_ATTEMPTS:
job.status = "error"
else:
job.status = "pending"
job.lease_token = None
job.lease_expires_at = None
job.error = (error or "")[:1000]
job.updated_at = datetime.now(UTC)
return True
async def release(self, token: str, job_ids: list[int]) -> int:
"""Hand the agent's still-leased jobs back to pending NOW (graceful stop),
so another worker picks them up immediately instead of waiting out the
lease. Scoped to the token's own leases. Returns rows released."""
if not job_ids:
return 0
now = datetime.now(UTC)
res = await self.session.execute(
update(GpuJob)
.where(
GpuJob.id.in_(job_ids),
GpuJob.lease_token == token,
GpuJob.status == "leased",
)
.values(
status="pending", lease_token=None, leased_at=None,
lease_expires_at=None, updated_at=now,
)
)
return res.rowcount or 0
async def recover_orphaned(self) -> int:
"""Reset every expired lease back to pending — catches agents that died
mid-job (no graceful release). Run on a short beat so the queue recovers
+ reads honestly even when no worker is actively leasing. Returns rows
recovered."""
now = datetime.now(UTC)
res = await self.session.execute(
update(GpuJob)
.where(GpuJob.status == "leased", GpuJob.lease_expires_at < now)
.values(
status="pending", lease_token=None, leased_at=None,
lease_expires_at=None, updated_at=now,
)
)
return res.rowcount or 0
-490
View File
@@ -1,490 +0,0 @@
"""Production heads: train + score the per-concept classifiers (#114).
The eval (#1130, tag_eval.py) proved the spine; this is its production form.
- TRAIN (sync, ml worker — needs scikit-learn): for every general/character tag
with enough labelled positives, fit a logistic-regression head on the FROZEN
SigLIP embeddings (positives + negatives = rejections + sampled unlabeled),
derive an honest suggest threshold + earned-auto-apply point from CROSS-
VALIDATED scores, and upsert a TagHead row. Reuses tag_eval's proven data
loaders + metric helpers so production heads match the eval's measured numbers.
- SCORE (async, API worker — numpy via pgvector, NO scikit-learn): score one
image's embedding against all current heads → the suggestions the rail shows,
REPLACING Camie predictions + per-tag centroids.
scikit-learn is imported lazily inside the train path so the API worker can still
import this module to enqueue training + to score (scoring needs only numpy).
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import Session
from ...models import (
HeadAutoApplyRun,
HeadTrainingRun,
ImageRecord,
ImageRegion,
MLSettings,
Tag,
TagHead,
TagKind,
TagSuggestionRejection,
)
from ...models.tag import image_tag
from .tag_eval import (
_auto_apply_point,
_ids_with_tag,
_l2norm,
_load_embeddings,
_metrics_from_scores,
_rejected_ids,
_safe_folds,
_sample_unlabeled,
)
log = logging.getLogger(__name__)
DEFAULT_NEG_RATIO = 3
DEFAULT_CV_FOLDS = 5
MIN_POSITIVES_FLOOR = 8 # hard floor; settings.head_min_positives can raise it
_UNLABELED_POOL = 4000
_EXAMPLES_MIN = 8 # need at least this many embedded +/- to fit a head
# Only these tag kinds get heads (the surfaced suggestion categories).
_HEAD_KINDS = (TagKind.general, TagKind.character)
# tag.kind -> the suggestion category the rail groups under.
_CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
class HeadTrainingAlreadyRunning(Exception):
"""Raised by start_head_training_run when a run is already in flight."""
def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
"""Create a HeadTrainingRun (status='running') + dispatch the ml-queue task.
Returns the run id. One training run at a time (light guard)."""
existing = session.execute(
select(HeadTrainingRun.id).where(HeadTrainingRun.status == "running")
).scalar_one_or_none()
if existing is not None:
raise HeadTrainingAlreadyRunning(existing)
norm = _normalize_params(session, params)
run = HeadTrainingRun(
params=norm, status="running", last_progress_at=datetime.now(UTC)
)
session.add(run)
session.flush()
run_id = run.id
from ...tasks.ml import train_heads as _task
_task.delay(run_id)
return run_id
def _settings(session: Session) -> MLSettings:
return session.execute(
select(MLSettings).where(MLSettings.id == 1)
).scalar_one()
def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
params = params or {}
s = _settings(session)
try:
min_pos = max(MIN_POSITIVES_FLOOR, int(params.get("min_positives", s.head_min_positives)))
except (TypeError, ValueError):
min_pos = max(MIN_POSITIVES_FLOOR, s.head_min_positives)
try:
neg_ratio = max(1, int(params.get("neg_ratio", DEFAULT_NEG_RATIO)))
except (TypeError, ValueError):
neg_ratio = DEFAULT_NEG_RATIO
try:
cv_folds = max(2, int(params.get("cv_folds", DEFAULT_CV_FOLDS)))
except (TypeError, ValueError):
cv_folds = DEFAULT_CV_FOLDS
try:
precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999)
except (TypeError, ValueError):
precision_target = s.head_auto_apply_precision
return {
"min_positives": min_pos,
"neg_ratio": neg_ratio,
"cv_folds": cv_folds,
"precision_target": round(precision_target, 4),
}
def _embedder_version(session: Session) -> str:
return _settings(session).embedder_model_version
def _eligible_tag_ids(session: Session, min_pos: int) -> list[int]:
"""Concept tags (general/character) with >= min_pos labelled images — the
set that gets a head. Counts all sources; source-aware filtering (#1133) is
a separate, optional refinement."""
rows = session.execute(
select(Tag.id)
.join(image_tag, image_tag.c.tag_id == Tag.id)
.where(Tag.kind.in_(_HEAD_KINDS))
.group_by(Tag.id)
.having(func.count(image_tag.c.image_record_id) >= min_pos)
).all()
return [r[0] for r in rows]
def train_all_heads(
session: Session, params: dict[str, Any], run: HeadTrainingRun | None = None
) -> dict[str, int]:
"""(Re)train a head for every eligible concept; prune heads whose tag is no
longer eligible. Commits per head so a SIGKILL leaves trained heads durable
(training is idempotent). Returns {n_trained, n_skipped}."""
import numpy as np
cfg = _normalize_params(session, params)
embedding_version = _embedder_version(session)
eligible = _eligible_tag_ids(session, cfg["min_positives"])
eligible_set = set(eligible)
trained = 0
skipped = 0
for i, tag_id in enumerate(eligible):
try:
ok = train_head(session, tag_id, embedding_version, cfg, np)
except Exception:
log.exception("train_head failed for tag %d", tag_id)
ok = False
session.commit()
trained += int(ok)
skipped += int(not ok)
if run is not None and i % 10 == 0:
run.last_progress_at = datetime.now(UTC)
session.commit()
# Retire heads whose concept dropped out of the eligible set (lost its
# positives, or the tag was re-kinded) so stale heads can't keep suggesting.
if eligible_set:
session.execute(delete(TagHead).where(TagHead.tag_id.not_in(eligible_set)))
else:
session.execute(delete(TagHead))
session.commit()
return {"n_trained": trained, "n_skipped": skipped}
def train_head(
session: Session, tag_id: int, embedding_version: str, cfg: dict, np
) -> bool:
"""Fit + upsert one head. Returns True if a head was written, False if the
concept had too few usable examples to train (the row is then removed)."""
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_predict
pos_ids = _ids_with_tag(session, tag_id)
if len(pos_ids) < cfg["min_positives"]:
session.execute(delete(TagHead).where(TagHead.tag_id == tag_id))
return False
pos_set = set(pos_ids)
rejected = [i for i in _rejected_ids(session, tag_id) if i not in pos_set]
want_neg = max(len(pos_ids) * cfg["neg_ratio"], _EXAMPLES_MIN * 4)
sampled = _sample_unlabeled(
session, pos_set | set(rejected), min(_UNLABELED_POOL, want_neg)
)
neg_ids = rejected + [i for i in sampled if i not in pos_set]
emb = _load_embeddings(session, pos_ids + neg_ids)
pos = [emb[i] for i in pos_ids if i in emb]
neg = [emb[i] for i in neg_ids if i in emb]
if len(pos) < _EXAMPLES_MIN or len(neg) < _EXAMPLES_MIN:
session.execute(delete(TagHead).where(TagHead.tag_id == tag_id))
return False
X = np.vstack(pos + neg).astype(np.float32)
y = np.array([1] * len(pos) + [0] * len(neg))
Xn = _l2norm(X, np)
clf = LogisticRegression(max_iter=1000, class_weight="balanced")
cv = StratifiedKFold(
n_splits=_safe_folds(y, cfg["cv_folds"], np), shuffle=True, random_state=0
)
# Honest thresholds from out-of-fold scores; deployable weights from a final
# fit on ALL the data.
cv_probs = cross_val_predict(clf, Xn, y, cv=cv, method="predict_proba")[:, 1]
metrics = _metrics_from_scores(y, cv_probs, np)
auto = _auto_apply_point(y, cv_probs, cfg["precision_target"], np)
clf.fit(Xn, y)
head = session.get(TagHead, tag_id)
if head is None:
head = TagHead(tag_id=tag_id)
session.add(head)
head.embedding_version = embedding_version
head.weights = clf.coef_[0].astype(np.float32).tolist()
head.bias = float(clf.intercept_[0])
head.suggest_threshold = float(metrics["threshold"])
head.auto_apply_threshold = float(auto["threshold"]) if auto else None
head.n_pos = len(pos)
head.n_neg = len(neg)
head.ap = float(metrics["ap"])
head.precision_cv = float(metrics["precision"])
head.recall = float(metrics["recall"])
head.trained_at = datetime.now(UTC)
head.metrics = {"f1": metrics["f1"], "auto_apply": auto}
return True
# --- Scoring (async, API worker) -----------------------------------------
# Score one image against every current head to produce the rail's suggestions.
# A tiny in-process cache holds the stacked weight matrix keyed on (count,
# max(trained_at)) so a retrain invalidates it without per-request weight loads.
_HEADS_CACHE: dict[str, Any] = {"key": None, "heads": None}
async def _current_heads(session: AsyncSession, embedding_version: str):
"""Stacked (W, b, thresholds, tag_id/name/category) for heads matching the
current embedding, cached until the next retrain."""
import numpy as np
sig = (
await session.execute(
select(func.count(), func.max(TagHead.trained_at)).where(
TagHead.embedding_version == embedding_version
)
)
).one()
key = f"{embedding_version}:{sig[0]}:{sig[1].isoformat() if sig[1] else '-'}"
cached = _HEADS_CACHE.get("heads")
if cached is not None and _HEADS_CACHE.get("key") == key:
return cached
rows = (
await session.execute(
select(
TagHead.tag_id, Tag.name, Tag.kind,
TagHead.weights, TagHead.bias,
TagHead.suggest_threshold, TagHead.auto_apply_threshold,
)
.join(Tag, Tag.id == TagHead.tag_id)
.where(TagHead.embedding_version == embedding_version)
)
).all()
if not rows:
loaded = {"W": None, "rows": []}
else:
W = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in rows])
b = np.asarray([r.bias for r in rows], dtype=np.float32)
thr = np.asarray([r.suggest_threshold for r in rows], dtype=np.float32)
meta = [
{
"tag_id": r.tag_id,
"name": r.name,
"category": _CATEGORY.get(r.kind, "general"),
"auto_apply_threshold": r.auto_apply_threshold,
}
for r in rows
]
loaded = {"W": W, "b": b, "thr": thr, "meta": meta}
_HEADS_CACHE["key"] = key
_HEADS_CACHE["heads"] = loaded
return loaded
async def score_image(
session: AsyncSession, image_id: int, threshold_override: float | None = None,
) -> list[dict]:
"""Suggestions for one image from the trained heads: [{tag_id, name,
category, score}], ranked. A concept surfaces when its score clears the
head's own suggest_threshold — or, when threshold_override is given (the
typed-dropdown "show everything" mode), that flat floor instead (0 → every
head). Empty if the image has no embedding or no heads exist yet.
MAX-OVER-BAG: the image is scored as a BAG of embeddings — the whole-image
vector PLUS every concept-region crop the agent embedded (same model
version) — and each head takes its MAX score across the bag. A small/local
concept (glasses, a stomach bulge) that the whole-image vector washes out
can still surface from the crop where it dominates. The whole-image vector is
always in the bag, so this can never score lower than whole-image alone."""
import numpy as np
img = await session.get(ImageRecord, image_id)
if img is None or img.siglip_embedding is None:
return []
settings = await _settings_async(session)
heads = await _current_heads(session, settings.embedder_model_version)
if heads["W"] is None:
return []
bag = [np.asarray(img.siglip_embedding, dtype=np.float32)]
region_vecs = (
await session.execute(
select(ImageRegion.siglip_embedding)
.where(ImageRegion.image_record_id == image_id)
.where(ImageRegion.siglip_embedding.is_not(None))
.where(ImageRegion.embedding_version == settings.embedder_model_version)
)
).all()
for (vec,) in region_vecs:
if vec is not None:
bag.append(np.asarray(vec, dtype=np.float32))
X = np.vstack(bag) # (B, D)
norms = np.linalg.norm(X, axis=1, keepdims=True)
norms[norms == 0] = 1.0
Xn = X / norms
Z = Xn @ heads["W"].T + heads["b"] # (B, H)
probs = (1.0 / (1.0 + np.exp(-Z))).max(axis=0) # (H,) best over the bag
out = []
for i, p in enumerate(probs):
cut = threshold_override if threshold_override is not None else heads["thr"][i]
if p >= cut:
m = heads["meta"][i]
out.append({
"tag_id": m["tag_id"],
"name": m["name"],
"category": m["category"],
"score": float(p),
})
out.sort(key=lambda d: d["score"], reverse=True)
return out
async def _settings_async(session: AsyncSession) -> MLSettings:
return (
await session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
# --- Earned auto-apply (sync, ml worker) ---------------------------------
# A graduated head can apply its tag to images it scores above the head's
# auto_apply_threshold, without a human. Gated by a master switch + a support
# floor so a precise-looking but under-supported head can't spray tags.
_AUTO_APPLY_CHUNK = 5000
class HeadAutoApplyAlreadyRunning(Exception):
"""Raised when an auto-apply sweep is already in flight."""
class HeadAutoApplyDisabled(Exception):
"""Raised when a real (non-dry-run) sweep is requested but the master
switch (head_auto_apply_enabled) is off."""
def start_head_auto_apply_run(session: Session, params: dict[str, Any]) -> int:
"""Create a HeadAutoApplyRun + dispatch the ml-queue sweep. dry_run previews
(writes nothing); a real sweep needs the master switch on. One run at a time."""
dry_run = bool((params or {}).get("dry_run", False))
existing = session.execute(
select(HeadAutoApplyRun.id).where(HeadAutoApplyRun.status == "running")
).scalar_one_or_none()
if existing is not None:
raise HeadAutoApplyAlreadyRunning(existing)
if not dry_run and not _settings(session).head_auto_apply_enabled:
raise HeadAutoApplyDisabled()
run = HeadAutoApplyRun(
dry_run=dry_run, params={"dry_run": dry_run}, status="running",
last_progress_at=datetime.now(UTC),
)
session.add(run)
session.flush()
run_id = run.id
from ...tasks.ml import apply_head_tags as _task
_task.delay(run_id)
return run_id
def _auto_apply_heads(session: Session, embedding_version: str, min_pos: int):
"""Eligible heads to fire: graduated (auto_apply_threshold set), enough
support, current embedding. Returns the row list (tag_id/name/weights/...)."""
return session.execute(
select(
TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias,
TagHead.auto_apply_threshold,
)
.join(Tag, Tag.id == TagHead.tag_id)
.where(TagHead.embedding_version == embedding_version)
.where(TagHead.auto_apply_threshold.is_not(None))
.where(TagHead.n_pos >= min_pos)
).all()
def auto_apply_sweep(
session: Session, run: HeadAutoApplyRun, dry_run: bool
) -> dict[str, Any]:
"""Score every embedded image against the eligible heads and apply (or, for
dry_run, just count) each head's tag where score >= its auto_apply_threshold
and the tag isn't already applied or rejected on that image. Streams
embeddings in chunks; commits per chunk on a real run. Returns
{n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
import numpy as np
from sqlalchemy.dialects.postgresql import insert as pg_insert
settings = _settings(session)
rows = _auto_apply_heads(
session, settings.embedder_model_version,
settings.head_auto_apply_min_positives,
)
if not rows:
return {"n_applied": 0, "concepts": []}
W = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in rows])
b = np.asarray([r.bias for r in rows], dtype=np.float32)
thr = np.asarray([r.auto_apply_threshold for r in rows], dtype=np.float32)
tag_ids = [r.tag_id for r in rows]
names = [r.name for r in rows]
# Skip images that already carry, or have rejected, each tag.
skip = {tid: set() for tid in tag_ids}
for tid in tag_ids:
for (iid,) in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
):
skip[tid].add(iid)
for (iid,) in session.execute(
select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == tid
)
):
skip[tid].add(iid)
applied = [0] * len(rows)
scanned = 0
all_ids = list(session.execute(
select(ImageRecord.id).where(ImageRecord.siglip_embedding.is_not(None))
).scalars())
for start in range(0, len(all_ids), _AUTO_APPLY_CHUNK):
chunk = all_ids[start:start + _AUTO_APPLY_CHUNK]
emb = _load_embeddings(session, chunk)
cids = [i for i in chunk if i in emb]
if not cids:
continue
Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b))) # (N, H)
scanned += len(cids)
for h in range(len(rows)):
tid = tag_ids[h]
for idx in np.where(probs[:, h] >= thr[h])[0]:
iid = cids[int(idx)]
if iid in skip[tid]:
continue
skip[tid].add(iid)
applied[h] += 1
if not dry_run:
session.execute(
pg_insert(image_tag)
.values(image_record_id=iid, tag_id=tid, source="head_auto")
.on_conflict_do_nothing()
)
if not dry_run:
session.commit()
run.last_progress_at = datetime.now(UTC)
session.commit()
concepts = [
{"tag_id": tag_ids[h], "name": names[h], "applied": applied[h],
"scanned": scanned, "threshold": float(thr[h])}
for h in range(len(rows))
]
return {"n_applied": sum(applied), "concepts": concepts}
-59
View File
@@ -1,59 +0,0 @@
"""Region read/write for the crop pipeline (#114).
The GPU agent's results endpoint calls replace_regions() to store a freshly
detected/embedded set; the character matcher + concept-bag scorer read via
get_regions(). Replacement is scoped BY KIND so the figure pipeline and the
concept pipeline don't clobber each other.
"""
from typing import Any
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import ImageRegion
class RegionService:
def __init__(self, session: AsyncSession):
self.session = session
async def get_regions(
self, image_id: int, kinds: list[str] | None = None
) -> list[ImageRegion]:
stmt = select(ImageRegion).where(ImageRegion.image_record_id == image_id)
if kinds:
stmt = stmt.where(ImageRegion.kind.in_(kinds))
return list(
(await self.session.execute(stmt.order_by(ImageRegion.id))).scalars()
)
async def replace_regions(
self, image_id: int, kinds: list[str], regions: list[dict[str, Any]]
) -> int:
"""Replace this image's regions OF THE GIVEN KINDS with `regions` (a
re-detect/re-propose supersedes the prior set without touching other
kinds). Each region dict: {kind, bbox:(x,y,w,h), score?, detector_version?,
crop_version?, embedding_version?, ccip_embedding?, siglip_embedding?}.
Returns the number inserted."""
await self.session.execute(
delete(ImageRegion)
.where(ImageRegion.image_record_id == image_id)
.where(ImageRegion.kind.in_(kinds))
)
n = 0
for r in regions:
rx, ry, rw, rh = r["bbox"]
self.session.add(ImageRegion(
image_record_id=image_id, kind=r["kind"],
frame_time=r.get("frame_time"),
rx=rx, ry=ry, rw=rw, rh=rh,
score=r.get("score"),
detector_version=r.get("detector_version"),
crop_version=r.get("crop_version"),
embedding_version=r.get("embedding_version"),
ccip_embedding=r.get("ccip_embedding"),
siglip_embedding=r.get("siglip_embedding"),
))
n += 1
return n
+205 -73
View File
@@ -1,23 +1,24 @@
"""The suggestion read-path: trained HEADS score one image's frozen embedding """The suggestion read-path: raw predictions + centroids -> alias-resolved,
into alias-resolved, category-grouped, ranked suggestions. threshold-filtered, category-grouped, ranked suggestions for one image.
Tagging-v2 (#114): suggestions now come from the per-concept heads that LEARN
from the operator's tags (services/ml/heads.py) — the Camie prediction source
and the per-tag SigLIP centroid have been REMOVED. A head exists only for an
existing concept tag, so every suggestion is a canonical tag (no raw model key,
no alias remap, no creates-new). Rejected tags stay in the list FLAGGED (not
dropped) so the rail can show + reverse a dismissal.
""" """
from dataclasses import dataclass, field from dataclasses import dataclass, field
from sqlalchemy import select from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ...models import ImageRecord, TagSuggestionRejection from ...models import (
ImagePrediction,
ImageRecord,
MLSettings,
Tag,
TagSuggestionRejection,
)
from ...models.tag import image_tag from ...models.tag import image_tag
from .ccip import match_image as ccip_match_image from .aliases import AliasService
from .heads import score_image from .centroids import CentroidService
from .tag_name import normalize as normalize_tag_name
from .tagger import SURFACED_CATEGORIES
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -28,7 +29,7 @@ class Suggestion:
display_name: str display_name: str
category: str category: str
score: float score: float
source: str # 'head' | 'ccip' | 'both' (Camie tagger/centroid removed in v2) source: str # 'tagger' | 'centroid' | 'both'
creates_new_tag: bool creates_new_tag: bool
# raw_name = the booru model vocab key behind this suggestion. It's the key # raw_name = the booru model vocab key behind this suggestion. It's the key
# an alias MUST be stored under (resolution looks up the raw key), so the # an alias MUST be stored under (resolution looks up the raw key), so the
@@ -38,11 +39,6 @@ class Suggestion:
# via_alias = this suggestion was surfaced because an operator alias remapped # via_alias = this suggestion was surfaced because an operator alias remapped
# the raw prediction to this canonical tag. Lets the UI mark it + offer undo. # the raw prediction to this canonical tag. Lets the UI mark it + offer undo.
via_alias: bool = False via_alias: bool = False
# rejected = the operator dismissed this tag for this image (a stored
# TagSuggestionRejection). It stays in the list — flagged, not dropped — so
# the rejection is VISIBLE and REVERSIBLE in the rail (misclick recovery,
# operator-asked 2026-06-27) instead of silently vanishing or re-suggesting.
rejected: bool = False
@dataclass @dataclass
@@ -53,24 +49,67 @@ class SuggestionList:
class SuggestionService: class SuggestionService:
def __init__(self, session: AsyncSession): def __init__(self, session: AsyncSession):
self.session = session self.session = session
self.aliases = AliasService(session)
self.centroids = CentroidService(session)
async def _settings(self) -> MLSettings:
return (
await self.session.execute(select(MLSettings).where(MLSettings.id == 1))
).scalar_one()
async def _load_predictions(self, image_id: int) -> dict:
"""Predictions for one image from the normalized image_prediction
table (#768), in the {raw_name: {category, confidence}} shape the rest
of this service consumed from the old JSON column — so all downstream
threshold/alias/merge logic is unchanged."""
rows = (
await self.session.execute(
select(
ImagePrediction.raw_name,
ImagePrediction.category,
ImagePrediction.score,
).where(ImagePrediction.image_record_id == image_id)
)
).all()
return {
r.raw_name: {"category": r.category, "confidence": r.score}
for r in rows
}
def _threshold_for(
self, s: MLSettings, category: str, override: float | None = None,
) -> float:
# 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired;
# both fall through to the 1.01 "never surfaces" default like any
# unsurfaced category.
# override (the typed-dropdown "show everything the model saw" mode)
# applies to the surfaced categories only — unsurfaced ones are already
# skipped before the threshold check, so they can't leak in.
if override is not None:
return override
return {
"character": s.suggestion_threshold_character,
"general": s.suggestion_threshold_general,
}.get(category, 1.01)
async def for_image( async def for_image(
self, image_id: int, threshold_override: float | None = None, self, image_id: int, *, threshold_override: float | None = None,
) -> SuggestionList: ) -> SuggestionList:
"""Head-scored suggestions for one image, grouped by category and ranked. """Ranked suggestions for one image.
Each trained head scores the image's frozen embedding; a concept surfaces threshold_override surfaces EVERY stored tagger prediction (down to the
when its score clears the head's own suggest threshold. threshold_override ingest STORE_FLOOR) regardless of the configured per-category suggestion
(used by the typed tag-input dropdown's "show everything" mode) replaces thresholds — backs the tag-input dropdown's "search all of the model's
that per-head cut with a flat floor (0 → every head), so a low-scoring predictions, including low-confidence ones, in the canonical formatting"
concept can still be typed + picked in canonical formatting. mode (operator-asked 2026-06-09). The Suggestions panel still calls with
no override so it stays the curated above-threshold list."""
Already-applied tags are dropped; rejected tags stay FLAGGED and sink to
the bottom of their category so a dismissal is visible + reversible."""
img = await self.session.get(ImageRecord, image_id) img = await self.session.get(ImageRecord, image_id)
if img is None: if img is None:
return SuggestionList() return SuggestionList()
settings = await self._settings()
predictions: dict = await self._load_predictions(image_id)
applied = set( applied = set(
( (
await self.session.execute( await self.session.execute(
@@ -90,50 +129,148 @@ class SuggestionService:
).scalars().all() ).scalars().all()
) )
hits = await score_image( # --- Camie predictions ---
self.session, image_id, threshold_override=threshold_override # candidates carry (raw_name, display_name, category, confidence).
) # raw_name = the booru-formatted vocab key, kept for alias_map
# CCIP character matches OVERLAY the SigLIP character heads — a # lookup since alias rows are hand-curated against raw keys.
# complementary, identity-specialized signal with different failure modes # display_name = normalize_tag_name(raw_name) — what the operator
# (CCIP needs a detected figure; heads work whole-image). Merged by tag: # sees AND what gets written to tag.name on Accept.
# 'both' when they corroborate, taking the higher score. candidates: list[tuple[str, str, str, float]] = []
ccip_hits = await ccip_match_image(self.session, image_id) for name, p in predictions.items():
category = p.get("category", "general")
if category not in SURFACED_CATEGORIES:
continue
conf = float(p.get("confidence", 0.0))
if conf < self._threshold_for(settings, category, threshold_override):
continue
display = normalize_tag_name(name)
if display is None:
# emoticon / pure-punctuation vocab entry — drop entirely
continue
candidates.append((name, display, category, conf))
merged: dict[tuple[str, int], dict] = {} alias_map = await self.aliases.resolve_many(
for h in hits: [(raw, c) for raw, _disp, c, _conf in candidates]
merged[(h["category"], h["tag_id"])] = { )
"name": h["name"], "score": h["score"], "source": "head",
} merged: dict[object, Suggestion] = {}
for c in ccip_hits:
key = ("character", c["tag_id"]) def _merge(key, sug: Suggestion):
ex = merged.get(key) existing = merged.get(key)
if ex is not None: if existing is None:
ex["source"] = "both" merged[key] = sug
ex["score"] = max(ex["score"], c["score"]) elif sug.score > existing.score:
merged[key] = Suggestion(
canonical_tag_id=existing.canonical_tag_id,
display_name=existing.display_name,
category=existing.category,
score=sug.score,
source="both"
if existing.source != sug.source
else existing.source,
creates_new_tag=existing.creates_new_tag,
# Keep the alias identity from `existing`: the tagger pass
# (which carries raw_name / via_alias) runs before centroid
# augmentation, so it's always the first writer for a key.
raw_name=existing.raw_name,
via_alias=existing.via_alias,
)
for raw, display, category, conf in candidates:
canonical = alias_map.get((raw, category))
if canonical is not None:
if canonical.id in applied or canonical.id in rejected:
continue
_merge(
canonical.id,
Suggestion(
canonical_tag_id=canonical.id,
display_name=canonical.name,
category=category,
score=conf,
source="tagger",
creates_new_tag=False,
raw_name=raw,
via_alias=True,
),
)
else: else:
merged[key] = { # Case-insensitive match on BOTH the raw camie key AND
"name": c["name"], "score": c["score"], "source": "ccip", # the normalized form — covers legacy underscore-named
} # Tag rows accepted before normalization shipped, AND
# any tag the operator created with the human form.
existing_tag = (
await self.session.execute(
select(Tag).where(
func.lower(Tag.name).in_(
[raw.lower(), display.lower()]
)
)
)
).scalars().first()
if existing_tag is not None:
if (
existing_tag.id in applied
or existing_tag.id in rejected
):
continue
_merge(
existing_tag.id,
Suggestion(
canonical_tag_id=existing_tag.id,
display_name=existing_tag.name,
category=category,
score=conf,
source="tagger",
creates_new_tag=False,
raw_name=raw,
via_alias=False,
),
)
else:
_merge(
f"raw:{display}:{category}",
Suggestion(
canonical_tag_id=None,
display_name=display,
category=category,
score=conf,
source="tagger",
creates_new_tag=True,
raw_name=raw,
via_alias=False,
),
)
# --- Centroid augmentation ---
hits = await self.centroids.find_similar_tags(image_id, limit=30)
for hit in hits:
if hit.similarity < settings.centroid_similarity_threshold:
continue
if hit.tag_id in applied or hit.tag_id in rejected:
continue
tag = await self.session.get(Tag, hit.tag_id)
if tag is None:
continue
cat = tag.kind.value if hasattr(tag.kind, "value") else str(tag.kind)
display_cat = cat if cat in SURFACED_CATEGORIES else "general"
_merge(
tag.id,
Suggestion(
canonical_tag_id=tag.id,
display_name=tag.name,
category=display_cat,
score=hit.similarity,
source="centroid",
creates_new_tag=False,
),
)
result = SuggestionList() result = SuggestionList()
for (cat, tag_id), m in merged.items(): for sug in merged.values():
if tag_id in applied: result.by_category.setdefault(sug.category, []).append(sug)
continue
result.by_category.setdefault(cat, []).append(
Suggestion(
canonical_tag_id=tag_id,
display_name=m["name"],
category=cat,
score=m["score"],
source=m["source"],
creates_new_tag=False,
rejected=tag_id in rejected,
)
)
for cat in result.by_category: for cat in result.by_category:
# Live suggestions first (by score), rejected ones sink to the result.by_category[cat].sort(key=lambda s: s.score, reverse=True)
# bottom of the category — visible for recovery, out of the way.
result.by_category[cat].sort(key=lambda s: (s.rejected, -s.score))
return result return result
async def for_selection( async def for_selection(
@@ -160,11 +297,6 @@ class SuggestionService:
for s in items: for s in items:
if s.canonical_tag_id is None or s.creates_new_tag: if s.canonical_tag_id is None or s.creates_new_tag:
continue continue
# for_image keeps rejected tags (flagged) for the rail;
# bulk consensus must still ignore them — a tag dismissed on
# an image isn't a suggestion for that image.
if s.rejected:
continue
st = stats.get(s.canonical_tag_id) st = stats.get(s.canonical_tag_id)
if st is None: if st is None:
st = { st = {
-430
View File
@@ -1,430 +0,0 @@
"""Head-vs-centroid tagging eval (#1130, milestone #114 slice 1).
Proves the "frozen embedding + small trained head (with negatives)" spine on the
operator's OWN data, reusing the SigLIP embeddings already stored on
image_record. For each concept tag it compares:
- CENTROID baseline (the old approach): cosine to the mean of positive vectors.
- HEAD (the new approach): logistic regression trained on positives + negatives.
and reports cross-validated precision/recall/AP for both, a LEARNING CURVE
(accuracy as the number of tagged positives grows), and example image ids to
eyeball.
numpy + scikit-learn are imported LAZILY inside run_eval so the API worker (base
image, no ML stack) can still import start_tag_eval_run to enqueue the ml-queue
task — the heavy compute only runs on the ml worker.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from ...models import (
ImageRecord,
Tag,
TagEvalRun,
TagKind,
TagPositiveConfirmation,
TagSuggestionRejection,
)
from ...models.tag import image_tag
log = logging.getLogger(__name__)
# The operator's real concept list (mix of whole-ish + small/local cues). The
# admin trigger can override; this is the default eval set.
DEFAULT_CONCEPTS = [
"glasses", "cat", "dog", "horse", "goblin",
"cum", "lactation", "fellatio", "xray", "stomach bulge",
]
DEFAULT_CURVE_POINTS = [10, 30, 100, 300]
DEFAULT_NEG_RATIO = 3 # negatives per positive (rejections + sampled unlabeled)
DEFAULT_CV_FOLDS = 5
MIN_POSITIVES = 8 # below this, a concept can't be evaluated meaningfully
_UNLABELED_POOL = 4000 # cap on sampled unlabeled rows pulled per concept
_EXAMPLES_K = 12
def start_tag_eval_run(session: Session, params: dict[str, Any]) -> int:
"""Create a TagEvalRun (status='running') and dispatch the ml-queue task.
Returns the new run id. Light guard: one running eval at a time."""
existing = session.execute(
select(TagEvalRun.id).where(TagEvalRun.status == "running")
).scalar_one_or_none()
if existing is not None:
raise EvalAlreadyRunning(existing)
norm = _normalize_params(params)
run = TagEvalRun(params=norm, status="running", last_progress_at=datetime.now(UTC))
session.add(run)
session.flush()
run_id = run.id
# Same enqueue-by-import pattern api/suggestions.py uses for ml tasks; the
# commit happens in the API handler so row + dispatch are visible together.
from ...tasks.ml import tag_eval_run as _task
_task.delay(run_id)
return run_id
class EvalAlreadyRunning(Exception):
"""Raised by start_tag_eval_run when an eval is already in flight."""
def _normalize_params(params: dict[str, Any] | None) -> dict[str, Any]:
params = params or {}
concepts = [str(c).strip() for c in (params.get("concepts") or []) if str(c).strip()]
try:
neg_ratio = max(1, int(params.get("neg_ratio", DEFAULT_NEG_RATIO)))
except (TypeError, ValueError):
neg_ratio = DEFAULT_NEG_RATIO
try:
cv_folds = max(2, int(params.get("cv_folds", DEFAULT_CV_FOLDS)))
except (TypeError, ValueError):
cv_folds = DEFAULT_CV_FOLDS
try:
auto_top_n = min(max(int(params.get("auto_top_n", 0) or 0), 0), 200)
except (TypeError, ValueError):
auto_top_n = 0
try:
precision_target = min(max(float(params.get("precision_target", 0.97)), 0.5), 0.999)
except (TypeError, ValueError):
precision_target = 0.97
# No explicit concepts and auto-discovery off → fall back to the hand list.
if not concepts and not auto_top_n:
concepts = list(DEFAULT_CONCEPTS)
curve = params.get("curve_points") or DEFAULT_CURVE_POINTS
curve = sorted({int(n) for n in curve if int(n) > 0})
return {
"concepts": concepts,
"neg_ratio": neg_ratio,
"cv_folds": cv_folds,
"auto_top_n": auto_top_n,
"precision_target": round(precision_target, 4),
"curve_points": curve,
}
def _top_general_concepts(session: Session, n: int, min_count: int) -> list[str]:
"""The n most-tagged general (concept) tags with >= min_count images — a fast
server-side way to broaden the eval beyond the hand-picked list (counts all
sources; source-aware filtering is a separate concern)."""
rows = session.execute(
select(Tag.name)
.join(image_tag, image_tag.c.tag_id == Tag.id)
.where(Tag.kind == TagKind.general)
.group_by(Tag.id)
.having(func.count(image_tag.c.image_record_id) >= min_count)
.order_by(func.count(image_tag.c.image_record_id).desc())
.limit(n)
).all()
return [r[0] for r in rows]
def _resolve_tag_id(session: Session, name: str) -> int | None:
"""Case-insensitive tag-name match; if several share a name, take the one
applied to the most images (the one the operator actually uses)."""
rows = session.execute(
select(Tag.id, func.count(image_tag.c.image_record_id))
.outerjoin(image_tag, image_tag.c.tag_id == Tag.id)
.where(func.lower(Tag.name) == name.lower())
.group_by(Tag.id)
.order_by(func.count(image_tag.c.image_record_id).desc())
).all()
return rows[0][0] if rows else None
def _ids_with_tag(session: Session, tag_id: int) -> list[int]:
return [
r[0] for r in session.execute(
select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tag_id)
).all()
]
def _rejected_ids(session: Session, tag_id: int) -> list[int]:
return [
r[0] for r in session.execute(
select(TagSuggestionRejection.image_record_id)
.where(TagSuggestionRejection.tag_id == tag_id)
).all()
]
def _confirmed_ids(session: Session, tag_id: int) -> set[int]:
"""Positives the operator explicitly affirmed ('keep') — excluded from the
doubts list so confirmed-correct images don't resurface every run."""
return {
r[0] for r in session.execute(
select(TagPositiveConfirmation.image_record_id)
.where(TagPositiveConfirmation.tag_id == tag_id)
).all()
}
def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
"""Random image ids (with an embedding) NOT carrying the tag. Concepts are
sparse, so an untagged image is almost always a true negative."""
stmt = (
select(ImageRecord.id)
.where(ImageRecord.siglip_embedding.is_not(None))
.order_by(func.random())
.limit(limit)
)
if exclude:
stmt = stmt.where(ImageRecord.id.not_in(exclude))
return [r[0] for r in session.execute(stmt).all()]
def _load_embeddings(session: Session, ids: list[int]) -> dict[int, Any]:
import numpy as np
out: dict[int, Any] = {}
if not ids:
return out
# Chunk the IN list to stay well under psycopg's parameter ceiling.
for i in range(0, len(ids), 2000):
chunk = ids[i:i + 2000]
for rid, emb in session.execute(
select(ImageRecord.id, ImageRecord.siglip_embedding)
.where(ImageRecord.id.in_(chunk))
.where(ImageRecord.siglip_embedding.is_not(None))
).all():
out[rid] = np.asarray(emb, dtype=np.float32)
return out
def run_eval(session: Session, params: dict[str, Any]) -> dict[str, Any]:
"""Compute the full report. Per-concept failures are captured, not fatal."""
import numpy as np
cfg = _normalize_params(params)
# Auto-discovery: union the explicit concepts with the top-N most-tagged
# general tags (server-side, fast) so the eval can broaden itself.
concepts = list(cfg["concepts"])
if cfg["auto_top_n"]:
seen = {c.lower() for c in concepts}
for name in _top_general_concepts(session, cfg["auto_top_n"], MIN_POSITIVES):
if name.lower() not in seen:
concepts.append(name)
seen.add(name.lower())
cfg["concepts"] = concepts
concepts_out = []
for name in cfg["concepts"]:
try:
concepts_out.append(_eval_concept(session, name, cfg, np))
except Exception as exc: # one bad concept shouldn't kill the run
log.exception("tag-eval concept %r failed", name)
concepts_out.append({"name": name, "skipped": f"error: {exc}"})
return {
"generated_at": datetime.now(UTC).isoformat(),
"params": cfg,
"concepts": concepts_out,
}
def _eval_concept(session: Session, name: str, cfg: dict, np) -> dict[str, Any]:
tag_id = _resolve_tag_id(session, name)
if tag_id is None:
return {"name": name, "skipped": "no such tag"}
pos_ids = _ids_with_tag(session, tag_id)
if len(pos_ids) < MIN_POSITIVES:
return {"name": name, "tag_id": tag_id, "n_pos": len(pos_ids),
"skipped": f"too few positives (<{MIN_POSITIVES})"}
neg_ratio = cfg["neg_ratio"]
pos_set = set(pos_ids)
rejected = [i for i in _rejected_ids(session, tag_id) if i not in pos_set]
want_neg = max(len(pos_ids) * neg_ratio, _EXAMPLES_K * 4)
sampled = _sample_unlabeled(session, pos_set | set(rejected),
min(_UNLABELED_POOL, want_neg))
neg_ids = rejected + [i for i in sampled if i not in pos_set]
emb = _load_embeddings(session, pos_ids + neg_ids)
pos = [(i, emb[i]) for i in pos_ids if i in emb]
neg = [(i, emb[i]) for i in neg_ids if i in emb]
if len(pos) < MIN_POSITIVES or len(neg) < MIN_POSITIVES:
return {"name": name, "tag_id": tag_id, "n_pos": len(pos),
"n_neg": len(neg), "skipped": "too few embedded examples"}
ids = np.array([i for i, _ in pos] + [i for i, _ in neg])
X = np.vstack([v for _, v in pos] + [v for _, v in neg]).astype(np.float32)
y = np.array([1] * len(pos) + [0] * len(neg))
Xn = _l2norm(X, np)
head = _eval_head(Xn, y, cfg["cv_folds"], cfg["precision_target"], np)
centroid = _eval_centroid(Xn, y, cfg["cv_folds"], np)
curve = _learning_curve(Xn, y, cfg["curve_points"], neg_ratio, np)
confirmed = _confirmed_ids(session, tag_id)
examples = _examples(session, Xn, y, ids, np, set(rejected), confirmed)
return {
"name": name, "tag_id": tag_id,
"n_pos": len(pos), "n_neg": len(neg),
"n_rejected": len(rejected),
"head": head, "centroid": centroid,
"curve": curve, "examples": examples,
}
def _l2norm(X, np):
n = np.linalg.norm(X, axis=1, keepdims=True)
n[n == 0] = 1.0
return X / n
def _metrics_from_scores(y, scores, np) -> dict[str, float]:
from sklearn.metrics import average_precision_score, precision_recall_curve
ap = float(average_precision_score(y, scores))
prec, rec, thr = precision_recall_curve(y, scores)
f1 = (2 * prec * rec) / np.clip(prec + rec, 1e-9, None)
best = int(np.argmax(f1))
# thr has len = len(prec)-1; map best index safely.
t = float(thr[min(best, len(thr) - 1)]) if len(thr) else 0.5
return {
"ap": round(ap, 4),
"precision": round(float(prec[best]), 4),
"recall": round(float(rec[best]), 4),
"f1": round(float(f1[best]), 4),
"threshold": round(t, 4),
}
def _safe_folds(y, folds, np) -> int:
minority = int(min(np.bincount(y)))
return max(2, min(folds, minority))
def _eval_head(Xn, y, folds, target, np) -> dict[str, float]:
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_predict
clf = LogisticRegression(max_iter=1000, class_weight="balanced")
cv = StratifiedKFold(n_splits=_safe_folds(y, folds, np), shuffle=True,
random_state=0)
probs = cross_val_predict(clf, Xn, y, cv=cv, method="predict_proba")[:, 1]
m = _metrics_from_scores(y, probs, np)
m["auto_apply"] = _auto_apply_point(y, probs, target, np)
return m
def _auto_apply_point(y, scores, target, np) -> dict | None:
"""The auto-apply operating point: the threshold that yields the MOST recall
while holding precision >= target. This answers 'could this concept fire
without a human, and how much would it catch?' Returns None if no threshold
reaches the precision target (concept not auto-apply-ready)."""
from sklearn.metrics import precision_recall_curve
prec, rec, thr = precision_recall_curve(y, scores)
best = None # (threshold, precision, recall) maximizing recall s.t. prec>=target
for i in range(len(thr)): # thr[i] corresponds to prec[i], rec[i]
if prec[i] >= target and (best is None or rec[i] > best[2]):
best = (float(thr[i]), float(prec[i]), float(rec[i]))
if best is None:
return None
return {
"target": round(float(target), 4),
"threshold": round(best[0], 4),
"precision": round(best[1], 4),
"recall": round(best[2], 4),
}
def _eval_centroid(Xn, y, folds, np) -> dict[str, float]:
"""Cross-validated cosine-to-positive-mean — the OLD method's quality."""
from sklearn.model_selection import StratifiedKFold
cv = StratifiedKFold(n_splits=_safe_folds(y, folds, np), shuffle=True,
random_state=0)
scores = np.zeros(len(y), dtype=np.float32)
for train, test in cv.split(Xn, y):
c = Xn[train][y[train] == 1].mean(axis=0)
cn = c / (np.linalg.norm(c) or 1.0)
scores[test] = Xn[test] @ cn
return _metrics_from_scores(y, scores, np)
def _learning_curve(Xn, y, points, neg_ratio, np) -> list[dict[str, float]]:
"""Hold out a fixed test split; train the head on a growing number of
positives and watch AP/F1 climb — answers 'does tagging more sharpen it?'"""
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(0)
idx = np.arange(len(y))
try:
tr, te = train_test_split(idx, test_size=0.3, stratify=y, random_state=0)
except ValueError:
return []
tr_pos = tr[y[tr] == 1]
tr_neg = tr[y[tr] == 0]
out = []
for n in points:
if n > len(tr_pos):
break
sp = rng.choice(tr_pos, size=n, replace=False)
nn = min(len(tr_neg), n * neg_ratio)
sn = rng.choice(tr_neg, size=nn, replace=False)
sub = np.concatenate([sp, sn])
clf = LogisticRegression(max_iter=1000, class_weight="balanced")
clf.fit(Xn[sub], y[sub])
prob = clf.predict_proba(Xn[te])[:, 1]
m = _metrics_from_scores(y[te], prob, np)
out.append({"n_pos": int(n), "ap": m["ap"], "f1": m["f1"]})
return out
def _examples(session, Xn, y, ids, np, rejected_set, confirmed_set) -> dict[str, list[dict]]:
"""Train on all data, then surface: top-scoring negatives the operator has
NOT already rejected (= fresh suggestions) and lowest-scoring POSITIVES the
operator has NOT already confirmed (= unreviewed doubts). Excluding rejected
ids stops an adjudicated near-miss from resurfacing in 'would suggest';
excluding confirmed ids stops a 'kept' correct positive from resurfacing in
'head doubts' every run. Resolves thumbnail urls for a self-contained report."""
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter=1000, class_weight="balanced")
clf.fit(Xn, y)
s = clf.predict_proba(Xn)[:, 1]
neg_idx = np.where(y == 0)[0]
pos_idx = np.where(y == 1)[0]
top_neg = []
for i in neg_idx[np.argsort(s[neg_idx])[::-1]]: # high score → low
rid = int(ids[i])
if rid in rejected_set:
continue # already told the head 'no' — don't re-suggest it
top_neg.append(rid)
if len(top_neg) >= _EXAMPLES_K:
break
low_pos = []
for i in pos_idx[np.argsort(s[pos_idx])]: # low score → high
rid = int(ids[i])
if rid in confirmed_set:
continue # already kept/confirmed — don't re-doubt it
low_pos.append(rid)
if len(low_pos) >= _EXAMPLES_K:
break
thumbs = _resolve_thumbs(session, top_neg + low_pos)
return {
"head_would_suggest": [thumbs[i] for i in top_neg if i in thumbs],
"head_doubts_positive": [thumbs[i] for i in low_pos if i in thumbs],
}
def _resolve_thumbs(session, ids: list[int]) -> dict[int, dict]:
from ..gallery_service import thumbnail_url
out: dict[int, dict] = {}
if not ids:
return out
for rid, tp, sha, mime in session.execute(
select(
ImageRecord.id, ImageRecord.thumbnail_path,
ImageRecord.sha256, ImageRecord.mime,
).where(ImageRecord.id.in_(ids))
).all():
out[rid] = {"id": rid, "thumbnail_url": thumbnail_url(tp, sha, mime)}
return out
+1 -31
View File
@@ -66,10 +66,6 @@ class ProvenanceService:
).scalars().all() ).scalars().all()
return [_attachment_dict(a) for a in rows] return [_attachment_dict(a) for a in rows]
async def _attachment_by_id(self, attachment_id: int) -> list[dict]:
att = await self.session.get(PostAttachment, attachment_id)
return [_attachment_dict(att)] if att is not None else []
async def for_image(self, image_id: int) -> dict | None: async def for_image(self, image_id: int) -> dict | None:
rec = await self.session.get(ImageRecord, image_id) rec = await self.session.get(ImageRecord, image_id)
if rec is None: if rec is None:
@@ -89,33 +85,7 @@ class ProvenanceService:
) )
rows = (await self.session.execute(stmt)).all() rows = (await self.session.execute(stmt)).all()
post_ids = [ip.post_id for ip, _p, _s, _a in rows] post_ids = [ip.post_id for ip, _p, _s, _a in rows]
# Prefer the EXACT archive this file came out of (milestone #87): if the attachments = await self._attachments_for_posts(post_ids)
# originating post's provenance row records from_attachment_id, the image
# was extracted from that one .zip/.rar, so show only it — not the dozens
# of unrelated archives a "High Resolution Files" bundle post carries.
from_att_id = next(
(
ip.from_attachment_id
for ip, _p, _s, _a in rows
if ip.post_id == rec.primary_post_id
and ip.from_attachment_id is not None
),
None,
)
if from_att_id is not None:
attachments = await self._attachment_by_id(from_att_id)
else:
# No recorded containing archive (loose download, or pre-backfill):
# scope to the originating post only, not every pHash-linked post.
# primary_post_id is the post this file was actually captured from;
# fall back to all linked posts when it's unset (older rows /
# filesystem imports).
attach_post_ids = (
[rec.primary_post_id]
if rec.primary_post_id is not None
else post_ids
)
attachments = await self._attachments_for_posts(attach_post_ids)
return { return {
"image_id": image_id, "image_id": image_id,
"provenance": [ "provenance": [
+16 -47
View File
@@ -53,28 +53,12 @@ class TagDirectoryService:
raise ValueError("limit must be between 1 and 200") raise ValueError("limit must be between 1 and 200")
fandom = aliased(Tag) fandom = aliased(Tag)
member = aliased(Tag) count_col = func.count(image_tag.c.image_record_id).label("image_count")
# image_count aggregates the tag's images INCLUDING, for a fandom, every stmt = (
# image carrying one of its characters. `member` is the tag actually on select(Tag, fandom.name.label("fandom_name"), count_col)
# the image; an image counts for the outer Tag if that applied tag IS the .outerjoin(fandom, Tag.fandom_id == fandom.id)
# Tag (direct) OR is a character whose fandom_id is the Tag (the fandom .outerjoin(image_tag, image_tag.c.tag_id == Tag.id)
# leg). Both legs correlate the OUTER Tag.id at a SINGLE level — a nested .group_by(Tag.id, fandom.name)
# `tag_id IN (SELECT ... WHERE member.fandom_id == Tag.id)` does NOT
# correlate and silently counts every fandom-character globally (~every
# tag collapses to the same inflated number). DISTINCT so an image with
# the fandom AND ≥1 of its characters counts once; a correlated scalar
# subquery keeps it 1 row per tag with no group_by.
count_col = (
select(func.count(image_tag.c.image_record_id.distinct()))
.select_from(image_tag)
.join(member, member.id == image_tag.c.tag_id)
.where(or_(image_tag.c.tag_id == Tag.id, member.fandom_id == Tag.id))
.correlate(Tag)
.scalar_subquery()
.label("image_count")
)
stmt = select(Tag, fandom.name.label("fandom_name"), count_col).outerjoin(
fandom, Tag.fandom_id == fandom.id
) )
if kind is not None: if kind is not None:
stmt = stmt.where(Tag.kind == kind) stmt = stmt.where(Tag.kind == kind)
@@ -117,34 +101,19 @@ class TagDirectoryService:
async def _previews(self, tag_ids: list[int]) -> dict[int, list[str]]: async def _previews(self, tag_ids: list[int]) -> dict[int, list[str]]:
if not tag_ids: if not tag_ids:
return {} return {}
# Preview pool per card = images tagged with the card's tag DIRECTLY,
# UNION images carrying a character whose fandom_id is the card's tag (so
# a fandom card previews its characters' images). UNION dedups the
# overlap. For non-fandom cards the via-fandom leg is empty.
member = aliased(Tag)
direct = select(
image_tag.c.tag_id.label("card_id"),
image_tag.c.image_record_id.label("image_record_id"),
).where(image_tag.c.tag_id.in_(tag_ids))
via_fandom = (
select(
member.fandom_id.label("card_id"),
image_tag.c.image_record_id.label("image_record_id"),
)
.select_from(image_tag)
.join(member, member.id == image_tag.c.tag_id)
.where(member.fandom_id.in_(tag_ids))
)
scoped = direct.union(via_fandom).subquery()
rn = func.row_number().over( rn = func.row_number().over(
partition_by=scoped.c.card_id, partition_by=image_tag.c.tag_id,
order_by=scoped.c.image_record_id.desc(), order_by=image_tag.c.image_record_id.desc(),
).label("rn") ).label("rn")
sub = select( sub = (
scoped.c.card_id.label("tag_id"), select(
scoped.c.image_record_id.label("image_record_id"), image_tag.c.tag_id.label("tag_id"),
image_tag.c.image_record_id.label("image_record_id"),
rn, rn,
).subquery() )
.where(image_tag.c.tag_id.in_(tag_ids))
.subquery()
)
stmt = ( stmt = (
select( select(
sub.c.tag_id, sub.c.tag_id,
+2 -48
View File
@@ -1,10 +1,4 @@
"""Shared tag-with-fandom query columns, serialization, and membership. """Shared tag-with-fandom query columns + serialization.
`image_in_tag_scope`/`image_in_any_tag_scope` are the SINGLE source of truth for
"which images belong to a tag" — a fandom tag also owns the images of its
characters (Tag.fandom_id), derived at query time rather than materialized. The
gallery scope, directory count/previews, and the cleanup impact count all route
through them so a fandom can never under-count its characters' images.
Resolving a character tag's fandom NAME via a Tag self-join (Tag.fandom_id -> the Resolving a character tag's fandom NAME via a Tag self-join (Tag.fandom_id -> the
fandom Tag) and serializing the canonical fandom Tag) and serializing the canonical
@@ -17,47 +11,7 @@ TagDirectoryService selects the FULL Tag ORM plus an image-count aggregate (a
different select shape), so it keeps its own variant — not folded in here. different select shape), so it keeps its own variant — not folded in here.
""" """
from sqlalchemy import exists, or_, select from ..models import Tag
from sqlalchemy.orm import aliased
from ..models import ImageRecord, Tag
from ..models.tag import image_tag
def _fandom_member_char_ids(tids):
"""Subquery of character tag ids owned by any fandom in `tids` (via
Tag.fandom_id). Empty for non-fandom tids, so callers can OR it in
unconditionally — a fandom aggregates its characters' images, every other
kind degrades to direct-only."""
char = aliased(Tag)
return select(char.id).where(char.fandom_id.in_(tids))
def image_in_tag_scope(tid):
"""Correlated EXISTS: the current ImageRecord belongs to tag `tid` — either
tagged with it DIRECTLY, or (when `tid` is a fandom) carrying a character
whose fandom_id == `tid`. This is the SINGLE membership predicate shared by
the gallery scope, the directory count/previews, and the cleanup impact
count, so a fandom never under-counts the images of its characters."""
return exists().where(
image_tag.c.image_record_id == ImageRecord.id,
or_(
image_tag.c.tag_id == tid,
image_tag.c.tag_id.in_(_fandom_member_char_ids([tid])),
),
)
def image_in_any_tag_scope(tids):
"""Like `image_in_tag_scope` but for an OR-group: the image carries AT LEAST
ONE of `tids` directly, or a character of any fandom in `tids`."""
return exists().where(
image_tag.c.image_record_id == ImageRecord.id,
or_(
image_tag.c.tag_id.in_(tids),
image_tag.c.tag_id.in_(_fandom_member_char_ids(tids)),
),
)
def fandom_join_alias(): def fandom_join_alias():
+1 -159
View File
@@ -9,7 +9,7 @@ from sqlalchemy import and_, case, exists, func, select, text, update
from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import AsyncSession
from ..models import HeadMetric, Tag, TagHead, TagKind, image_tag from ..models import Tag, TagKind, image_tag
from ..models.tag_allowlist import TagAllowlist from ..models.tag_allowlist import TagAllowlist
from ..models.tag_reference_embedding import TagReferenceEmbedding from ..models.tag_reference_embedding import TagReferenceEmbedding
from .db_helpers import get_or_create from .db_helpers import get_or_create
@@ -79,23 +79,6 @@ class MergeResult:
source_deleted: bool source_deleted: bool
@dataclass(frozen=True)
class MergePreview:
"""Non-mutating projection of what merge(source→target) would do (#8).
Shares the apply's predicates (rule 93) so the preview can't drift."""
source_id: int
source_name: str
target_id: int
target_name: str
compatible: bool # same kind + fandom — would apply succeed?
images_moving: int # source links that move (image lacks target)
images_already_on_target: int # source links dropped (image has target)
source_total: int # images carrying source
series_pages: int # series pages repointed
will_alias: bool # source name kept as a protective alias
sample_thumbnails: list[str] # a few of the moving images
class TagService: class TagService:
def __init__(self, session: AsyncSession): def __init__(self, session: AsyncSession):
self.session = session self.session = session
@@ -215,18 +198,6 @@ class TagService:
async def add_to_image(self, image_id: int, tag_id: int, source: str = "manual") -> None: async def add_to_image(self, image_id: int, tag_id: int, source: str = "manual") -> None:
"""Idempotent: re-adding an existing tag does nothing.""" """Idempotent: re-adding an existing tag does nothing."""
# A genuinely-new MANUAL add of a tag that already has a head is an
# UNDER-FIRE signal — the auto-system should have caught it (#114 obs).
is_new = source == "manual" and (
await self.session.execute(
select(image_tag.c.tag_id).where(
and_(
image_tag.c.image_record_id == image_id,
image_tag.c.tag_id == tag_id,
)
)
)
).first() is None
stmt = pg_insert(image_tag).values( stmt = pg_insert(image_tag).values(
image_record_id=image_id, tag_id=tag_id, source=source image_record_id=image_id, tag_id=tag_id, source=source
) )
@@ -234,22 +205,8 @@ class TagService:
index_elements=["image_record_id", "tag_id"] index_elements=["image_record_id", "tag_id"]
) )
await self.session.execute(stmt) await self.session.execute(stmt)
if is_new:
await self._note_under_fire(tag_id)
async def remove_from_image(self, image_id: int, tag_id: int) -> None: async def remove_from_image(self, image_id: int, tag_id: int) -> None:
# Removing an auto-applied (source='head_auto') tag is a MISFIRE — read
# the source BEFORE deleting, since it's lost with the row (#114 obs).
src = (
await self.session.execute(
select(image_tag.c.source).where(
and_(
image_tag.c.image_record_id == image_id,
image_tag.c.tag_id == tag_id,
)
)
)
).scalar_one_or_none()
await self.session.execute( await self.session.execute(
image_tag.delete().where( image_tag.delete().where(
and_( and_(
@@ -258,31 +215,6 @@ class TagService:
) )
) )
) )
if src == "head_auto":
await self._bump_metric(tag_id, "n_misfires")
async def _note_under_fire(self, tag_id: int) -> None:
"""Count an under-fire only when the tag actually has a head."""
has_head = (
await self.session.execute(
select(TagHead.tag_id).where(TagHead.tag_id == tag_id)
)
).first() is not None
if has_head:
await self._bump_metric(tag_id, "n_underfires")
async def _bump_metric(self, tag_id: int, column: str) -> None:
"""Increment a HeadMetric counter (upsert), keyed by tag so it survives
head retrain/prune."""
col = HeadMetric.__table__.c[column]
await self.session.execute(
pg_insert(HeadMetric)
.values(tag_id=tag_id, **{column: 1})
.on_conflict_do_update(
index_elements=["tag_id"],
set_={column: col + 1, "updated_at": func.now()},
)
)
async def list_for_image(self, image_id: int) -> Sequence: async def list_for_image(self, image_id: int) -> Sequence:
"""Tags on an image, ordered (kind, name). Each row carries the fandom's """Tags on an image, ordered (kind, name). Each row carries the fandom's
@@ -444,96 +376,6 @@ class TagService:
await self.session.flush() await self.session.flush()
return tag return tag
async def merge_preview(
self, source_id: int, target_id: int
) -> MergePreview:
"""Read-only dry-run of merge(source→target): the same counts the apply
would produce, plus a thumbnail sample of the images that move. Raises
TagValidationError only for missing/self-merge (so the UI can render a
kind/fandom-mismatch warning rather than a hard error)."""
if source_id == target_id:
raise TagValidationError("Cannot merge a tag into itself")
source = await self.session.get(Tag, source_id)
target = await self.session.get(Tag, target_id)
if source is None or target is None:
raise TagValidationError("Tag not found")
compatible = source.kind == target.kind and (
(source.fandom_id or 0) == (target.fandom_id or 0)
)
# Mirrors _repoint_image_tags: links whose image already has target are
# dropped (already_on_target); the rest move.
target_images = select(image_tag.c.image_record_id).where(
image_tag.c.tag_id == target_id
)
moving = await self.session.scalar(
select(func.count())
.select_from(image_tag)
.where(
image_tag.c.tag_id == source_id,
image_tag.c.image_record_id.notin_(target_images),
)
)
already = await self.session.scalar(
select(func.count())
.select_from(image_tag)
.where(
image_tag.c.tag_id == source_id,
image_tag.c.image_record_id.in_(target_images),
)
)
from ..models.series_page import SeriesPage
series_pages = await self.session.scalar(
select(func.count())
.select_from(SeriesPage)
.where(SeriesPage.series_tag_id == source_id)
)
will_alias = await self._keep_as_alias(source_id)
sample = await self._merge_sample_thumbnails(source_id, target_images)
return MergePreview(
source_id=source_id,
source_name=source.name,
target_id=target_id,
target_name=target.name,
compatible=compatible,
images_moving=moving or 0,
images_already_on_target=already or 0,
source_total=(moving or 0) + (already or 0),
series_pages=series_pages or 0,
will_alias=will_alias,
sample_thumbnails=sample,
)
async def _merge_sample_thumbnails(
self, source_id: int, target_images
) -> list[str]:
"""Up to 6 thumbnails of the images that WOULD move (carry source, not
target) — a sanity check that the merge targets the right content."""
from ..models import ImageRecord
from .gallery_service import thumbnail_url
rows = (
await self.session.execute(
select(
ImageRecord.thumbnail_path,
ImageRecord.sha256,
ImageRecord.mime,
)
.join(image_tag, image_tag.c.image_record_id == ImageRecord.id)
.where(
image_tag.c.tag_id == source_id,
image_tag.c.image_record_id.notin_(target_images),
)
.limit(6)
)
).all()
return [thumbnail_url(tp, sha, mime) for tp, sha, mime in rows]
async def merge(self, source_id: int, target_id: int) -> MergeResult: async def merge(self, source_id: int, target_id: int) -> MergeResult:
"""Transactionally repoint every FK from source→target, optionally """Transactionally repoint every FK from source→target, optionally
keep source's name as a tagger alias, delete source. Atomic: any keep source's name as a tagger alias, delete source. Atomic: any
-214
View File
@@ -13,15 +13,12 @@ from ..celery_app import celery
from ..models import ( from ..models import (
BackupRun, BackupRun,
DownloadEvent, DownloadEvent,
HeadAutoApplyRun,
HeadTrainingRun,
ImageRecord, ImageRecord,
ImportBatch, ImportBatch,
ImportSettings, ImportSettings,
ImportTask, ImportTask,
LibraryAuditRun, LibraryAuditRun,
Source, Source,
TagEvalRun,
TaskRun, TaskRun,
) )
from ..utils.phash import compute_phash from ..utils.phash import compute_phash
@@ -96,15 +93,6 @@ BACKUP_DB_STALL_THRESHOLD_MINUTES = 40
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m). # Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
# 2h15m gives a 10-min buffer. # 2h15m gives a 10-min buffer.
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135 LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
# tag-eval (#1130) has a 30-min soft limit; flag a run with no progress past 40.
TAG_EVAL_STALL_THRESHOLD_MINUTES = 40
TAG_EVAL_KEEP_RUNS = 20
# head training (#114) has a 60-min soft limit; flag no-progress past 75.
HEAD_TRAINING_STALL_THRESHOLD_MINUTES = 75
HEAD_TRAINING_KEEP_RUNS = 20
# head auto-apply (#114) shares the 60-min soft limit; flag past 75.
HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES = 75
HEAD_AUTO_APPLY_KEEP_RUNS = 20
# Import batches finalize only after every child ImportTask hits a # Import batches finalize only after every child ImportTask hits a
# terminal state. The recovery sweep targets the case where every # terminal state. The recovery sweep targets the case where every
# task is done but the batch never got its closing UPDATE # task is done but the batch never got its closing UPDATE
@@ -721,208 +709,6 @@ def recover_stalled_library_audit_runs() -> int:
return recovered return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_tag_eval_runs")
def recover_stalled_tag_eval_runs() -> int:
"""Flip TagEvalRun rows stuck in 'running' past the stall threshold to
'error', and prune old runs to the last TAG_EVAL_KEEP_RUNS (retention,
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=TAG_EVAL_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
update(TagEvalRun)
.where(TagEvalRun.status == "running")
.where(
func.coalesce(TagEvalRun.last_progress_at, TagEvalRun.started_at)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{TAG_EVAL_STALL_THRESHOLD_MINUTES} min)"
),
)
)
# Retention: keep only the most recent N runs.
keep = session.execute(
select(TagEvalRun.id).order_by(TagEvalRun.id.desc())
.limit(TAG_EVAL_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(TagEvalRun).where(TagEvalRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info("recover_stalled_tag_eval_runs: recovered %d rows", recovered)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_training_runs")
def recover_stalled_head_training_runs() -> int:
"""Flip HeadTrainingRun rows stuck in 'running' past the stall threshold to
'error', and prune old runs to the last HEAD_TRAINING_KEEP_RUNS (retention,
rule 89). Runs every 5 min on the maintenance lane; no-op when idle."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=HEAD_TRAINING_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
update(HeadTrainingRun)
.where(HeadTrainingRun.status == "running")
.where(
func.coalesce(
HeadTrainingRun.last_progress_at, HeadTrainingRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_TRAINING_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadTrainingRun.id).order_by(HeadTrainingRun.id.desc())
.limit(HEAD_TRAINING_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadTrainingRun).where(HeadTrainingRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_training_runs: recovered %d rows", recovered
)
return recovered
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs")
def recover_stalled_head_auto_apply_runs() -> int:
"""Flip stalled HeadAutoApplyRun 'running' rows to 'error' + prune to the
last HEAD_AUTO_APPLY_KEEP_RUNS (retention, rule 89). 5-min maintenance lane."""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES)
with SessionLocal() as session:
result = session.execute(
update(HeadAutoApplyRun)
.where(HeadAutoApplyRun.status == "running")
.where(
func.coalesce(
HeadAutoApplyRun.last_progress_at, HeadAutoApplyRun.started_at
)
< cutoff
)
.values(
status="error", finished_at=now,
error=(
f"stranded by recovery sweep (no progress for "
f"{HEAD_AUTO_APPLY_STALL_THRESHOLD_MINUTES} min)"
),
)
)
keep = session.execute(
select(HeadAutoApplyRun.id).order_by(HeadAutoApplyRun.id.desc())
.limit(HEAD_AUTO_APPLY_KEEP_RUNS)
).scalars().all()
if keep:
session.execute(
delete(HeadAutoApplyRun).where(HeadAutoApplyRun.id.not_in(keep))
)
session.commit()
recovered = result.rowcount or 0
if recovered:
log.info(
"recover_stalled_head_auto_apply_runs: recovered %d rows", recovered
)
return recovered
# Keep ~6 months of daily head-metric snapshots (enough to see tuning trends).
HEAD_METRICS_SNAPSHOT_RETENTION_DAYS = 180
@celery.task(name="backend.app.tasks.maintenance.snapshot_head_metrics")
def snapshot_head_metrics() -> int:
"""Daily per-concept observability point (#114): record each head-bearing
concept's auto-applied volume, cumulative misfires/under-fires, and the
head's measured quality — the time-series the operator tunes from. Prunes
points older than the retention window."""
from ..models import (
HeadMetric,
HeadMetricsSnapshot,
Tag,
TagHead,
)
from ..models.tag import image_tag
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
with SessionLocal() as session:
heads = {
r.tag_id: r for r in session.execute(
select(
TagHead.tag_id, TagHead.ap, TagHead.precision_cv,
TagHead.recall, TagHead.n_pos,
)
)
}
metrics = {
r.tag_id: r for r in session.execute(
select(
HeadMetric.tag_id, HeadMetric.n_misfires, HeadMetric.n_underfires
)
)
}
# .all() first: dict() of a bare Result tries the mapping protocol (a
# Result exposes .keys()) and subscripts it, which fails.
applied = dict(
session.execute(
select(image_tag.c.tag_id, func.count())
.where(image_tag.c.source == "head_auto")
.group_by(image_tag.c.tag_id)
).all()
)
tag_ids = set(heads) | set(metrics)
if not tag_ids:
return 0
names = dict(
session.execute(
select(Tag.id, Tag.name).where(Tag.id.in_(tag_ids))
).all()
)
for tid in tag_ids:
h = heads.get(tid)
m = metrics.get(tid)
session.add(HeadMetricsSnapshot(
tag_id=tid, name=names.get(tid, str(tid)),
snapshot_at=now,
n_auto_applied=applied.get(tid, 0),
n_misfires=m.n_misfires if m else 0,
n_underfires=m.n_underfires if m else 0,
ap=h.ap if h else None,
precision_cv=h.precision_cv if h else None,
recall=h.recall if h else None,
n_pos=h.n_pos if h else None,
))
session.execute(
delete(HeadMetricsSnapshot).where(
HeadMetricsSnapshot.snapshot_at
< now - timedelta(days=HEAD_METRICS_SNAPSHOT_RETENTION_DAYS)
)
)
session.commit()
return len(tag_ids)
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches") @celery.task(name="backend.app.tasks.maintenance.recover_stalled_import_batches")
def recover_stalled_import_batches() -> int: def recover_stalled_import_batches() -> int:
"""Finalize ImportBatch rows stuck in running past the hard limit """Finalize ImportBatch rows stuck in running past the hard limit
-397
View File
@@ -538,400 +538,3 @@ def recompute_centroids(self) -> int:
for tid in drifted: for tid in drifted:
recompute_centroid.delay(tid) recompute_centroid.delay(tid)
return len(drifted) return len(drifted)
@celery.task(
name="backend.app.tasks.ml.tag_eval_run",
bind=True,
# The head-vs-centroid eval (#1130) loads embeddings + fits sklearn heads
# for several concepts — minutes, not seconds. Runs on the ml queue because
# only that worker has numpy/scikit-learn.
soft_time_limit=1800, time_limit=2100,
)
def tag_eval_run(self, run_id: int) -> str:
"""Compute the eval report into the persisted TagEvalRun row so it survives
navigation (the admin card rehydrates from the row, not transient state)."""
from datetime import UTC, datetime
from ..models import TagEvalRun
from ..services.ml.tag_eval import run_eval
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = session.get(TagEvalRun, run_id)
if run is None:
return "missing"
run.last_progress_at = datetime.now(UTC)
session.commit()
try:
report = run_eval(session, run.params)
except SoftTimeLimitExceeded:
run.status = "error"
run.error = "timed out"
run.finished_at = datetime.now(UTC)
session.commit()
raise
except Exception as exc:
log.exception("tag_eval_run %d failed", run_id)
run.status = "error"
run.error = str(exc)
run.finished_at = datetime.now(UTC)
session.commit()
return "error"
run.report = report
run.status = "ready"
run.finished_at = datetime.now(UTC)
session.commit()
return "ready"
@celery.task(
name="backend.app.tasks.ml.train_heads",
bind=True,
# Trains a logistic-regression head per eligible concept over stored SigLIP
# embeddings — minutes for a full library. Runs on the ml queue (only that
# worker has scikit-learn). Commits per head so a kill leaves progress.
soft_time_limit=3600, time_limit=3900,
)
def train_heads(self, run_id: int) -> str:
"""(Re)train all eligible concept heads into tag_head, tracked by the
HeadTrainingRun row so the admin card shows live + historical status."""
from datetime import UTC, datetime
from ..models import HeadTrainingRun
from ..services.ml.heads import train_all_heads
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = session.get(HeadTrainingRun, run_id)
if run is None:
return "missing"
run.last_progress_at = datetime.now(UTC)
session.commit()
try:
result = train_all_heads(session, run.params, run)
except SoftTimeLimitExceeded:
run.status = "error"
run.error = "timed out"
run.finished_at = datetime.now(UTC)
session.commit()
raise
except Exception as exc:
log.exception("train_heads %d failed", run_id)
run.status = "error"
run.error = str(exc)
run.finished_at = datetime.now(UTC)
session.commit()
return "error"
run.n_trained = result["n_trained"]
run.n_skipped = result["n_skipped"]
run.status = "ready"
run.finished_at = datetime.now(UTC)
session.commit()
return "ready"
@celery.task(name="backend.app.tasks.ml.scheduled_train_heads")
def scheduled_train_heads() -> str:
"""Nightly passive retrain (#114): fold the day's accepts/rejects + any
newly-eligible concepts into the heads without the operator clicking. Skips
if a run is already in flight (one at a time). Creates + COMMITS the run row
before dispatching so the ml-queue worker can always find it."""
from datetime import UTC, datetime
from sqlalchemy import select as sa_select
from ..models import HeadTrainingRun
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
running = session.execute(
sa_select(HeadTrainingRun.id).where(HeadTrainingRun.status == "running")
).scalar_one_or_none()
if running is not None:
return "already running"
run = HeadTrainingRun(
params={"source": "scheduled"}, status="running",
last_progress_at=datetime.now(UTC),
)
session.add(run)
session.commit()
run_id = run.id
train_heads.delay(run_id)
return "dispatched"
@celery.task(
name="backend.app.tasks.ml.apply_head_tags",
bind=True,
# Scores the whole library against the graduated heads and applies their
# tags (or, dry_run, just counts). Streams embeddings in chunks; numpy only,
# but ml queue keeps it off the API workers. Commits per chunk.
soft_time_limit=3600, time_limit=3900,
)
def apply_head_tags(self, run_id: int) -> str:
"""Run an earned-auto-apply sweep into the persisted HeadAutoApplyRun row."""
from datetime import UTC, datetime
from ..models import HeadAutoApplyRun
from ..services.ml.heads import auto_apply_sweep
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
run = session.get(HeadAutoApplyRun, run_id)
if run is None:
return "missing"
run.last_progress_at = datetime.now(UTC)
session.commit()
try:
result = auto_apply_sweep(session, run, run.dry_run)
except SoftTimeLimitExceeded:
run.status = "error"
run.error = "timed out"
run.finished_at = datetime.now(UTC)
session.commit()
raise
except Exception as exc:
log.exception("apply_head_tags %d failed", run_id)
run.status = "error"
run.error = str(exc)
run.finished_at = datetime.now(UTC)
session.commit()
return "error"
run.n_applied = result["n_applied"]
run.report = {"concepts": result["concepts"]}
run.status = "ready"
run.finished_at = datetime.now(UTC)
session.commit()
return "ready"
@celery.task(name="backend.app.tasks.ml.scheduled_apply_head_tags")
def scheduled_apply_head_tags() -> str:
"""Daily passive auto-apply sweep (#114) — only when the master switch is on.
Skips if a sweep is already in flight. Creates + COMMITS the run before
dispatching so the worker always finds it."""
from datetime import UTC, datetime
from sqlalchemy import select as sa_select
from ..models import HeadAutoApplyRun, MLSettings
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
enabled = session.execute(
sa_select(MLSettings.head_auto_apply_enabled).where(MLSettings.id == 1)
).scalar_one_or_none()
if not enabled:
return "disabled"
running = session.execute(
sa_select(HeadAutoApplyRun.id).where(HeadAutoApplyRun.status == "running")
).scalar_one_or_none()
if running is not None:
return "already running"
run = HeadAutoApplyRun(
dry_run=False, params={"dry_run": False, "source": "scheduled"},
status="running", last_progress_at=datetime.now(UTC),
)
session.add(run)
session.commit()
run_id = run.id
apply_head_tags.delay(run_id)
return "dispatched"
@celery.task(name="backend.app.tasks.ml.enqueue_gpu_backfill")
def enqueue_gpu_backfill(task_name: str) -> int:
"""Enqueue a gpu_job for every image that still needs `task_name` (one
INSERT…SELECT, so it scales to a full library). The desktop agent drains the
queue over HTTP. Returns the number enqueued.
'siglip' gates on the RESULT (no concept region yet) rather than on a prior
job, so it picks up the back-catalogue of images that were CCIP-embedded
before concept crops existed, and retries images whose concept embed failed —
without re-touching their figure/CCIP regions."""
from sqlalchemy import exists, insert, literal
from sqlalchemy import select as sa_select
from ..models import GpuJob, ImageRecord, ImageRegion
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
if task_name == "siglip":
has_concept = exists().where(
ImageRegion.image_record_id == ImageRecord.id,
ImageRegion.kind == "concept",
)
queued = exists().where(
GpuJob.image_record_id == ImageRecord.id,
GpuJob.task == "siglip",
GpuJob.status.in_(["pending", "leased"]),
)
sel = sa_select(
ImageRecord.id, literal("siglip"), literal("pending")
).where(~has_concept).where(~queued)
else:
already = exists().where(
GpuJob.image_record_id == ImageRecord.id,
GpuJob.task == task_name,
GpuJob.status.in_(["pending", "leased", "done"]),
)
sel = sa_select(
ImageRecord.id, literal(task_name), literal("pending")
).where(~already)
# RETURNING + count: result.rowcount is unreliable for INSERT…SELECT.
rows = session.execute(
insert(GpuJob)
.from_select(["image_record_id", "task", "status"], sel)
.returning(GpuJob.id)
).fetchall()
session.commit()
return len(rows)
@celery.task(name="backend.app.tasks.ml.recover_orphaned_gpu_jobs")
def recover_orphaned_gpu_jobs() -> int:
"""Reset expired GPU-job leases back to pending — recovers work orphaned by an
agent that died mid-job (no graceful release). Short beat cadence so orphans
get picked back up quickly + the queue counts read honestly. Returns the
number recovered."""
from datetime import UTC, datetime
from sqlalchemy import update
from ..models import GpuJob
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
now = datetime.now(UTC)
res = session.execute(
update(GpuJob)
.where(GpuJob.status == "leased", GpuJob.lease_expires_at < now)
.values(
status="pending", lease_token=None, leased_at=None,
lease_expires_at=None, updated_at=now,
)
)
session.commit()
return res.rowcount or 0
@celery.task(
name="backend.app.tasks.ml.scheduled_ccip_auto_apply",
soft_time_limit=1800, time_limit=2100,
)
def scheduled_ccip_auto_apply() -> str:
"""Auto-tag confident CCIP character matches (source='ccip_auto') so identity
tags keep flowing without a button. No-op unless ccip_auto_apply_enabled.
References come only from single-character images (unambiguous); a tag is
applied where any figure's best cosine to a character's prototypes clears
ccip_auto_apply_threshold and it isn't already applied/rejected. Reversible."""
import numpy as np
from sqlalchemy import func
from sqlalchemy import select as sa_select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..models import ImageRegion, MLSettings, Tag, TagKind, TagSuggestionRejection
from ..models.tag import image_tag
fig = ("face", "figure")
def _l2(m):
n = np.linalg.norm(m, axis=1, keepdims=True)
n[n == 0] = 1.0
return m / n
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
s = session.get(MLSettings, 1)
if s is None or not s.ccip_auto_apply_enabled:
return "disabled"
thr = float(s.ccip_auto_apply_threshold)
single = (
sa_select(image_tag.c.image_record_id)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.group_by(image_tag.c.image_record_id)
.having(func.count() == 1)
)
ref_rows = session.execute(
sa_select(image_tag.c.tag_id, ImageRegion.ccip_embedding)
.select_from(ImageRegion)
.join(
image_tag,
image_tag.c.image_record_id == ImageRegion.image_record_id,
)
.join(Tag, Tag.id == image_tag.c.tag_id)
.where(Tag.kind == TagKind.character)
.where(ImageRegion.kind.in_(fig))
.where(ImageRegion.ccip_embedding.is_not(None))
.where(ImageRegion.image_record_id.in_(single))
).all()
if not ref_rows:
return "no-references"
by_char: dict[int, list] = {}
for tid, vec in ref_rows:
by_char.setdefault(tid, []).append(vec)
ref_tags = list(by_char)
mats = [_l2(np.asarray(by_char[t], dtype=np.float32)) for t in ref_tags]
allref = np.vstack(mats) # (total, 768)
seg = np.cumsum([0] + [len(m) for m in mats])[:-1] # per-char start
# Per character: images that already carry OR rejected the tag — skip.
skip = {t: set() for t in ref_tags}
for t in ref_tags:
for (iid,) in session.execute(
sa_select(image_tag.c.image_record_id).where(
image_tag.c.tag_id == t
)
):
skip[t].add(iid)
for (iid,) in session.execute(
sa_select(TagSuggestionRejection.image_record_id).where(
TagSuggestionRejection.tag_id == t
)
):
skip[t].add(iid)
img_ids = list(session.execute(
sa_select(ImageRegion.image_record_id)
.where(ImageRegion.kind.in_(fig), ImageRegion.ccip_embedding.is_not(None))
.distinct()
).scalars())
applied = 0
chunk_n = 500
for start in range(0, len(img_ids), chunk_n):
chunk = img_ids[start:start + chunk_n]
rows = session.execute(
sa_select(ImageRegion.image_record_id, ImageRegion.ccip_embedding)
.where(
ImageRegion.image_record_id.in_(chunk),
ImageRegion.kind.in_(fig),
ImageRegion.ccip_embedding.is_not(None),
)
).all()
by_img: dict[int, list] = {}
for iid, vec in rows:
by_img.setdefault(iid, []).append(vec)
for iid, vecs in by_img.items():
q = _l2(np.asarray(vecs, dtype=np.float32)) # (nq, 768)
colmax = (q @ allref.T).max(axis=0) # (total,)
charmax = np.maximum.reduceat(colmax, seg) # (n_chars,)
for ci in np.where(charmax >= thr)[0]:
t = ref_tags[int(ci)]
if iid in skip[t]:
continue
skip[t].add(iid)
session.execute(
pg_insert(image_tag)
.values(
image_record_id=iid, tag_id=t, source="ccip_auto",
)
.on_conflict_do_nothing()
)
applied += 1
session.commit()
return f"applied={applied}"
+3 -7
View File
@@ -72,14 +72,10 @@ import PipelineStatusChip from './PipelineStatusChip.vue'
const system = useSystemStore() const system = useSystemStore()
onMounted(() => system.refreshHealth()) onMounted(() => system.refreshHealth())
// Every route with a meta.title is a nav entry. Order by meta.navOrder — // Same mechanism the old sidebar used: every route with a meta.title is a
// router.getRoutes() does NOT guarantee declaration order, so explicit numbers // nav entry, in router declaration order. Auto-tracks future routes.
// pin the sequence (e.g. Explore after Gallery). Routes without one fall to the
// end. Auto-tracks future routes.
const navRoutes = computed(() => const navRoutes = computed(() =>
router.getRoutes() router.getRoutes().filter(r => r.meta?.title)
.filter(r => r.meta?.title)
.sort((a, b) => (a.meta.navOrder ?? 999) - (b.meta.navOrder ?? 999))
) )
// Content links for the centered desktop row — everything EXCEPT Settings, // Content links for the centered desktop row — everything EXCEPT Settings,
// which is config and gets pinned to the right edge instead. // which is config and gets pinned to the right edge instead.
@@ -1,68 +0,0 @@
<template>
<!-- Thin debounced tag-search autocomplete: emits `pick` with the chosen tag
and self-clears. Shared by the gallery's advanced tag-query builder; the
filter bar's own inline search predates this and also folds in artists. -->
<v-autocomplete
v-model="selected"
:items="items"
:loading="loading"
item-title="name" item-value="id"
no-filter hide-details density="compact" variant="outlined"
:placeholder="placeholder"
prepend-inner-icon="mdi-tag-plus-outline"
:aria-label="placeholder"
@update:search="onSearch"
@update:model-value="onPick"
>
<template #item="{ props: itemProps, item }">
<v-list-item v-bind="itemProps" :title="item.raw.name">
<template #subtitle>
{{ item.raw.fandom_name ? `character · ${item.raw.fandom_name}` : item.raw.kind }}
</template>
</v-list-item>
</template>
<template #no-data>
<v-list-item :title="searchedOnce ? 'No tags match' : 'Type to search tags'" />
</template>
</v-autocomplete>
</template>
<script setup>
import { ref, onBeforeUnmount } from 'vue'
import { useApi } from '../../composables/useApi.js'
defineProps({ placeholder: { type: String, default: 'Add tag…' } })
const emit = defineEmits(['pick'])
const api = useApi()
const selected = ref(null)
const items = ref([])
const loading = ref(false)
const searchedOnce = ref(false)
let debounce = null
function onSearch (q) {
if (debounce) clearTimeout(debounce)
if (!q || !q.trim()) { items.value = []; return }
debounce = setTimeout(async () => {
loading.value = true
try {
items.value = (await api.get('/api/tags/autocomplete', { params: { q, limit: 10 } })) || []
} catch {
items.value = []
} finally {
loading.value = false
searchedOnce.value = true
}
}, 250)
}
function onPick (id) {
const tag = items.value.find((i) => i.id === id)
selected.value = null
items.value = []
if (tag) emit('pick', tag)
}
onBeforeUnmount(() => { if (debounce) clearTimeout(debounce) })
</script>
@@ -27,34 +27,11 @@
</v-autocomplete> </v-autocomplete>
<div class="fc-filterbar__chips"> <div class="fc-filterbar__chips">
<!-- Include tag chips: click the body to flip to exclude, to remove
(#6 light editor one model, two editors). -->
<v-chip <v-chip
v-for="id in store.filter.tag_ids" :key="`t${id}`" v-for="id in store.filter.tag_ids" :key="`t${id}`"
size="small" closable :color="chipColor(id)" variant="tonal" size="small" closable :color="chipColor(id)" variant="tonal"
class="fc-filterbar__tagchip"
title="Click to exclude this tag instead"
@click="toggleTagPolarity(id)"
@click:close="removeTag(id)" @click:close="removeTag(id)"
>{{ store.tagLabels[id] || `#${id}` }}</v-chip> >{{ store.tagLabels[id] || `#${id}` }}</v-chip>
<!-- Exclude tag chips (red, minus). Click body to flip back to include. -->
<v-chip
v-for="id in store.filter.tag_exclude" :key="`x${id}`"
size="small" closable color="error" variant="tonal"
class="fc-filterbar__tagchip"
title="Excluded — click to include instead"
@click="toggleTagPolarity(id)"
@click:close="removeExclude(id)"
><v-icon start size="x-small">mdi-minus</v-icon>{{ store.tagLabels[id] || `#${id}` }}</v-chip>
<!-- Advanced OR-groups can't render as flat chips; one affordance opens
the builder where they live. -->
<v-chip
v-if="orGroupCount"
size="small" color="accent" variant="tonal"
prepend-icon="mdi-filter-cog-outline"
title="Open the advanced tag filter"
@click="advancedOpen = true"
>{{ orGroupCount }} OR-group{{ orGroupCount > 1 ? 's' : '' }}</v-chip>
<v-chip <v-chip
v-if="store.filter.artist_id" v-if="store.filter.artist_id"
size="small" closable color="accent" variant="tonal" size="small" closable color="accent" variant="tonal"
@@ -91,13 +68,6 @@
@update:model-value="setSort" @update:model-value="setSort"
/> />
<v-btn
:color="advancedActive ? 'accent' : undefined"
:variant="advancedActive ? 'tonal' : 'text'"
size="small" prepend-icon="mdi-filter-cog-outline"
@click="advancedOpen = true"
>Advanced{{ advancedCount ? ` (${advancedCount})` : '' }}</v-btn>
<v-btn <v-btn
:color="refineOpen ? 'accent' : undefined" :color="refineOpen ? 'accent' : undefined"
:variant="refineOpen || hasRefineFilters ? 'tonal' : 'text'" :variant="refineOpen || hasRefineFilters ? 'tonal' : 'text'"
@@ -113,18 +83,6 @@
</div> </div>
<GalleryFacetPanel v-if="refineOpen" /> <GalleryFacetPanel v-if="refineOpen" />
<v-dialog v-model="advancedOpen" max-width="640" scrollable>
<TagQueryBuilder
v-if="advancedOpen"
:include="store.filter.tag_ids"
:or-groups="store.filter.tag_or"
:exclude="store.filter.tag_exclude"
:labels="store.tagLabels"
@apply="onAdvancedApply"
@close="advancedOpen = false"
/>
</v-dialog>
</div> </div>
</template> </template>
@@ -135,7 +93,6 @@ import { useApi } from '../../composables/useApi.js'
import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js' import { cloneFilter, filterToQuery, useGalleryStore } from '../../stores/gallery.js'
import { useTagStore } from '../../stores/tags.js' import { useTagStore } from '../../stores/tags.js'
import GalleryFacetPanel from './GalleryFacetPanel.vue' import GalleryFacetPanel from './GalleryFacetPanel.vue'
import TagQueryBuilder from './TagQueryBuilder.vue'
const store = useGalleryStore() const store = useGalleryStore()
const tagStore = useTagStore() const tagStore = useTagStore()
@@ -160,19 +117,8 @@ const refineCount = computed(() => {
}) })
const hasRefineFilters = computed(() => refineCount.value > 0) const hasRefineFilters = computed(() => refineCount.value > 0)
// The structured tag filter beyond plain includes (OR-groups + excludes) —
// drives the Advanced button's active state + its count badge.
const advancedOpen = ref(false)
const orGroupCount = computed(() => store.filter.tag_or.length)
const advancedCount = computed(
() => store.filter.tag_or.length + store.filter.tag_exclude.length,
)
const advancedActive = computed(() => advancedOpen.value || advancedCount.value > 0)
const hasActiveFilters = computed(() => const hasActiveFilters = computed(() =>
store.filter.tag_ids.length > 0 || store.filter.tag_ids.length > 0 ||
store.filter.tag_exclude.length > 0 ||
store.filter.tag_or.length > 0 ||
store.filter.artist_id != null || store.filter.artist_id != null ||
store.filter.media_type != null || store.filter.media_type != null ||
store.filter.sort !== 'newest' || store.filter.sort !== 'newest' ||
@@ -249,34 +195,6 @@ function onPick(value) {
function removeTag(id) { function removeTag(id) {
pushFilter((n) => { n.tag_ids = n.tag_ids.filter((t) => t !== id) }) pushFilter((n) => { n.tag_ids = n.tag_ids.filter((t) => t !== id) })
} }
function removeExclude(id) {
pushFilter((n) => { n.tag_exclude = n.tag_exclude.filter((t) => t !== id) })
}
// Flip a tag between include and exclude in place (light-editor toggle). A tag
// is only ever in one of the two lists.
function toggleTagPolarity(id) {
pushFilter((n) => {
if (n.tag_ids.includes(id)) {
n.tag_ids = n.tag_ids.filter((t) => t !== id)
if (!n.tag_exclude.includes(id)) n.tag_exclude.push(id)
} else {
n.tag_exclude = n.tag_exclude.filter((t) => t !== id)
if (!n.tag_ids.includes(id)) n.tag_ids.push(id)
}
})
}
// The advanced builder hands back the whole tag model at once.
function onAdvancedApply({ tag_ids, tag_or, tag_exclude, labels }) {
for (const [id, name] of Object.entries(labels || {})) {
store.noteTagLabel(Number(id), name)
}
pushFilter((n) => {
n.tag_ids = tag_ids
n.tag_or = tag_or
n.tag_exclude = tag_exclude
})
advancedOpen.value = false
}
function clearArtist() { function clearArtist() {
store.noteArtistLabel(null) store.noteArtistLabel(null)
pushFilter((n) => { n.artist_id = null }) pushFilter((n) => { n.artist_id = null })
@@ -339,12 +257,6 @@ function pushFilter(mutate) {
} }
.fc-filterbar__search { max-width: 320px; min-width: 200px; } .fc-filterbar__search { max-width: 320px; min-width: 200px; }
.fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; } .fc-filterbar__chips { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
/* The tag chips' bodies toggle include/exclude — signal they're clickable. */
.fc-filterbar__tagchip { cursor: pointer; }
.fc-filterbar__tagchip:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 1px;
}
.fc-filterbar__sort { max-width: 150px; } .fc-filterbar__sort { max-width: 150px; }
/* Phones: the search's 200px min-width jams the wrapping bar. Give search its /* Phones: the search's 200px min-width jams the wrapping bar. Give search its
@@ -1,236 +0,0 @@
<template>
<v-card class="fc-tqb">
<v-card-title class="fc-tqb__head">
<v-icon icon="mdi-filter-cog-outline" size="small" class="mr-2" />
Advanced tag filter
<v-spacer />
<v-btn
icon="mdi-close" variant="text" size="small"
aria-label="Close advanced tag filter" @click="$emit('close')"
/>
</v-card-title>
<v-card-text>
<p class="text-body-2 fc-tqb__hint">
An image must match <strong>every</strong> group below, and a group
matches when the image carries <strong>any</strong> tag in it
(AND of ORs). Excluded tags are never shown.
</p>
<!-- AND-of-OR groups -->
<div class="fc-tqb__section">
<div class="fc-tqb__section-title">Must match all of</div>
<p v-if="!groups.length" class="fc-tqb__empty">
No groups yet add a tag below to start.
</p>
<template v-for="(g, gi) in groups" :key="gi">
<div v-if="gi > 0" class="fc-tqb__and">AND</div>
<div class="fc-tqb__group">
<div class="fc-tqb__group-chips">
<template v-for="(id, ci) in g" :key="id">
<span v-if="ci > 0" class="fc-tqb__or">or</span>
<v-chip
size="small" closable variant="tonal"
:color="kindColor(id)"
@click:close="removeFromGroup(gi, id)"
>{{ label(id) }}</v-chip>
</template>
<span v-if="!g.length" class="fc-tqb__empty">empty</span>
</div>
<div class="fc-tqb__group-actions">
<TagPicker
placeholder="or…" class="fc-tqb__picker"
@pick="(t) => addToGroup(gi, t)"
/>
<v-btn
icon="mdi-delete-outline" variant="text" size="small"
aria-label="Remove this group" @click="removeGroup(gi)"
/>
</div>
</div>
</template>
<div class="fc-tqb__add">
<TagPicker
placeholder="Add a tag (new group)…" class="fc-tqb__picker"
@pick="addNewGroup"
/>
</div>
</div>
<v-divider class="my-4" />
<!-- exclude -->
<div class="fc-tqb__section">
<div class="fc-tqb__section-title">Exclude (must NOT have)</div>
<div class="fc-tqb__group-chips">
<v-chip
v-for="id in excludeIds" :key="id"
size="small" closable color="error" variant="tonal"
@click:close="removeExclude(id)"
>
<v-icon start size="x-small">mdi-minus</v-icon>{{ label(id) }}
</v-chip>
<span v-if="!excludeIds.length" class="fc-tqb__empty">none</span>
</div>
<div class="fc-tqb__add">
<TagPicker
placeholder="Exclude a tag…" class="fc-tqb__picker"
@pick="addExclude"
/>
</div>
</div>
</v-card-text>
<v-card-actions class="fc-tqb__actions">
<v-btn
variant="text" size="small" :disabled="!isDirty && isEmpty"
@click="clearAll"
>Clear</v-btn>
<v-spacer />
<v-btn variant="text" @click="$emit('close')">Cancel</v-btn>
<v-btn color="accent" variant="flat" rounded="pill" @click="apply">
Apply filter
</v-btn>
</v-card-actions>
</v-card>
</template>
<script setup>
import { ref, computed } from 'vue'
import { useTagStore } from '../../stores/tags.js'
import TagPicker from '../common/TagPicker.vue'
// One editor for the whole structured tag model. It works purely on "AND-of-OR
// groups + exclude": singleton includes (tag_ids) and OR-groups (tag_or) are
// unified into one `groups` list here, then split back apart on Apply so the
// URL stays compact (singletons serialize to tag_id, multi-tag groups to
// tag_or). Writes the SAME model the light chips edit — two editors, one model.
const props = defineProps({
include: { type: Array, default: () => [] }, // tag_ids
orGroups: { type: Array, default: () => [] }, // tag_or
exclude: { type: Array, default: () => [] }, // tag_exclude
labels: { type: Object, default: () => ({}) }, // id -> name (best effort)
})
const emit = defineEmits(['apply', 'close'])
const tagStore = useTagStore()
// Seed local draft from props (the dialog is recreated on each open, so a
// setup-time seed is the current filter every time).
const groups = ref([
...props.include.map((id) => [id]),
...props.orGroups.map((g) => [...g]),
])
const excludeIds = ref([...props.exclude])
// Names/kinds learned as tags are picked, layered over the labels passed in.
const nameMap = ref({ ...props.labels })
const kindMap = ref({})
function label (id) { return nameMap.value[id] || `#${id}` }
function kindColor (id) { return tagStore.colorFor(kindMap.value[id] || 'general') }
function _note (tag) {
nameMap.value = { ...nameMap.value, [tag.id]: tag.name }
kindMap.value = { ...kindMap.value, [tag.id]: tag.kind }
}
function _dropFromGroups (id) {
groups.value = groups.value
.map((g) => g.filter((t) => t !== id))
.filter((g) => g.length)
}
function addToGroup (gi, tag) {
_note(tag)
excludeIds.value = excludeIds.value.filter((t) => t !== tag.id) // can't be both
if (!groups.value[gi].includes(tag.id)) groups.value[gi].push(tag.id)
}
function addNewGroup (tag) {
_note(tag)
excludeIds.value = excludeIds.value.filter((t) => t !== tag.id)
groups.value.push([tag.id])
}
function removeFromGroup (gi, id) {
groups.value[gi] = groups.value[gi].filter((t) => t !== id)
if (!groups.value[gi].length) groups.value.splice(gi, 1)
}
function removeGroup (gi) { groups.value.splice(gi, 1) }
function addExclude (tag) {
_note(tag)
_dropFromGroups(tag.id) // a tag can't be both required and excluded
if (!excludeIds.value.includes(tag.id)) excludeIds.value.push(tag.id)
}
function removeExclude (id) {
excludeIds.value = excludeIds.value.filter((t) => t !== id)
}
function clearAll () { groups.value = []; excludeIds.value = [] }
const isEmpty = computed(() => !groups.value.length && !excludeIds.value.length)
const isDirty = computed(() =>
JSON.stringify(groups.value) !== JSON.stringify(props.orGroups.length || props.include.length
? [...props.include.map((id) => [id]), ...props.orGroups]
: []) ||
JSON.stringify(excludeIds.value) !== JSON.stringify(props.exclude),
)
function apply () {
const clean = groups.value.filter((g) => g.length)
emit('apply', {
tag_ids: clean.filter((g) => g.length === 1).map((g) => g[0]),
tag_or: clean.filter((g) => g.length > 1),
tag_exclude: [...excludeIds.value],
labels: nameMap.value, // so freshly-picked chips show their name on return
})
}
</script>
<style scoped>
.fc-tqb__head { display: flex; align-items: center; }
.fc-tqb__hint {
color: rgb(var(--v-theme-on-surface-variant));
margin-bottom: 16px;
}
.fc-tqb__section { margin-bottom: 4px; }
.fc-tqb__section-title {
font-size: 12px; font-weight: 600; letter-spacing: 0.04em;
text-transform: uppercase;
color: rgb(var(--v-theme-on-surface-variant));
margin-bottom: 8px;
}
.fc-tqb__and {
font-size: 11px; font-weight: 700; letter-spacing: 0.08em;
color: rgb(var(--v-theme-accent));
margin: 6px 0 6px 2px;
}
.fc-tqb__group {
display: flex; align-items: center; gap: 12px;
flex-wrap: wrap;
padding: 8px 10px;
border: 1px solid rgba(var(--v-theme-on-surface), 0.12);
border-radius: 10px;
background: rgba(var(--v-theme-on-surface), 0.03);
}
.fc-tqb__group-chips {
display: flex; align-items: center; gap: 6px; flex-wrap: wrap;
flex: 1 1 220px; min-width: 0;
}
.fc-tqb__or {
font-size: 11px; font-style: italic;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-tqb__group-actions {
display: flex; align-items: center; gap: 4px;
}
.fc-tqb__picker { min-width: 180px; max-width: 240px; }
.fc-tqb__add { margin-top: 10px; max-width: 280px; }
.fc-tqb__empty {
font-size: 12px; font-style: italic;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-tqb__actions { padding: 8px 16px 16px; }
</style>
@@ -1,112 +0,0 @@
<template>
<section v-if="img" class="fc-meta" aria-label="Image details">
<!-- #4a: dimensions / size / type sit as a compact top block alongside a
small save action (operator-asked 2026-06-26) meta on the left, the
download control shrunk to a floppy-disk icon on the right. -->
<dl class="fc-meta__grid">
<div v-if="img.width && img.height" class="fc-meta__item">
<dt>Dimensions</dt><dd>{{ img.width }} × {{ img.height }}</dd>
</div>
<div v-if="img.size_bytes" class="fc-meta__item">
<dt>Size</dt><dd>{{ humanSize(img.size_bytes) }}</dd>
</div>
<div v-if="img.mime" class="fc-meta__item">
<dt>Type</dt><dd>{{ shortType(img.mime) }}</dd>
</div>
</dl>
<!-- #4b: floppy-disk save icon = download; the kebab menu keeps Copy link.
Image-to-clipboard is OFF the table on the plain-HTTP origin (rule 95),
so the secondary action copies the link TEXT, which works everywhere. -->
<div class="fc-meta__actions">
<v-btn
icon="mdi-content-save" size="small" variant="tonal" color="accent"
aria-label="Download image" title="Download" @click="download"
/>
<v-menu location="bottom end">
<template #activator="{ props }">
<v-btn
v-bind="props" icon="mdi-dots-vertical" size="small" variant="text"
aria-label="More download options"
/>
</template>
<v-list density="compact">
<v-list-item prepend-icon="mdi-link-variant" title="Copy link" @click="copyLink" />
<v-list-item prepend-icon="mdi-content-save" title="Download" @click="download" />
</v-list>
</v-menu>
</div>
</section>
</template>
<script setup>
import { computed } from 'vue'
import { useModalStore } from '../../stores/modal.js'
import { copyText } from '../../utils/clipboard.js'
import { toast } from '../../utils/toast.js'
// `image` lets a non-modal surface (the Explore workspace) render the same
// meta + download block for its anchor. Defaults to the modal store's current
// image so the image modal is unchanged.
const props = defineProps({ image: { type: Object, default: null } })
const modal = useModalStore()
const img = computed(() => props.image ?? modal.current)
function humanSize (bytes) {
const units = ['B', 'KB', 'MB', 'GB']
let n = bytes
let i = 0
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++ }
return `${n < 10 && i > 0 ? n.toFixed(1) : Math.round(n)} ${units[i]}`
}
function shortType (mime) { return mime.split('/')[1]?.toUpperCase() || mime }
function absoluteUrl () {
const origin = typeof window !== 'undefined' ? window.location.origin : ''
return origin + img.value.image_url
}
function download () {
// Same-origin /images/* link — the download attribute lets the browser save
// the original (filename derived from the path) instead of navigating to it.
const a = document.createElement('a')
a.href = img.value.image_url
a.setAttribute('download', '')
document.body.appendChild(a)
a.click()
a.remove()
}
async function copyLink () {
try {
await copyText(absoluteUrl())
toast({ text: 'Link copied to clipboard', type: 'success' })
} catch {
toast({ text: 'Could not copy the link', type: 'error' })
}
}
</script>
<style scoped>
.fc-meta {
padding: 14px 16px 0;
display: flex; align-items: flex-start; gap: 12px;
}
.fc-meta__grid {
flex: 1 1 auto; min-width: 0;
display: flex; flex-wrap: wrap; gap: 4px 18px; margin: 0;
}
.fc-meta__actions {
flex: 0 0 auto;
display: flex; align-items: center; gap: 2px;
}
.fc-meta__item { display: flex; flex-direction: column; }
.fc-meta__item dt {
font-size: 10px; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-meta__item dd {
margin: 0; font-size: 13px; font-variant-numeric: tabular-nums;
color: rgb(var(--v-theme-on-surface));
}
</style>
+3 -29
View File
@@ -73,20 +73,11 @@
</div> </div>
<aside v-if="modal.current" class="fc-viewer__side"> <aside v-if="modal.current" class="fc-viewer__side">
<!-- Provenance, then the meta + save block just beneath it, then tags
+ suggestions all scroll together here (operator-asked
2026-06-26: meta/download sits directly above Tags, under
Provenance). -->
<div class="fc-viewer__side-main">
<ProvenancePanel /> <ProvenancePanel />
<ImageMetaBar />
<TagPanel /> <TagPanel />
</div> <!-- Non-blocking: fetches its own similar set after the modal is up;
<!-- while Related is PINNED to the bottom of the rail so it stays collapses silently if empty/slow/failed (see RelatedStrip). -->
reachable no matter how long Tags/Suggestions run (operator-asked <RelatedStrip />
2026-06-26). Non-blocking: fetches its own similar set; collapses
silently (and takes no footer space) if empty/slow/failed. -->
<RelatedStrip class="fc-viewer__related" />
</aside> </aside>
</div> </div>
</div> </div>
@@ -99,7 +90,6 @@ import { useModalStore } from '../../stores/modal.js'
import { arrowNavAllowed, isTextEntry } from '../../utils/textEntry.js' import { arrowNavAllowed, isTextEntry } from '../../utils/textEntry.js'
import ImageCanvas from './ImageCanvas.vue' import ImageCanvas from './ImageCanvas.vue'
import VideoCanvas from './VideoCanvas.vue' import VideoCanvas from './VideoCanvas.vue'
import ImageMetaBar from './ImageMetaBar.vue'
import TagPanel from './TagPanel.vue' import TagPanel from './TagPanel.vue'
import ProvenancePanel from './ProvenancePanel.vue' import ProvenancePanel from './ProvenancePanel.vue'
import RelatedStrip from './RelatedStrip.vue' import RelatedStrip from './RelatedStrip.vue'
@@ -317,18 +307,6 @@ function nextFrame() {
width: var(--fc-side-w); flex-shrink: 0; width: var(--fc-side-w); flex-shrink: 0;
background: rgb(var(--v-theme-surface)); background: rgb(var(--v-theme-surface));
border-left: 1px solid rgb(var(--v-theme-surface-light)); border-left: 1px solid rgb(var(--v-theme-surface-light));
/* Flex column: a scrolling main area + a pinned Related footer. */
display: flex; flex-direction: column; min-height: 0;
}
.fc-viewer__side-main {
flex: 1 1 auto; min-height: 0; overflow-y: auto;
}
.fc-viewer__related {
/* Pinned to the bottom; capped so a tall strip can't swallow the rail —
it scrolls internally past the cap. RelatedStrip's own border-top draws
the divider; it self-collapses (no space) when there's nothing to show. */
flex: 0 0 auto;
max-height: 45%;
overflow-y: auto; overflow-y: auto;
} }
@@ -359,10 +337,6 @@ function nextFrame() {
border-left: none; border-left: none;
border-top: 1px solid rgb(var(--v-theme-surface-light)); border-top: 1px solid rgb(var(--v-theme-surface-light));
} }
/* The whole body scrolls on mobile — don't nest scrolls or pin Related; let
the rail flow naturally (Related lands at the end). */
.fc-viewer__side-main { flex: none; overflow: visible; }
.fc-viewer__related { max-height: none; overflow: visible; }
/* Re-center the prev/next arrows over the 55vh image band (their base /* Re-center the prev/next arrows over the 55vh image band (their base
top:50% would land on the scrolling panel); next uses the full width top:50% would land on the scrolling panel); next uses the full width
now that the panel is below, not beside. Close + integrity badge keep now that the panel is below, not beside. Close + integrity badge keep
@@ -70,13 +70,7 @@
</div> </div>
<div v-if="attachments.length" class="fc-prov__attach"> <div v-if="attachments.length" class="fc-prov__attach">
<h4 class="fc-prov__attach-title"> <h4 class="fc-prov__attach-title">Attachments</h4>
{{ attachments.length === 1 ? 'Attachment' : `Attachments (${attachments.length})` }}
</h4>
<!-- Scroll-capped: a single post can carry dozens of archives (HR
bundle posts), which previously ballooned the panel past the
viewport. Mirror the cards' independent-scroll treatment. -->
<div class="fc-prov__attach-list">
<a <a
v-for="at in attachments" :key="at.id" v-for="at in attachments" :key="at.id"
class="fc-prov__attach-row" class="fc-prov__attach-row"
@@ -85,7 +79,6 @@
<span class="fc-prov__attach-size">({{ at.size_bytes }} B)</span> <span class="fc-prov__attach-size">({{ at.size_bytes }} B)</span>
</a> </a>
</div> </div>
</div>
</section> </section>
</template> </template>
@@ -97,19 +90,9 @@ import { useProvenanceStore } from '../../stores/provenance.js'
import { formatPostDate } from '../../utils/date.js' import { formatPostDate } from '../../utils/date.js'
import { toPlainText } from '../../utils/htmlSanitize.js' import { toPlainText } from '../../utils/htmlSanitize.js'
// `imageId`/`image` let a non-modal surface (the Explore workspace) render
// provenance for its anchor. Default to the modal store's current image so the
// image modal is unchanged. Provenance is its own system (loaded by id via the
// provenance store), so it only needs the right id + the artist fallback.
const props = defineProps({
imageId: { type: Number, default: null },
image: { type: Object, default: null },
})
const modal = useModalStore() const modal = useModalStore()
const prov = useProvenanceStore() const prov = useProvenanceStore()
const router = useRouter() const router = useRouter()
const effectiveId = computed(() => props.imageId ?? modal.currentImageId)
const effectiveImage = computed(() => props.image ?? modal.current)
// Per-post description collapse state (keyed by provenance_id). Default // Per-post description collapse state (keyed by provenance_id). Default
// collapsed so multiple posts don't each eat ~180px of the panel the // collapsed so multiple posts don't each eat ~180px of the panel the
@@ -117,21 +100,21 @@ const effectiveImage = computed(() => props.image ?? modal.current)
// 2026-05-28. Reset when the viewed image changes. // 2026-05-28. Reset when the viewed image changes.
const expanded = reactive({}) const expanded = reactive({})
function toggleDesc(id) { expanded[id] = !expanded[id] } function toggleDesc(id) { expanded[id] = !expanded[id] }
watch(() => effectiveId.value, () => { watch(() => modal.currentImageId, () => {
for (const k of Object.keys(expanded)) delete expanded[k] for (const k of Object.keys(expanded)) delete expanded[k]
}) })
watch( watch(
() => effectiveId.value, () => modal.currentImageId,
(id) => { if (id != null) prov.loadForImage(id) }, (id) => { if (id != null) prov.loadForImage(id) },
{ immediate: true } { immediate: true }
) )
const state = computed(() => const state = computed(() =>
effectiveId.value == null ? null : prov.imageProv(effectiveId.value) modal.currentImageId == null ? null : prov.imageProv(modal.currentImageId)
) )
const fallbackArtist = computed(() => effectiveImage.value?.artist || null) const fallbackArtist = computed(() => modal.current?.artist || null)
const showArtistFallback = computed(() => { const showArtistFallback = computed(() => {
const st = state.value const st = state.value
@@ -246,15 +229,6 @@ function openPost(postId, artistId) {
font-size: 13px; color: rgb(var(--v-theme-on-surface-variant)); font-size: 13px; color: rgb(var(--v-theme-on-surface-variant));
margin: 8px 0 4px; margin: 8px 0 4px;
} }
.fc-prov__attach-list {
/* ~7 rows visible before scrolling; the clipped row hints at more.
Hairline scrollbar matching .fc-prov__cards. */
max-height: 180px;
overflow-y: auto;
scrollbar-width: thin;
scrollbar-color: rgb(var(--v-theme-surface-light)) transparent;
padding-right: 4px;
}
.fc-prov__attach-row { .fc-prov__attach-row {
display: block; font-size: 13px; text-decoration: none; display: block; font-size: 13px; text-decoration: none;
color: rgb(var(--v-theme-accent)); padding: 2px 0; color: rgb(var(--v-theme-accent)); padding: 2px 0;
@@ -5,19 +5,12 @@
<div v-if="show" class="fc-related"> <div v-if="show" class="fc-related">
<div class="fc-related__head"> <div class="fc-related__head">
<span class="fc-related__title">Related</span> <span class="fc-related__title">Related</span>
<div class="fc-related__actions">
<v-btn
size="x-small" variant="text" color="accent"
prepend-icon="mdi-compass-outline"
@click="explore"
>Explore</v-btn>
<v-btn <v-btn
size="x-small" variant="text" color="accent" size="x-small" variant="text" color="accent"
:disabled="loading || !results.length" :disabled="loading || !results.length"
@click="seeAll" @click="seeAll"
>See all similar</v-btn> >See all similar</v-btn>
</div> </div>
</div>
<div class="fc-related__row"> <div class="fc-related__row">
<template v-if="loading"> <template v-if="loading">
<div v-for="n in 6" :key="n" class="fc-related__skel" /> <div v-for="n in 6" :key="n" class="fc-related__skel" />
@@ -101,14 +94,6 @@ function seeAll() {
modal.close() modal.close()
router.push({ name: 'gallery', query: { similar_to: String(id) } }) router.push({ name: 'gallery', query: { similar_to: String(id) } })
} }
// #94: leave the modal and open the dedicated Explore walk anchored here.
function explore() {
const id = modal.current?.id
if (!id) return
modal.close()
router.push({ name: 'explore', params: { imageId: String(id) } })
}
</script> </script>
<style scoped> <style scoped>
@@ -120,7 +105,6 @@ function explore() {
display: flex; align-items: center; justify-content: space-between; display: flex; align-items: center; justify-content: space-between;
margin-bottom: 8px; margin-bottom: 8px;
} }
.fc-related__actions { display: flex; align-items: center; gap: 2px; }
.fc-related__title { .fc-related__title {
font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em; font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.06em;
color: rgb(var(--v-theme-on-surface-variant)); color: rgb(var(--v-theme-on-surface-variant));
@@ -1,54 +1,29 @@
<template> <template>
<!-- Chip-card row: visible border + hover/focus state unifies the <!-- Chip-card row: visible border + hover/focus state unifies the
name, score, and action buttons as one "object" (operator-asked name, score, and action buttons as one "object" (operator-asked
2026-06-01). The row itself is informational; the green / red 2026-06-01). The row itself is informational; the explicit
verdict pair + 3-dot alias menu are the action affordances. --> Accept button + 3-dot menu are the action affordances. -->
<div class="fc-suggestion" :class="{ 'fc-suggestion--rejected': suggestion.rejected }"> <div class="fc-suggestion">
<span class="fc-suggestion__name"> <span class="fc-suggestion__name">
{{ suggestion.display_name }} {{ suggestion.display_name }}
<span v-if="suggestion.rejected" class="fc-suggestion__rejected-tag" <span v-if="suggestion.creates_new_tag" class="fc-suggestion__new"
title="You rejected this for this image — un-reject to recover">rejected</span>
<span v-else-if="suggestion.creates_new_tag" class="fc-suggestion__new"
title="No matching tag yet — accepting creates it">+ new</span> title="No matching tag yet — accepting creates it">+ new</span>
<span v-else-if="suggestion.via_alias" class="fc-suggestion__alias" <span v-else-if="suggestion.via_alias" class="fc-suggestion__alias"
:title="`Mapped from the tagger's “${suggestion.raw_name}” via an alias`">alias</span> :title="`Mapped from the tagger's “${suggestion.raw_name}” via an alias`">alias</span>
</span> </span>
<span class="fc-suggestion__score">{{ scorePct }}</span> <span class="fc-suggestion__score">{{ scorePct }}</span>
<!-- Green / red pair (operator-asked 2026-06-28) mirrors the eval <v-btn
card's verdict buttons: ✓ accepts the tag (positive), ✗ dismisses it class="fc-suggestion__accept"
for this image (records a TagSuggestionRejection — a hard negative the size="small" variant="tonal" color="accent"
heads train on). Together they occupy ~the footprint of the old single density="compact" rounded="pill"
Accept pill, so rejecting is now a one-click peer of accepting rather
than buried in the kebab. When the row is already rejected the ✗ swaps
to an undo (↶) so the rejection is reversible in place. -->
<div class="fc-suggestion__acts">
<button
class="fc-act fc-act--yes" type="button"
:aria-label="`Accept ${suggestion.display_name}`" :aria-label="`Accept ${suggestion.display_name}`"
:title="`Yes — tag ${suggestion.display_name}`"
@click="$emit('accept', suggestion)" @click="$emit('accept', suggestion)"
><v-icon size="16">mdi-check</v-icon></button> >
<button Accept
v-if="suggestion.rejected" </v-btn>
class="fc-act fc-act--undo" type="button"
:aria-label="`Un-reject ${suggestion.display_name}`"
:title="`Undo — restore ${suggestion.display_name} as a suggestion`"
@click="$emit('undismiss', suggestion)"
><v-icon size="16">mdi-undo-variant</v-icon></button>
<button
v-else
class="fc-act fc-act--no" type="button"
:aria-label="`Reject ${suggestion.display_name}`"
:title="`No — not ${suggestion.display_name}`"
@click="$emit('dismiss', suggestion)"
><v-icon size="16">mdi-close</v-icon></button>
</div>
<!-- Modal-safe kebab is baked into KebabMenu (this row lives in the <!-- Modal-safe kebab is baked into KebabMenu (this row lives in the
teleported image modal — #711). Only rendered when an alias action teleported image modal #711). -->
applies — dismiss now lives on the red ✗, so a centroid hit with no
alias option has no menu. -->
<KebabMenu <KebabMenu
v-if="hasMenu"
class="fc-suggestion__menu" size="small" variant="outlined" class="fc-suggestion__menu" size="small" variant="outlined"
:label="`More actions for ${suggestion.display_name}`" :label="`More actions for ${suggestion.display_name}`"
> >
@@ -67,6 +42,9 @@
> >
<v-list-item-title>Remove alias</v-list-item-title> <v-list-item-title>Remove alias</v-list-item-title>
</v-list-item> </v-list-item>
<v-list-item @click="$emit('dismiss', suggestion)">
<v-list-item-title>Dismiss for this image</v-list-item-title>
</v-list-item>
</KebabMenu> </KebabMenu>
</div> </div>
</template> </template>
@@ -76,15 +54,9 @@ import { computed } from 'vue'
import KebabMenu from '../common/KebabMenu.vue' import KebabMenu from '../common/KebabMenu.vue'
const props = defineProps({ suggestion: { type: Object, required: true } }) const props = defineProps({ suggestion: { type: Object, required: true } })
defineEmits(['accept', 'alias', 'remove-alias', 'dismiss', 'undismiss']) defineEmits(['accept', 'alias', 'remove-alias', 'dismiss'])
const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`) const scorePct = computed(() => `${Math.round(props.suggestion.score * 100)}%`)
// Kebab now only carries alias actions: show it when this suggestion can be
// aliased (raw model key, not yet aliased) or is already aliased (so it can be
// un-aliased). Centroid hits (no raw_name, no alias) have an empty menu → hide.
const hasMenu = computed(() =>
Boolean(props.suggestion.raw_name) || Boolean(props.suggestion.via_alias)
)
</script> </script>
<style scoped> <style scoped>
@@ -132,51 +104,12 @@ const hasMenu = computed(() =>
color: rgb(var(--v-theme-on-surface-variant, var(--v-theme-on-surface))); color: rgb(var(--v-theme-on-surface-variant, var(--v-theme-on-surface)));
font-family: 'JetBrains Mono', monospace; font-family: 'JetBrains Mono', monospace;
} }
/* Green ✓ / red ✗ verdict pair — same circular language as the eval card /* Vuetify's compact density doesn't shrink the tonal button enough
(TagEvalCard .fc-act) so accept/reject read identically across surfaces. */ for a tight row; clamp the min-width so Accept stays compact. */
.fc-suggestion__acts { .fc-suggestion__accept :deep(.v-btn__content) {
flex: 0 0 auto; display: flex; gap: 4px; font-size: 12px; letter-spacing: 0.02em;
}
.fc-act {
width: 26px; height: 26px; border-radius: 50%; border: none; cursor: pointer;
display: flex; align-items: center; justify-content: center; color: #fff;
opacity: 0.9; transition: transform 0.1s, opacity 0.1s;
}
.fc-act:hover { opacity: 1; transform: scale(1.1); }
.fc-act:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent)); outline-offset: 1px;
}
.fc-act--yes { background: rgb(var(--v-theme-success)); }
.fc-act--no { background: rgb(var(--v-theme-error)); }
/* Undo reads as neutral-secondary, not a verdict: outlined, not filled. */
.fc-act--undo {
background: transparent; color: rgb(var(--v-theme-on-surface-variant));
border: 1px solid rgb(var(--v-theme-on-surface-variant), 0.5);
} }
.fc-suggestion__menu { .fc-suggestion__menu {
flex: 0 0 auto; flex: 0 0 auto;
} }
/* Rejected state: the row stays put (recovery), dimmed + red-edged so it
reads as "handled, negative" without shouting over live suggestions. */
.fc-suggestion--rejected {
border-color: rgb(var(--v-theme-error), 0.4);
background: rgb(var(--v-theme-error), 0.06);
}
.fc-suggestion--rejected .fc-suggestion__name {
color: rgb(var(--v-theme-on-surface-variant));
text-decoration: line-through;
text-decoration-color: rgb(var(--v-theme-error), 0.6);
}
.fc-suggestion__rejected-tag {
display: inline-block;
font-size: 10px; font-weight: 600;
color: rgb(var(--v-theme-error));
background: rgb(var(--v-theme-error), 0.12);
border: 1px solid rgb(var(--v-theme-error), 0.4);
padding: 1px 6px; border-radius: 999px;
margin-left: 6px;
text-transform: uppercase; letter-spacing: 0.04em;
text-decoration: none;
}
</style> </style>
@@ -18,7 +18,6 @@
@alias="$emit('alias', $event)" @alias="$emit('alias', $event)"
@remove-alias="$emit('remove-alias', $event)" @remove-alias="$emit('remove-alias', $event)"
@dismiss="$emit('dismiss', $event)" @dismiss="$emit('dismiss', $event)"
@undismiss="$emit('undismiss', $event)"
/> />
</div> </div>
</div> </div>
@@ -34,7 +33,7 @@ const props = defineProps({
collapsible: { type: Boolean, default: false }, collapsible: { type: Boolean, default: false },
defaultOpen: { type: Boolean, default: true } defaultOpen: { type: Boolean, default: true }
}) })
defineEmits(['accept', 'alias', 'remove-alias', 'dismiss', 'undismiss']) defineEmits(['accept', 'alias', 'remove-alias', 'dismiss'])
const open = ref(props.collapsible ? props.defaultOpen : true) const open = ref(props.collapsible ? props.defaultOpen : true)
</script> </script>
@@ -12,25 +12,22 @@
No suggestions above threshold. No suggestions above threshold.
</div> </div>
<!-- Flows in the rail's main scroll area; Related is pinned to the bottom <template v-else>
of the rail (ImageViewer side layout), so a long suggestion set no
longer needs an internal scroll cap to keep Related reachable. -->
<div v-else class="fc-suggestions__list">
<SuggestionsCategoryGroup <SuggestionsCategoryGroup
v-for="cat in peopleCats" :key="cat" v-for="cat in peopleCats" :key="cat"
v-show="store.byCategory[cat] && store.byCategory[cat].length" v-show="store.byCategory[cat] && store.byCategory[cat].length"
:label="labelFor(cat)" :items="store.byCategory[cat] || []" :label="labelFor(cat)" :items="store.byCategory[cat] || []"
@accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias" @accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias"
@dismiss="onDismiss" @undismiss="onUndismiss" @dismiss="store.dismiss"
/> />
<SuggestionsCategoryGroup <SuggestionsCategoryGroup
v-if="store.byCategory.general && store.byCategory.general.length" v-if="store.byCategory.general && store.byCategory.general.length"
label="General" :items="store.byCategory.general" label="General" :items="store.byCategory.general"
collapsible :default-open="true" collapsible :default-open="true"
@accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias" @accept="onAccept" @alias="onAlias" @remove-alias="onRemoveAlias"
@dismiss="onDismiss" @undismiss="onUndismiss" @dismiss="store.dismiss"
/> />
</div> </template>
<v-dialog v-model="aliasDialog" max-width="480"> <v-dialog v-model="aliasDialog" max-width="480">
<AliasPickerDialog <AliasPickerDialog
@@ -50,25 +47,12 @@ import { useModalStore } from '../../stores/modal.js'
import SuggestionsCategoryGroup from './SuggestionsCategoryGroup.vue' import SuggestionsCategoryGroup from './SuggestionsCategoryGroup.vue'
import AliasPickerDialog from './AliasPickerDialog.vue' import AliasPickerDialog from './AliasPickerDialog.vue'
const props = defineProps({ const props = defineProps({ imageId: { type: Number, required: true } })
imageId: { type: Number, required: true }, // 'accepted' lets the parent return focus to the tag input after a suggestion is
// The tagging host whose chip rail to refresh after an accept. Defaults to // applied (operator-asked 2026-06-08).
// the modal store (image modal); the Explore workspace passes its anchor host const emit = defineEmits(['accepted'])
// so the same panel refreshes the right surface. See TagPanel.
host: { type: Object, default: null },
})
// 'accepted'/'dismissed' let the parent return focus to the tag input after a
// suggestion is accepted OR rejected, so the operator keeps the keyboard flow on
// the input without re-clicking (operator-asked 2026-06-08, 2026-06-30).
const emit = defineEmits(['accepted', 'dismissed'])
// Reject (✗) / un-reject (↶): apply the store change, then signal the parent to
// re-focus the tag input — same return-to-input behaviour as accept.
function onDismiss (s) { store.dismiss(s); emit('dismissed') }
function onUndismiss (s) { store.undismiss(s); emit('dismissed') }
const store = useSuggestionsStore() const store = useSuggestionsStore()
const modalStore = useModalStore() const modal = useModalStore()
const host = props.host || modalStore
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as // 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as
// suggestion categories. Only 'character' remains as a people-style // suggestion categories. Only 'character' remains as a people-style
@@ -96,7 +80,7 @@ watch(() => props.imageId, (id) => {
async function onAccept(s) { async function onAccept(s) {
try { try {
await store.accept(s) await store.accept(s)
await host.reloadTags() await modal.reloadTags()
emit('accepted') emit('accepted')
} catch (e) { } catch (e) {
toast({ text: `Accept failed: ${e.message}`, type: 'error' }) toast({ text: `Accept failed: ${e.message}`, type: 'error' })
@@ -110,7 +94,7 @@ async function onAliasConfirm(canonicalTagId) {
try { try {
await store.aliasAccept(aliasTarget.value, canonicalTagId) await store.aliasAccept(aliasTarget.value, canonicalTagId)
aliasDialog.value = false aliasDialog.value = false
await host.reloadTags() await modal.reloadTags()
emit('accepted') emit('accepted')
} catch (e) { } catch (e) {
toast({ text: `Alias failed: ${e.message}`, type: 'error' }) toast({ text: `Alias failed: ${e.message}`, type: 'error' })
@@ -141,12 +125,6 @@ async function onRemoveAlias(s) {
color: rgb(var(--v-theme-on-surface)); color: rgb(var(--v-theme-on-surface));
margin-bottom: 8px; margin-bottom: 8px;
} }
.fc-suggestions__list {
/* No internal scroll cap: the rail now scrolls suggestions in its single
main scroll area while Related is pinned to the bottom (ImageViewer side
layout), so suggestions flow naturally without a nested scrollbar. */
padding-right: 4px;
}
.fc-suggestions__skeleton { display: flex; flex-direction: column; gap: 8px; } .fc-suggestions__skeleton { display: flex; flex-direction: column; gap: 8px; }
.fc-suggestions__skel-row { .fc-suggestions__skel-row {
height: 18px; border-radius: 4px; height: 18px; border-radius: 4px;
@@ -72,7 +72,7 @@
</v-icon> </v-icon>
</template> </template>
<v-list-item-title> <v-list-item-title>
{{ createLabel }} Create "{{ parsedName }}" as {{ parsedKind }}
</v-list-item-title> </v-list-item-title>
</v-list-item> </v-list-item>
</template> </template>
@@ -178,34 +178,14 @@ watch(query, () => {
}, 200) }, 200)
}) })
// A same-name character ALREADY exists. Characters are unique by
// (name, kind, fandom), so this is still a valid distinct tag in another fandom.
const sameNameCharExists = computed(() =>
parsedKind.value === 'character' &&
hits.value.some(h =>
h.kind === 'character' && h.name.toLowerCase() === parsedName.value.toLowerCase(),
),
)
const allowCreate = computed(() => { const allowCreate = computed(() => {
const q = parsedName.value const q = parsedName.value
if (!q) return false if (!q) return false
// Characters disambiguate by fandom, so a same-named character in a DIFFERENT
// fandom is a valid new tag — always offer Create (the fandom picker resolves
// it; find_or_create is idempotent if you re-pick the same fandom). Other
// kinds are unique by (name, kind): an exact match means it already exists.
if (parsedKind.value === 'character') return true
return !hits.value.some(h => return !hits.value.some(h =>
h.name.toLowerCase() === q.toLowerCase() && h.kind === parsedKind.value, h.name.toLowerCase() === q.toLowerCase() && h.kind === parsedKind.value,
) )
}) })
const createLabel = computed(() =>
sameNameCharExists.value
? `Create another "${parsedName.value}" character (different fandom)`
: `Create "${parsedName.value}" as ${parsedKind.value}`,
)
function scorePct (s) { return `${Math.round(s.score * 100)}%` } function scorePct (s) { return `${Math.round(s.score * 100)}%` }
// This image's suggestions that match the typed query, minus any the server // This image's suggestions that match the typed query, minus any the server
@@ -222,10 +202,6 @@ const suggestionHits = computed(() => {
const out = [] const out = []
for (const list of Object.values(suggestions.allByCategory)) { for (const list of Object.values(suggestions.allByCategory)) {
for (const s of list || []) { for (const s of list || []) {
// Rejected suggestions now stay in allByCategory (flagged) so the panel
// can show + un-reject them; keep them OUT of the type-to-add dropdown,
// whose job is finding a tag to ADD (un-reject lives in the panel).
if (s.rejected) continue
const key = `${s.category}:${s.display_name.toLowerCase()}` const key = `${s.category}:${s.display_name.toLowerCase()}`
if (!s.display_name.toLowerCase().includes(q)) continue if (!s.display_name.toLowerCase().includes(q)) continue
if (seen.has(key)) continue if (seen.has(key)) continue
+1 -12
View File
@@ -3,10 +3,6 @@
<v-chip <v-chip
size="small" closable size="small" closable
:color="store.colorFor(tag.kind)" variant="tonal" :color="store.colorFor(tag.kind)" variant="tonal"
class="fc-tag-chip__nav"
role="link"
:title="`Browse images tagged “${tag.name}”`"
@click="$emit('navigate', tag)"
@click:close="$emit('remove', tag.id)" @click:close="$emit('remove', tag.id)"
> >
<v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon> <v-icon start size="x-small">{{ iconFor(tag.kind) }}</v-icon>
@@ -34,7 +30,7 @@ import { useTagStore } from '../../stores/tags.js'
import KebabMenu from '../common/KebabMenu.vue' import KebabMenu from '../common/KebabMenu.vue'
const props = defineProps({ tag: { type: Object, required: true } }) const props = defineProps({ tag: { type: Object, required: true } })
defineEmits(['remove', 'rename', 'set-fandom', 'navigate']) defineEmits(['remove', 'rename', 'set-fandom'])
const store = useTagStore() const store = useTagStore()
@@ -57,13 +53,6 @@ function iconFor (k) { return KIND_ICONS[k] || 'mdi-tag' }
<style scoped> <style scoped>
.fc-tag-chip { display: inline-flex; align-items: center; gap: 1px; } .fc-tag-chip { display: inline-flex; align-items: center; gap: 1px; }
/* The chip body navigates to the filtered gallery (#5); signal it's clickable.
The close ✕ (remove) and the sibling kebab stay as the explicit controls. */
.fc-tag-chip__nav { cursor: pointer; }
.fc-tag-chip__nav:focus-visible {
outline: 2px solid rgb(var(--v-theme-accent));
outline-offset: 1px;
}
.fc-tag-chip__kebab { opacity: 0.7; } .fc-tag-chip__kebab { opacity: 0.7; }
.fc-tag-chip:hover .fc-tag-chip__kebab { opacity: 1; } .fc-tag-chip:hover .fc-tag-chip__kebab { opacity: 1; }
.fc-tag-chip__fandom { opacity: 0.7; font-size: 0.85em; } .fc-tag-chip__fandom { opacity: 0.7; font-size: 0.85em; }
+13 -49
View File
@@ -3,12 +3,11 @@
<h3 class="fc-tag-panel__title">Tags</h3> <h3 class="fc-tag-panel__title">Tags</h3>
<div class="fc-tag-panel__chips"> <div class="fc-tag-panel__chips">
<TagChip <TagChip
v-for="tag in host.current?.tags || []" v-for="tag in modal.current?.tags || []"
:key="tag.id" :tag="tag" :key="tag.id" :tag="tag"
@remove="onRemove" @rename="openRename" @set-fandom="openSetFandom" @remove="onRemove" @rename="openRename" @set-fandom="openSetFandom"
@navigate="onNavigate"
/> />
<span v-if="!host.current?.tags?.length" class="text-caption">No tags yet.</span> <span v-if="!modal.current?.tags?.length" class="text-caption">No tags yet.</span>
</div> </div>
<v-divider class="my-3" /> <v-divider class="my-3" />
@@ -24,22 +23,12 @@
</v-alert> </v-alert>
<SuggestionsPanel <SuggestionsPanel
v-if="host.currentImageId != null" v-if="modal.currentImageId != null"
:image-id="host.currentImageId" :image-id="modal.currentImageId"
:host="host"
@accepted="focusTagInput" @accepted="focusTagInput"
@dismissed="focusTagInput"
/> />
<!-- @after-leave: when either dialog finishes closing (apply OR cancel), <v-dialog v-model="renameDialog" max-width="420">
hand focus back to the tag input. Fires after Vuetify's own
focus-return to the activator, so ours wins; keeps the keyboard flow on
the input (operator-asked 2026-06-26). Mobile guard preserved via
focusTagInput → TagAutocomplete. -->
<v-dialog
v-model="renameDialog" max-width="420"
@after-leave="focusTagInput"
>
<TagRenameDialog <TagRenameDialog
v-if="renameTarget" v-if="renameTarget"
:tag="renameTarget" :tag="renameTarget"
@@ -50,7 +39,6 @@
<v-dialog <v-dialog
v-model="fandomDialog" max-width="460" v-model="fandomDialog" max-width="460"
@after-enter="fandomSetRef?.focusSearch?.()" @after-enter="fandomSetRef?.focusSearch?.()"
@after-leave="focusTagInput"
> >
<FandomSetDialog <FandomSetDialog
ref="fandomSetRef" ref="fandomSetRef"
@@ -63,7 +51,6 @@
<script setup> <script setup>
import { ref } from 'vue' import { ref } from 'vue'
import { useRouter } from 'vue-router'
import { useModalStore } from '../../stores/modal.js' import { useModalStore } from '../../stores/modal.js'
import { useSuggestionsStore } from '../../stores/suggestions.js' import { useSuggestionsStore } from '../../stores/suggestions.js'
import TagChip from './TagChip.vue' import TagChip from './TagChip.vue'
@@ -72,52 +59,29 @@ import SuggestionsPanel from './SuggestionsPanel.vue'
import TagRenameDialog from './TagRenameDialog.vue' import TagRenameDialog from './TagRenameDialog.vue'
import FandomSetDialog from './FandomSetDialog.vue' import FandomSetDialog from './FandomSetDialog.vue'
// `host` is the tagging context — a store-like object exposing const modal = useModalStore()
// current / currentImageId + removeTag/addExistingTag/createAndAdd/reloadTags
// (+ optional close). Defaults to the modal store so the image modal is
// unchanged; the Explore workspace passes its own host bound to the anchor,
// reusing this panel verbatim for modal-parity tagging.
const props = defineProps({ host: { type: Object, default: null } })
const modalStore = useModalStore()
const host = props.host || modalStore
const suggestions = useSuggestionsStore() const suggestions = useSuggestionsStore()
const router = useRouter()
const errorMsg = ref(null) const errorMsg = ref(null)
const tagInputRef = ref(null) const tagInputRef = ref(null)
// #5: clicking a tag chip's body leaves the current surface and opens the
// gallery filtered for that single tag (a fresh filter — the obvious "show me
// more like this tag" move). Rename/set-fandom (kebab) and remove (✕) stay put.
async function onNavigate(tag) {
await host.close?.()
router.push({ name: 'gallery', query: { tag_id: String(tag.id) } })
}
// Return focus to the tag input after a suggestion is accepted (from the // Return focus to the tag input after a suggestion is accepted (from the
// Suggestions panel or the autocomplete dropdown) so the operator can keep // Suggestions panel or the autocomplete dropdown) so the operator can keep
// typing the next tag without re-clicking the field (operator-asked 2026-06-08). // typing the next tag without re-clicking the field (operator-asked 2026-06-08).
// Exposed so a host surface (the Explore workspace) can re-focus the field when
// it swaps the focused image, without re-clicking. The field's own mobile guard
// (TagAutocomplete) is preserved.
function focusTagInput() { tagInputRef.value?.focus?.() } function focusTagInput() { tagInputRef.value?.focus?.() }
defineExpose({ focusTagInput })
// Every tag mutation hands focus back to the input so the operator can keep
// typing the next tag without re-clicking — matches the accept-suggestion flow
// (operator-asked 2026-06-26; the Explore workspace leans on this hard).
async function onRemove(tagId) { async function onRemove(tagId) {
errorMsg.value = null errorMsg.value = null
try { await host.removeTag(tagId); focusTagInput() } try { await modal.removeTag(tagId) }
catch (e) { errorMsg.value = e.message } catch (e) { errorMsg.value = e.message }
} }
async function onPickExisting(hit) { async function onPickExisting(hit) {
errorMsg.value = null errorMsg.value = null
try { await host.addExistingTag(hit.id); focusTagInput() } try { await modal.addExistingTag(hit.id) }
catch (e) { errorMsg.value = e.message } catch (e) { errorMsg.value = e.message }
} }
async function onPickNew(payload) { async function onPickNew(payload) {
errorMsg.value = null errorMsg.value = null
try { await host.createAndAdd(payload); focusTagInput() } try { await modal.createAndAdd(payload) }
catch (e) { errorMsg.value = e.message } catch (e) { errorMsg.value = e.message }
} }
// A suggestion picked from the autocomplete dropdown runs the SAME path as the // A suggestion picked from the autocomplete dropdown runs the SAME path as the
@@ -127,7 +91,7 @@ async function onAcceptSuggestion(s) {
errorMsg.value = null errorMsg.value = null
try { try {
await suggestions.accept(s) await suggestions.accept(s)
await host.reloadTags() await modal.reloadTags()
focusTagInput() focusTagInput()
} catch (e) { errorMsg.value = e.message } } catch (e) { errorMsg.value = e.message }
} }
@@ -141,8 +105,8 @@ function openRename(tag) {
} }
async function onRenamed() { async function onRenamed() {
renameDialog.value = false renameDialog.value = false
// Reflect the new name in the current tag list without a full reload. // Reflect the new name in the modal's current tag list without a full reload.
await host.reloadTags() await modal.reloadTags()
} }
const fandomDialog = ref(false) const fandomDialog = ref(false)
@@ -155,7 +119,7 @@ function openSetFandom(tag) {
async function onFandomUpdated() { async function onFandomUpdated() {
fandomDialog.value = false fandomDialog.value = false
// A fandom change can merge the tag away; reload to reflect the new state. // A fandom change can merge the tag away; reload to reflect the new state.
await host.reloadTags() await modal.reloadTags()
} }
</script> </script>
@@ -1,12 +1,9 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-tag-arrow-right-outline" <v-card-title>Tag aliases ({{ store.rows.length }})</v-card-title>
:title="`Tag aliases (${store.rows.length})`"
blurb="Alternate tag spellings that map to a canonical tag."
>
<v-data-table-virtual <v-data-table-virtual
:headers="headers" :items="store.rows" :loading="store.loading" :headers="headers" :items="store.rows" :loading="store.loading"
height="360" density="compact" fixed-header height="360" density="compact"
no-data-text="No aliases yet create one from a suggestion's menu." no-data-text="No aliases yet create one from a suggestion's menu."
> >
<template #item.mapping="{ item }"> <template #item.mapping="{ item }">
@@ -21,13 +18,12 @@
/> />
</template> </template>
</v-data-table-virtual> </v-data-table-virtual>
</MaintenanceTile> </v-card>
</template> </template>
<script setup> <script setup>
import { onMounted } from 'vue' import { onMounted } from 'vue'
import { useAliasesStore } from '../../stores/aliases.js' import { useAliasesStore } from '../../stores/aliases.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
const store = useAliasesStore() const store = useAliasesStore()
const headers = [ const headers = [
@@ -1,64 +1,33 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-playlist-check" <v-card-title>Allowlisted tags ({{ store.rows.length }})</v-card-title>
:title="`Allowlisted tags (${store.rows.length})`"
blurb="Tags auto-applied to images that score above their threshold. Tune the
threshold and see how many images it would cover."
>
<v-data-table-virtual <v-data-table-virtual
:headers="headers" :items="store.rows" :loading="store.loading" :headers="headers" :items="store.rows" :loading="store.loading"
height="360" density="compact" fixed-header height="360" density="compact"
no-data-text="No tags on the allowlist yet accept a suggestion to add one." no-data-text="No tags on the allowlist yet accept a suggestion to add one."
> >
<template #item.applied_count="{ item }">
<span class="fc-num">{{ item.applied_count ?? '—' }}</span>
</template>
<template #item.min_confidence="{ item }"> <template #item.min_confidence="{ item }">
<div class="fc-thr">
<v-text-field <v-text-field
:model-value="item.min_confidence" type="number" :model-value="item.min_confidence" type="number"
density="compact" hide-details style="max-width: 100px;" density="compact" hide-details style="max-width: 100px;"
:min="floor" max="1" step="0.05" :min="floor" max="1" step="0.05"
:aria-label="`Auto-apply threshold for ${item.tag_name}`" @update:model-value="(v) => onThreshold(item.tag_id, v)"
@update:model-value="(v) => onThreshold(item, v)"
/> />
<span
v-if="proj[item.tag_id]"
class="fc-thr__proj"
:class="{ 'fc-thr__proj--loading': proj[item.tag_id].loading }"
:title="`At ${proj[item.tag_id].threshold}, a sweep would cover this many images`"
>≈ {{ proj[item.tag_id].count }} at {{ proj[item.tag_id].threshold }}</span>
</div>
</template> </template>
<template #item.coverage_count="{ item }">
<span class="fc-num" :title="`Images a sweep covers at ${item.min_confidence}`">
{{ item.coverage_count ?? '—' }}
</span>
</template>
<template #item.actions="{ item }"> <template #item.actions="{ item }">
<v-btn <v-btn
icon="mdi-delete" size="x-small" variant="text" color="error" icon="mdi-delete" size="x-small" variant="text" color="error"
:aria-label="`Remove ${item.tag_name} from the allowlist`"
@click="store.remove(item.tag_id)" @click="store.remove(item.tag_id)"
/> />
</template> </template>
</v-data-table-virtual> </v-data-table-virtual>
<p class="fc-muted text-caption mt-2"> </v-card>
<strong>Applied</strong> = images currently carrying the tag.
<strong>Covers</strong> = images a sweep would auto-apply it to at the
current threshold. Lower the threshold to cover more (less certain) images.
</p>
</MaintenanceTile>
</template> </template>
<script setup> <script setup>
import { computed, onMounted, reactive } from 'vue' import { computed, onMounted } from 'vue'
import { useAllowlistStore } from '../../stores/allowlist.js' import { useAllowlistStore } from '../../stores/allowlist.js'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
const store = useAllowlistStore() const store = useAllowlistStore()
const ml = useMLStore() const ml = useMLStore()
@@ -68,53 +37,21 @@ const ml = useMLStore()
const floor = computed(() => ml.settings?.tagger_store_floor ?? 0.70) const floor = computed(() => ml.settings?.tagger_store_floor ?? 0.70)
const headers = [ const headers = [
{ title: 'Tag', key: 'tag_name', sortable: true }, { title: 'Tag', key: 'tag_name', sortable: true },
{ title: 'Kind', key: 'tag_kind', sortable: true, width: 100 }, { title: 'Kind', key: 'tag_kind', sortable: true, width: 110 },
{ title: 'Applied', key: 'applied_count', sortable: true, width: 90 }, { title: 'Min confidence', key: 'min_confidence', sortable: false, width: 140 },
{ title: 'Min confidence', key: 'min_confidence', sortable: false, width: 220 }, { title: '', key: 'actions', sortable: false, width: 60 }
{ title: 'Covers', key: 'coverage_count', sortable: true, width: 90 },
{ title: '', key: 'actions', sortable: false, width: 56 }
] ]
// Per-row live projection while the operator drags a threshold:
// proj[tagId] = { threshold, count, loading }
const proj = reactive({})
onMounted(() => { onMounted(() => {
store.load() store.load()
if (!ml.settings) ml.loadSettings() if (!ml.settings) ml.loadSettings()
}) })
const debounces = {} let debounce = null
function onThreshold(item, value) { function onThreshold(tagId, value) {
const tagId = item.tag_id if (debounce) clearTimeout(debounce)
debounce = setTimeout(() => {
const v = Math.max(parseFloat(value), floor.value) const v = Math.max(parseFloat(value), floor.value)
if (!(v > 0 && v <= 1)) return if (v > 0 && v <= 1) store.updateThreshold(tagId, v)
const shown = Number(v.toFixed(2))
// Optimistic live projection box (loading until the count returns).
proj[tagId] = { threshold: shown, count: proj[tagId]?.count ?? '…', loading: true }
if (debounces[tagId]) clearTimeout(debounces[tagId])
debounces[tagId] = setTimeout(async () => {
try {
const { count } = await store.coverage(tagId, v)
proj[tagId] = { threshold: shown, count, loading: false }
} catch {
delete proj[tagId] // drop the projection rather than show a wrong number
}
// Commit the new threshold (also refreshes the row's stored coverage_count).
store.updateThreshold(tagId, v)
}, 500) }, 500)
} }
</script> </script>
<style scoped>
.fc-num { font-variant-numeric: tabular-nums; }
.fc-thr { display: flex; align-items: center; gap: 10px; }
.fc-thr__proj {
font-size: 12px;
font-variant-numeric: tabular-nums;
color: rgb(var(--v-theme-accent));
white-space: nowrap;
}
.fc-thr__proj--loading { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
</style>
@@ -1,12 +1,9 @@
<template> <template>
<!-- #713: re-extract PostAttachments that are really archives but were filed <!-- #713: re-extract PostAttachments that are really archives but were filed
opaquely before the magic-byte gate, and attach their images to the post. --> opaquely before the magic-byte gate, and attach their images to the post. -->
<MaintenanceTile <v-card>
icon="mdi-folder-zip-outline" <v-card-title>Re-extract archive attachments</v-card-title>
title="Re-extract archives" <v-card-text>
blurb="Extract images from zip attachments filed opaquely."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
Some posts attached an archive (zip) whose images weren't extracted Some posts attached an archive (zip) whose images weren't extracted
because the downloaded file had no usable extension. This scans existing because the downloaded file had no usable extension. This scans existing
@@ -18,7 +15,8 @@
</v-btn> </v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span> <span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" /> <QueueStatusBar queue="maintenance" queue-label="Maintenance" />
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
@@ -26,7 +24,6 @@ import { ref } from 'vue'
import { useApi } from '../../composables/useApi.js' import { useApi } from '../../composables/useApi.js'
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const api = useApi() const api = useApi()
@@ -1,9 +1,7 @@
<template> <template>
<MaintenanceTile <v-card class="fc-backup-card">
icon="mdi-database-export" <CardHeading icon="mdi-database-export" title="Backups" />
title="Backups" <v-card-text>
blurb="Database + image backups, schedules and restore."
>
<p class="fc-muted text-body-2 mb-4"> <p class="fc-muted text-body-2 mb-4">
pg_dump for the database (fast, nightly-schedulable); pg_dump for the database (fast, nightly-schedulable);
tar+zstd for the images (slow, manual only). Files live in tar+zstd for the images (slow, manual only). Files live in
@@ -82,6 +80,8 @@
@delete="onDelete" @delete="onDelete"
@tag="onTag" @tag="onTag"
/> />
</v-card-text>
<BackupConfirmModal <BackupConfirmModal
v-model="confirmOpen" v-model="confirmOpen"
:action="confirmAction" :action="confirmAction"
@@ -90,7 +90,7 @@
:description="confirmDescription" :description="confirmDescription"
@confirm="onConfirmSubmit" @confirm="onConfirmSubmit"
/> />
</MaintenanceTile> </v-card>
</template> </template>
<script setup> <script setup>
@@ -98,7 +98,7 @@ import { computed, onMounted, onUnmounted, ref } from 'vue'
import { useBackupStore } from '../../stores/backup.js' import { useBackupStore } from '../../stores/backup.js'
import BackupConfirmModal from '../modal/DestructiveConfirmModal.vue' import BackupConfirmModal from '../modal/DestructiveConfirmModal.vue'
import MaintenanceTile from '../common/MaintenanceTile.vue' import CardHeading from '../common/CardHeading.vue'
import BackupRunsTable from './BackupRunsTable.vue' import BackupRunsTable from './BackupRunsTable.vue'
const store = useBackupStore() const store = useBackupStore()
@@ -1,10 +1,7 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-vector-triangle" <v-card-title>Tag centroids</v-card-title>
title="Tag centroids" <v-card-text>
blurb="Rebuild SigLIP centroids for similarity suggestions."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
Rebuild the per-tag SigLIP centroids that power similarity-based Rebuild the per-tag SigLIP centroids that power similarity-based
suggestions. Runs nightly automatically; trigger manually after a suggestions. Runs nightly automatically; trigger manually after a
@@ -15,14 +12,14 @@
</v-btn> </v-btn>
<span v-if="done" class="ml-3 text-caption">Enqueued.</span> <span v-if="done" class="ml-3 text-caption">Enqueued.</span>
<QueueStatusBar queue="ml" queue-label="ML" /> <QueueStatusBar queue="ml" queue-label="ML" />
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { ref } from 'vue' import { ref } from 'vue'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const store = useMLStore() const store = useMLStore()
const busy = ref(false) const busy = ref(false)
@@ -1,10 +1,7 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-database-cog" <v-card-title>Database maintenance</v-card-title>
title="Database maintenance" <v-card-text>
blurb="VACUUM ANALYZE + per-table bloat stats."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
VACUUM (ANALYZE) reclaims dead-tuple bloat which slows the random VACUUM (ANALYZE) reclaims dead-tuple bloat which slows the random
showcase, since it samples physical blocks and refreshes the query showcase, since it samples physical blocks and refreshes the query
@@ -45,7 +42,8 @@
<p v-else class="text-caption mt-3" style="opacity: 0.6;"> <p v-else class="text-caption mt-3" style="opacity: 0.6;">
No table statistics yet. No table statistics yet.
</p> </p>
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
@@ -53,7 +51,6 @@ import { onMounted, ref } from 'vue'
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { useDbMaintenanceStore } from '../../stores/dbMaintenance.js' import { useDbMaintenanceStore } from '../../stores/dbMaintenance.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const store = useDbMaintenanceStore() const store = useDbMaintenanceStore()
@@ -1,270 +0,0 @@
<template>
<MaintenanceTile
icon="mdi-expansion-card"
title="GPU agent (CCIP + crops)"
blurb="Connect a desktop-GPU agent to embed characters (CCIP) and crops. It pulls work over HTTP — your database and Redis stay private."
:open="true"
>
<p class="fc-muted text-body-2 mb-3">
The agent is a container you run on the machine with the GPU. It
authenticates with the token below, leases jobs from this server, computes
on the GPU, and posts results back all over HTTP. Start it when you want
a burst; stop it to reclaim the card.
</p>
<!-- Token -->
<div class="fc-section-h mb-1">Agent token</div>
<div v-if="loading" class="fc-muted text-body-2">Loading</div>
<template v-else>
<div v-if="tokenValue" class="fc-token">
<code class="fc-token__val">{{ masked ? maskedToken : tokenValue }}</code>
<v-btn
size="x-small" variant="text" :icon="masked ? 'mdi-eye' : 'mdi-eye-off'"
:title="masked ? 'Reveal' : 'Hide'" @click="masked = !masked"
/>
<v-btn
size="x-small" variant="text" icon="mdi-content-copy"
title="Copy token" @click="onCopy"
/>
<v-btn
size="small" variant="text" color="accent" class="ml-auto"
prepend-icon="mdi-refresh" :loading="rotating" @click="onRotate"
>Rotate</v-btn>
</div>
<div v-else>
<v-btn
color="accent" variant="flat" rounded="pill" size="small"
prepend-icon="mdi-key-plus" :loading="rotating" @click="onRotate"
>Generate token</v-btn>
</div>
<p class="fc-muted text-caption mt-2 mb-0">
Point the agent at <code>{{ baseUrl }}</code> with this token. Rotating
invalidates the old token update the agent after you rotate.
</p>
</template>
<!-- Queue -->
<div class="fc-section-h mt-5 mb-2">Work queue</div>
<div class="fc-queue">
<div class="fc-q"><div class="fc-q__n">{{ queue.pending }}</div><div class="fc-q__l">pending</div></div>
<div class="fc-q"><div class="fc-q__n">{{ queue.leased }}</div><div class="fc-q__l">in flight</div></div>
<div class="fc-q"><div class="fc-q__n fc-good">{{ queue.done }}</div><div class="fc-q__l">done</div></div>
<div class="fc-q"><div class="fc-q__n" :class="queue.error ? 'fc-weak' : ''">{{ queue.error }}</div><div class="fc-q__l">errored</div></div>
</div>
<v-btn
class="mt-4" color="accent" variant="tonal" rounded="pill" size="small"
prepend-icon="mdi-account-box-multiple" :loading="backfilling" @click="onBackfill"
>Queue character embedding (CCIP)</v-btn>
<p class="fc-muted text-caption mt-2 mb-0">
Enqueues every image that doesn't have a CCIP embedding yet. Nothing
processes until the agent is running.
</p>
<v-btn
class="mt-3" color="accent" variant="tonal" rounded="pill" size="small"
prepend-icon="mdi-crop" :loading="backfillingSiglip" @click="onBackfillSiglip"
>Queue concept crops (SigLIP)</v-btn>
<p class="fc-muted text-caption mt-2 mb-0">
Enqueues every image that doesn't have concept-crop embeddings yet the
localized vectors that help small/local tags (glasses, etc.) surface. New
images get these automatically; this catches the back-catalogue.
</p>
<!-- Match strictness -->
<div class="fc-section-h mt-5 mb-1">Character-match strictness</div>
<div v-if="ml.settings" class="d-flex align-center" style="gap:12px">
<v-slider
v-model="threshold" :min="0.70" :max="0.95" :step="0.01"
color="accent" hide-details density="compact" class="flex-grow-1"
:loading="savingThreshold" @end="onSaveThreshold"
/>
<span class="fc-q__n" style="font-size:16px">{{ threshold.toFixed(2) }}</span>
</div>
<p class="fc-muted text-caption mt-1 mb-0">
How close a figure must be (CCIP cosine) to suggest a character. Higher =
stricter fewer but more confident matches. 0.85 recommended; below ~0.80
a heavily-tagged character starts matching everything.
</p>
<!-- Auto-apply -->
<div v-if="ml.settings" class="d-flex align-center mt-5" style="gap:12px">
<v-switch
v-model="autoApply" color="accent" hide-details density="compact"
:loading="savingAuto" label="Auto-apply confident matches"
@update:model-value="onSaveAuto"
/>
<v-text-field
v-model.number="autoThreshold" type="number" min="0.80" max="0.99"
step="0.01" density="compact" hide-details variant="outlined"
style="max-width:96px" :disabled="!autoApply" label="at"
@change="onSaveAuto"
/>
</div>
<p class="fc-muted text-caption mt-1 mb-0">
When on, a very-confident character match tags the image on its own (daily,
reversible) so identity tags keep flowing without review. Stricter than
the suggest cut; 0.92 recommended.
</p>
</MaintenanceTile>
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import { useGpuStore } from '../../stores/gpu.js'
import { useMLStore } from '../../stores/ml.js'
import { copyText } from '../../utils/clipboard.js'
const store = useGpuStore()
const ml = useMLStore()
const loading = ref(true)
const tokenValue = ref(null)
const masked = ref(true)
const rotating = ref(false)
const backfilling = ref(false)
const backfillingSiglip = ref(false)
const threshold = ref(0.85)
const savingThreshold = ref(false)
const autoApply = ref(true)
const autoThreshold = ref(0.92)
const savingAuto = ref(false)
const queue = ref({ pending: 0, leased: 0, done: 0, error: 0 })
let pollTimer = null
const baseUrl = computed(() => window.location.origin)
const maskedToken = computed(() => {
const t = tokenValue.value || ''
return t.length > 8 ? `${t.slice(0, 4)}••••••••${t.slice(-4)}` : '••••••••'
})
onMounted(async () => {
try {
tokenValue.value = (await store.token()).token
} catch { /* non-fatal */ } finally {
loading.value = false
}
await refreshQueue()
pollTimer = setInterval(() => { if (!document.hidden) refreshQueue() }, 5000)
try {
await ml.loadSettings()
if (ml.settings?.ccip_match_threshold != null) {
threshold.value = ml.settings.ccip_match_threshold
}
if (ml.settings?.ccip_auto_apply_enabled != null) {
autoApply.value = ml.settings.ccip_auto_apply_enabled
autoThreshold.value = ml.settings.ccip_auto_apply_threshold
}
} catch { /* non-fatal */ }
})
async function onSaveAuto() {
savingAuto.value = true
try {
await ml.patchSettings({
ccip_auto_apply_enabled: autoApply.value,
ccip_auto_apply_threshold: autoThreshold.value,
})
toast({ text: 'Auto-apply settings saved', type: 'success' })
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
savingAuto.value = false
}
}
onUnmounted(() => { if (pollTimer) clearInterval(pollTimer) })
async function onSaveThreshold() {
savingThreshold.value = true
try {
await ml.patchSettings({ ccip_match_threshold: threshold.value })
toast({ text: `Match strictness set to ${threshold.value.toFixed(2)}`, type: 'success' })
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
savingThreshold.value = false
}
}
async function refreshQueue() {
try { queue.value = await store.status() } catch { /* non-fatal */ }
}
async function onRotate() {
rotating.value = true
try {
tokenValue.value = (await store.rotateToken()).token
masked.value = false
toast({ text: 'New agent token generated — update your agent', type: 'success' })
} catch (e) {
toast({ text: `Could not rotate token: ${e.message}`, type: 'error' })
} finally {
rotating.value = false
}
}
async function onCopy() {
try {
await copyText(tokenValue.value || '') // resolves on success, throws on fail
toast({ text: 'Token copied', type: 'success' })
} catch {
toast({ text: 'Copy failed — select and copy manually', type: 'warning' })
}
}
async function onBackfill() {
backfilling.value = true
try {
await store.backfill('ccip')
toast({ text: 'Queued CCIP embedding — run the agent to process it', type: 'success' })
await refreshQueue()
} catch (e) {
toast({ text: `Could not queue backfill: ${e.message}`, type: 'error' })
} finally {
backfilling.value = false
}
}
async function onBackfillSiglip() {
backfillingSiglip.value = true
try {
await store.backfill('siglip')
toast({ text: 'Queued concept crops — run the agent to process them', type: 'success' })
await refreshQueue()
} catch (e) {
toast({ text: `Could not queue backfill: ${e.message}`, type: 'error' })
} finally {
backfillingSiglip.value = false
}
}
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-token {
display: flex; align-items: center; gap: 4px;
background: rgb(var(--v-theme-surface-light)); border-radius: 6px;
padding: 4px 6px 4px 10px;
}
.fc-token__val {
font-family: 'JetBrains Mono', monospace; font-size: 13px;
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
}
.fc-queue { display: flex; gap: 24px; }
.fc-q__n {
font-size: 20px; font-weight: 700; line-height: 1.1;
font-family: 'JetBrains Mono', monospace;
}
.fc-q__l {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style>
@@ -1,455 +0,0 @@
<template>
<MaintenanceTile
icon="mdi-brain"
title="Concept heads (the learning suggester)"
blurb="Train the per-concept heads that turn your tags into suggestions — they replace Camie and sharpen every time you accept or reject."
:open="headCount > 0 || running"
>
<p class="fc-muted text-body-2 mb-3">
A <strong>head</strong> is a tiny classifier trained on the SigLIP
embeddings already stored on your images your positives plus your
negatives (rejections). One is built per general/character concept with at
least <strong>{{ minPositives }}</strong> tagged images. Retrain after a
tagging session to fold in your latest accepts/rejects; scoring is live, so
the rail reflects a retrain on the next image you open.
</p>
<!-- Summary stats -->
<div class="fc-stats mb-3">
<div class="fc-stat">
<div class="fc-stat__n">{{ headCount }}</div>
<div class="fc-stat__l">heads</div>
</div>
<div class="fc-stat">
<div class="fc-stat__n">{{ graduatedCount }}</div>
<div class="fc-stat__l" title="Heads precise enough to auto-apply without review">auto-apply ready</div>
</div>
<div class="fc-stat">
<div class="fc-stat__n fc-stat__n--time">{{ lastTrained }}</div>
<div class="fc-stat__l">last trained</div>
</div>
</div>
<v-btn
v-if="!running"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-play" :loading="busy" @click="onTrain"
>{{ headCount > 0 ? 'Retrain heads' : 'Train heads' }}</v-btn>
<div v-if="running" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 fc-muted">Training (started {{ startedAgo }})</div>
</div>
<v-alert
v-if="lastError"
type="error" variant="tonal" density="compact" class="mt-3"
>Training failed: {{ lastError }}</v-alert>
<!-- Empty state -->
<div v-if="!running && headCount === 0" class="fc-empty mt-4">
<v-icon size="32" color="accent">mdi-brain</v-icon>
<p class="fc-muted text-body-2 mt-2 mb-0">
No heads yet. Tag a handful of images for the concepts you care about,
then train each concept with {{ minPositives }} tags becomes a head.
</p>
</div>
<!-- Per-concept table -->
<div v-if="heads.length" class="mt-4">
<div class="fc-muted text-caption mb-2">
{{ heads.length }} concept{{ heads.length === 1 ? '' : 's' }}, strongest first
(AP = average precision; auto-apply = precise enough to fire without review)
</div>
<div class="fc-table-wrap">
<table class="fc-table">
<thead>
<tr>
<th class="fc-l">Concept</th>
<th>Cat</th>
<th class="fc-r" title="Tagged positives the head trained on">+tags</th>
<th class="fc-r">AP</th>
<th class="fc-r" title="Precision at the suggest operating point">P</th>
<th class="fc-r" title="Recall at the suggest operating point">R</th>
<th class="fc-c"></th>
</tr>
</thead>
<tbody>
<tr v-for="h in heads" :key="h.tag_id">
<td class="fc-l">{{ h.name }}</td>
<td><span class="fc-cat">{{ h.category }}</span></td>
<td class="fc-r fc-mono">{{ h.n_pos }}</td>
<td class="fc-r fc-mono" :class="apClass(h.ap)">{{ pct(h.ap) }}</td>
<td class="fc-r fc-mono">{{ pct(h.precision) }}</td>
<td class="fc-r fc-mono">{{ pct(h.recall) }}</td>
<td class="fc-c">
<v-icon v-if="h.auto_apply" size="16" color="success"
title="Auto-apply ready">mdi-lightning-bolt</v-icon>
<span v-else class="fc-muted"></span>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<!-- Earned auto-apply -->
<div class="fc-auto mt-6">
<div class="d-flex align-center mb-1" style="gap: 10px;">
<v-icon size="18" color="accent">mdi-lightning-bolt</v-icon>
<span class="fc-section-h">Auto-apply</span>
<v-switch
v-model="autoEnabled" :loading="settingBusy" hide-details density="compact"
color="success" class="ml-auto" @update:model-value="onToggleAuto"
/>
</div>
<p class="fc-muted text-body-2 mb-3">
Graduated heads (, with {{ autoMinPosInput }} examples) apply their tag
on their own where they clear {{ Math.round((autoPrecisionInput || 0) * 100) }}%
precision. Every auto-tag is reversible; removing one teaches the head it
misfired.
</p>
<div class="d-flex mb-3" style="gap: 12px;">
<v-text-field
v-model.number="autoPrecisionInput" label="Precision target"
type="number" min="0.5" max="0.999" step="0.01" density="compact"
hide-details style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveSettings"
/>
<v-text-field
v-model.number="autoMinPosInput" label="Min examples to fire"
type="number" min="1" density="compact" hide-details
style="max-width: 200px;" :disabled="settingBusy"
@change="onSaveSettings"
/>
</div>
<div class="d-flex" style="gap: 8px;">
<v-btn
size="small" variant="tonal" color="accent" rounded="pill"
prepend-icon="mdi-eye-outline" :loading="autoBusy" @click="onPreview"
>Preview</v-btn>
<v-btn
size="small" variant="flat" color="accent" rounded="pill"
prepend-icon="mdi-lightning-bolt"
:loading="autoBusy || autoRunning" :disabled="!autoEnabled"
@click="onApplyNow"
>Apply now</v-btn>
</div>
<div v-if="autoRunning" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 fc-muted">Sweeping the library</div>
</div>
<div v-if="lastSweep && !autoRunning" class="mt-3">
<div class="fc-muted text-caption mb-1">
{{ lastSweep.dry_run ? 'Preview' : 'Applied' }} ·
{{ formatTime(lastSweep.finished_at) }} ·
{{ lastSweep.dry_run ? 'would apply' : 'applied' }}
<strong>{{ sweepTotal(lastSweep) }}</strong>
tag{{ sweepTotal(lastSweep) === 1 ? '' : 's' }}
</div>
<div v-if="sweepConcepts(lastSweep).length" class="fc-chips">
<span v-for="c in sweepConcepts(lastSweep)" :key="c.tag_id" class="fc-chip">
{{ c.name }} <strong>{{ c.applied }}</strong>
</span>
</div>
</div>
</div>
<!-- Performance / tuning -->
<div v-if="metricsConcepts.length" class="mt-5">
<div class="fc-section-h mb-1">How auto-apply is landing</div>
<div class="fc-muted text-caption mb-2">
Misfire = an auto-tag you removed; missed = a tag you added by hand that a
head should have caught. Tune the precision target from the misfire rate.
</div>
<div class="fc-table-wrap">
<table class="fc-table">
<thead>
<tr>
<th class="fc-l">Concept</th>
<th class="fc-r" title="Tags currently auto-applied">applied</th>
<th class="fc-r" title="Auto-tags you removed">misfires</th>
<th class="fc-r" title="Removed / (applied + removed)">rate</th>
<th class="fc-r" title="Tags you added by hand that a head exists for">missed</th>
</tr>
</thead>
<tbody>
<tr v-for="c in metricsConcepts" :key="c.tag_id">
<td class="fc-l">{{ c.name }}</td>
<td class="fc-r fc-mono">{{ c.n_auto_applied }}</td>
<td class="fc-r fc-mono">{{ c.n_misfires }}</td>
<td class="fc-r fc-mono" :class="rateClass(c.misfire_rate)">
{{ ratePct(c.misfire_rate) }}
</td>
<td class="fc-r fc-mono">{{ c.n_underfires }}</td>
</tr>
</tbody>
</table>
</div>
</div>
</MaintenanceTile>
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import { useHeadsStore } from '../../stores/heads.js'
import { useMLStore } from '../../stores/ml.js'
const store = useHeadsStore()
const mlSettings = useMLStore()
const summary = ref(null)
const busy = ref(false)
let pollTimer = null
// --- Auto-apply state ---
const autoEnabled = ref(false)
const autoPrecisionInput = ref(0.97)
const autoMinPosInput = ref(30)
const settingBusy = ref(false)
const autoBusy = ref(false)
const autoStatus = ref(null)
const metricsData = ref(null)
let autoTimer = null
const autoRunning = computed(() => autoStatus.value?.running_id != null)
const lastSweep = computed(() =>
(autoStatus.value?.runs || []).find(r => r.status !== 'running') || null)
const metricsConcepts = computed(() => metricsData.value?.concepts ?? [])
const headCount = computed(() => summary.value?.head_count ?? 0)
const graduatedCount = computed(() => summary.value?.graduated_count ?? 0)
const heads = computed(() => summary.value?.heads ?? [])
const running = computed(() => summary.value?.running_id != null)
const minPositives = computed(() => mlSettings.settings?.head_min_positives ?? 8)
const lastTrained = computed(() =>
summary.value?.last_trained_at ? relTime(summary.value.last_trained_at) : 'never')
// Surface the most recent terminal run's error (if it ended in error).
const lastError = computed(() => {
const r = (summary.value?.runs || []).find(x => x.status !== 'running')
return r && r.status === 'error' ? r.error : null
})
const startedAgo = computed(() => {
const r = (summary.value?.runs || []).find(x => x.status === 'running')
return r?.started_at ? formatTime(r.started_at) : ''
})
onMounted(async () => {
// Settings power the copy + the auto-apply tuning inputs.
try {
await mlSettings.loadSettings()
const s = mlSettings.settings || {}
autoEnabled.value = !!s.head_auto_apply_enabled
autoPrecisionInput.value = s.head_auto_apply_precision ?? 0.97
autoMinPosInput.value = s.head_auto_apply_min_positives ?? 30
} catch { /* non-fatal */ }
await refresh()
if (running.value) startPoll()
await refreshAuto()
if (autoRunning.value) startAutoPoll()
refreshMetrics()
})
onUnmounted(() => { stopPoll(); stopAutoPoll() })
async function refresh() {
try {
summary.value = await store.status()
} catch { /* non-fatal — the card still offers a fresh train */ }
}
function startPoll() {
stopPoll()
pollTimer = setInterval(async () => {
await refresh()
if (!running.value) stopPoll()
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onTrain() {
busy.value = true
try {
await store.train()
await refresh()
startPoll()
} catch (e) {
const msg = e.body?.running_id ? 'Training is already running.' : e.message
toast({ text: `Could not start training: ${msg}`, type: 'error' })
} finally {
busy.value = false
}
}
// --- Auto-apply ---
async function refreshAuto() {
try { autoStatus.value = await store.autoApplyStatus() } catch { /* non-fatal */ }
}
async function refreshMetrics() {
try { metricsData.value = await store.metrics() } catch { /* non-fatal */ }
}
function startAutoPoll() {
stopAutoPoll()
autoTimer = setInterval(async () => {
const was = autoRunning.value
await refreshAuto()
// Sweep just finished → refresh the counts + landing metrics.
if (was && !autoRunning.value) { refreshMetrics(); refresh() }
if (!autoRunning.value) stopAutoPoll()
}, 4000)
}
function stopAutoPoll() { if (autoTimer) { clearInterval(autoTimer); autoTimer = null } }
async function onToggleAuto(val) {
settingBusy.value = true
try {
await mlSettings.patchSettings({ head_auto_apply_enabled: !!val })
toast({ text: val ? 'Auto-apply on' : 'Auto-apply off', type: 'success' })
} catch (e) {
autoEnabled.value = !val // revert the switch
toast({ text: `Could not update: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
}
async function onSaveSettings() {
settingBusy.value = true
try {
await mlSettings.patchSettings({
head_auto_apply_precision: Number(autoPrecisionInput.value),
head_auto_apply_min_positives: Number(autoMinPosInput.value),
})
} catch (e) {
toast({ text: `Could not save: ${e.message}`, type: 'error' })
} finally {
settingBusy.value = false
}
}
function onPreview() { startSweep(true) }
function onApplyNow() { startSweep(false) }
async function startSweep(dryRun) {
autoBusy.value = true
try {
await store.autoApply(dryRun)
await refreshAuto()
startAutoPoll()
} catch (e) {
const code = e.body?.error
const msg = code === 'auto_apply_already_running' ? 'A sweep is already running.'
: code === 'auto_apply_disabled' ? 'Enable auto-apply first.'
: e.message
toast({ text: `Could not start sweep: ${msg}`, type: 'error' })
} finally {
autoBusy.value = false
}
}
function sweepConcepts(run) {
return (run?.report?.concepts || [])
.filter(c => c.applied > 0)
.sort((a, b) => b.applied - a.applied)
}
function sweepTotal(run) { return run?.n_applied ?? 0 }
function ratePct(x) { return x == null ? '—' : `${Math.round(x * 100)}%` }
function rateClass(x) {
if (x == null) return ''
if (x <= 0.03) return 'fc-good'
if (x <= 0.1) return 'fc-ok'
return 'fc-weak'
}
function pct(x) { return x == null ? '—' : `${Math.round(x * 100)}%` }
function apClass(ap) {
if (ap == null) return ''
if (ap >= 0.85) return 'fc-good'
if (ap >= 0.7) return 'fc-ok'
return 'fc-weak'
}
function formatTime(iso) {
if (!iso) return ''
try { return new Date(iso).toLocaleString() } catch { return iso }
}
function relTime(iso) {
try {
const d = (Date.now() - new Date(iso).getTime()) / 1000
if (d < 60) return 'just now'
if (d < 3600) return `${Math.floor(d / 60)}m ago`
if (d < 86400) return `${Math.floor(d / 3600)}h ago`
return `${Math.floor(d / 86400)}d ago`
} catch { return iso }
}
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-section-h {
font-size: 13px; font-weight: 700; letter-spacing: 0.03em;
text-transform: uppercase; color: rgb(var(--v-theme-on-surface));
}
.fc-auto {
border-top: 1px solid rgb(var(--v-theme-surface-light)); padding-top: 16px;
}
.fc-chips { display: flex; flex-wrap: wrap; gap: 6px; }
.fc-chip {
font-size: 12px; padding: 2px 8px; border-radius: 999px;
background: rgb(var(--v-theme-surface-light));
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-chip strong { color: rgb(var(--v-theme-on-surface)); }
.fc-stats { display: flex; gap: 28px; }
.fc-stat__n {
font-size: 22px; font-weight: 700; line-height: 1.1;
color: rgb(var(--v-theme-on-surface));
font-family: 'JetBrains Mono', monospace;
}
.fc-stat__n--time { font-size: 15px; font-weight: 600; }
.fc-stat__l {
font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em;
color: rgb(var(--v-theme-on-surface-variant));
}
.fc-empty {
text-align: center; padding: 18px 12px;
border: 1px dashed rgb(var(--v-theme-surface-light)); border-radius: 8px;
}
.fc-table-wrap {
max-height: 360px; overflow-y: auto;
border: 1px solid rgb(var(--v-theme-surface-light)); border-radius: 8px;
}
.fc-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.fc-table thead th {
position: sticky; top: 0; z-index: 1;
background: rgb(var(--v-theme-surface));
text-align: right; padding: 6px 10px; font-weight: 600;
color: rgb(var(--v-theme-on-surface-variant));
border-bottom: 1px solid rgb(var(--v-theme-surface-light));
white-space: nowrap;
}
.fc-table td {
padding: 5px 10px; text-align: right;
border-bottom: 1px solid rgba(var(--v-theme-surface-light), 0.5);
}
.fc-table tbody tr:hover { background: rgb(var(--v-theme-surface-light)); }
.fc-l { text-align: left !important; }
.fc-r { text-align: right; }
.fc-c { text-align: center !important; }
.fc-mono { font-family: 'JetBrains Mono', monospace; }
.fc-cat {
font-size: 10px; text-transform: uppercase; letter-spacing: 0.03em;
color: rgb(var(--v-theme-on-surface-variant));
background: rgb(var(--v-theme-surface-light));
padding: 1px 6px; border-radius: 999px;
}
.fc-good { color: rgb(var(--v-theme-success)); }
.fc-ok { color: rgb(var(--v-theme-on-surface)); }
.fc-weak { color: rgb(var(--v-theme-error)); }
</style>
@@ -31,7 +31,7 @@
</CardHeading> </CardHeading>
<v-data-table-virtual <v-data-table-virtual
:headers="headers" :items="store.tasks" :loading="store.tasksLoading" :headers="headers" :items="store.tasks" :loading="store.tasksLoading"
height="480" density="compact" fixed-header no-data-text="No tasks yet trigger a scan above." height="480" density="compact" no-data-text="No tasks yet trigger a scan above."
> >
<template #item.status="{ item }"> <template #item.status="{ item }">
<v-chip :color="statusColor(item.status)" size="small" variant="tonal"> <v-chip :color="statusColor(item.status)" size="small" variant="tonal">
@@ -1,10 +1,7 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-refresh" <v-card-title>ML backfill</v-card-title>
title="ML backfill" <v-card-text>
blurb="Re-run tagging + embeddings on images missing them."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
Re-run Camie + SigLIP on images missing predictions or embeddings Re-run Camie + SigLIP on images missing predictions or embeddings
for the current model versions. Safe to re-run. for the current model versions. Safe to re-run.
@@ -14,14 +11,14 @@
</v-btn> </v-btn>
<span v-if="done" class="ml-3 text-caption">Enqueued.</span> <span v-if="done" class="ml-3 text-caption">Enqueued.</span>
<QueueStatusBar queue="ml" queue-label="ML" /> <QueueStatusBar queue="ml" queue-label="ML" />
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { ref } from 'vue' import { ref } from 'vue'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const store = useMLStore() const store = useMLStore()
const busy = ref(false) const busy = ref(false)
@@ -1,10 +1,7 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-tune" <v-card-title>Suggestion thresholds</v-card-title>
title="Suggestion thresholds" <v-card-text v-if="store.settings">
blurb="Confidence cutoffs that gate auto-suggested tags + video sampling."
>
<div v-if="store.settings">
<v-row v-for="f in fields" :key="f.key"> <v-row v-for="f in fields" :key="f.key">
<v-col cols="12"> <v-col cols="12">
<v-slider <v-slider
@@ -66,16 +63,15 @@
/> />
</v-col> </v-col>
</v-row> </v-row>
</div> </v-card-text>
<div v-else><v-skeleton-loader type="paragraph" /></div> <v-card-text v-else><v-skeleton-loader type="paragraph" /></v-card-text>
</MaintenanceTile> </v-card>
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { reactive, watch } from 'vue' import { reactive, watch } from 'vue'
import { useMLStore } from '../../stores/ml.js' import { useMLStore } from '../../stores/ml.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
const store = useMLStore() const store = useMLStore()
// 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as // 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired as
@@ -1,46 +1,31 @@
<template> <template>
<div class="fc-maint"> <div class="fc-maint">
<p class="fc-muted text-body-2 mb-5"> <p class="text-body-2 mb-4">
One-off backfills, tagging config and storage tools. The ML backfill and Operational backfills and tagging controls. The ML backfill and centroid
centroid recompute also run nightly; the allowlist auto-applies accepted recompute run nightly automatically; the allowlist auto-applies accepted
tags. Click a tile to open it. tags to new and existing images. Use the cards below to trigger a
one-off pass.
</p> </p>
<div class="fc-maint__grid">
<section class="fc-section">
<h3 class="fc-section__title">Backfills &amp; reprocessing</h3>
<p class="fc-section__hint">Re-run tagging, thumbnails, extraction and DB upkeep.</p>
<div class="fc-tile-grid">
<MLBackfillCard /> <MLBackfillCard />
<CentroidRecomputeCard /> <CentroidRecomputeCard />
<ThumbnailBackfillCard /> <ThumbnailBackfillCard />
<ArchiveReextractCard />
<MissingFileRepairCard />
<DbMaintenanceCard />
</div> </div>
</section> <MLThresholdSliders class="mt-4" />
<AllowlistTable class="mt-4" />
<section class="fc-section"> <AliasTable class="mt-4" />
<h3 class="fc-section__title">Tagging</h3> <DbMaintenanceCard class="mt-6" />
<p class="fc-section__hint"> <ArchiveReextractCard class="mt-6" />
Suggestion thresholds, the auto-apply allowlist and tag aliases. <MissingFileRepairCard class="mt-6" />
</p> <BackupCard class="mt-6" />
<div class="fc-tile-stack"> <!-- VideoDedupCard + GatedPurgeCard moved to the Cleanup tab (2026-06-18):
<MLThresholdSliders /> both are destructive content cleanup, so "remove unwanted stuff" now
<HeadsCard /> lives in one place. -->
<GpuAgentCard /> <!-- TODO(2026-06-18): tile-ify this tab's cards into sections too (pass 2). -->
<AllowlistTable /> <!-- TagMaintenanceCard moved to Cleanup tab (v26.05.25.7) — it
<AliasTable /> operates on the existing library which fits the Cleanup-tab
<TagEvalCard /> theme, and clusters with the other audit cards. LegacyMigrationCard
</div> removed once the one-and-done GS/IR migration cutover completed. -->
</section>
<section class="fc-section">
<h3 class="fc-section__title">Storage</h3>
<p class="fc-section__hint">Database + image backups and restore.</p>
<div class="fc-tile-stack">
<BackupCard />
</div>
</section>
</div> </div>
</template> </template>
@@ -50,21 +35,18 @@ import { onMounted, onUnmounted } from 'vue'
import MLBackfillCard from './MLBackfillCard.vue' import MLBackfillCard from './MLBackfillCard.vue'
import CentroidRecomputeCard from './CentroidRecomputeCard.vue' import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue' import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import ArchiveReextractCard from './ArchiveReextractCard.vue'
import MissingFileRepairCard from './MissingFileRepairCard.vue'
import DbMaintenanceCard from './DbMaintenanceCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue' import MLThresholdSliders from './MLThresholdSliders.vue'
import HeadsCard from './HeadsCard.vue'
import GpuAgentCard from './GpuAgentCard.vue'
import AllowlistTable from './AllowlistTable.vue' import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue' import AliasTable from './AliasTable.vue'
import TagEvalCard from './TagEvalCard.vue' import DbMaintenanceCard from './DbMaintenanceCard.vue'
import ArchiveReextractCard from './ArchiveReextractCard.vue'
import MissingFileRepairCard from './MissingFileRepairCard.vue'
import BackupCard from './BackupCard.vue' import BackupCard from './BackupCard.vue'
import { useSystemActivityStore } from '../../stores/systemActivity.js' import { useSystemActivityStore } from '../../stores/systemActivity.js'
// Poll queue depths so each task tile's QueueStatusBar shows live pending // Poll queue depths so each card's QueueStatusBar shows live pending
// counts — the operator can see a backfill is already queued/running before // counts — the operator can see a backfill is already queued/running
// re-triggering it. // before re-triggering it.
const activity = useSystemActivityStore() const activity = useSystemActivityStore()
let qTimer = null let qTimer = null
onMounted(() => { onMounted(() => {
@@ -79,29 +61,9 @@ onUnmounted(() => {
</script> </script>
<style scoped> <style scoped>
.fc-section { margin-bottom: 24px; } .fc-maint__grid {
.fc-section__title {
font-size: 0.78rem;
font-weight: 700;
letter-spacing: 0.06em;
text-transform: uppercase;
color: rgb(var(--v-theme-accent));
margin-bottom: 2px;
}
.fc-section__hint {
font-size: 0.8rem;
color: rgb(var(--v-theme-on-surface-variant));
margin-bottom: 12px;
}
.fc-tile-grid {
display: grid; display: grid;
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
gap: 14px; gap: 16px;
align-items: start;
}
.fc-tile-stack {
display: flex;
flex-direction: column;
gap: 12px;
} }
</style> </style>
@@ -1,12 +1,9 @@
<template> <template>
<!-- #859: delete ImageRecords whose backing file is gone from disk (orphans <!-- #859: delete ImageRecords whose backing file is gone from disk (orphans
left by the external-attach unlink bug) so they stop 404-ing on playback. --> left by the external-attach unlink bug) so they stop 404-ing on playback. -->
<MaintenanceTile <v-card>
icon="mdi-file-remove-outline" <v-card-title>Repair missing-file records</v-card-title>
title="Repair missing files" <v-card-text>
blurb="Remove records whose file is gone from disk."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
Scans every image/video record and removes any whose underlying file is Scans every image/video record and removes any whose underlying file is
gone from disk orphaned records that otherwise 404 when you open the gone from disk orphaned records that otherwise 404 when you open the
@@ -19,7 +16,8 @@
</v-btn> </v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued </span> <span v-if="queued" class="ml-3 text-caption text-success">Queued </span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" /> <QueueStatusBar queue="maintenance" queue-label="Maintenance" />
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
@@ -27,7 +25,6 @@ import { ref } from 'vue'
import { useApi } from '../../composables/useApi.js' import { useApi } from '../../composables/useApi.js'
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const api = useApi() const api = useApi()
@@ -42,8 +42,8 @@
:loading="committing" :loading="committing"
@click="onCommit" @click="onCommit"
>Delete {{ preview.count }} bare post(s)</v-btn> >Delete {{ preview.count }} bare post(s)</v-btn>
<span v-if="bareResult" class="ml-3 text-caption text-success"> <span v-if="deleted != null" class="ml-3 text-caption text-success">
Deleted {{ bareResult.deleted ?? 0 }} ✓ Deleted {{ deleted }} ✓
</span> </span>
</div> </div>
</MaintenanceTile> </MaintenanceTile>
@@ -88,47 +88,78 @@
:loading="merging" :loading="merging"
@click="onMergeDupes" @click="onMergeDupes"
>Merge {{ dupPreview.posts_to_merge }} duplicate(s)</v-btn> >Merge {{ dupPreview.posts_to_merge }} duplicate(s)</v-btn>
<span v-if="dupResult" class="ml-3 text-caption text-success"> <span v-if="merged != null" class="ml-3 text-caption text-success">
Merged {{ dupResult.merged ?? 0 }} Merged {{ merged }}
</span> </span>
</div> </div>
</MaintenanceTile> </MaintenanceTile>
</template> </template>
<script setup> <script setup>
import { computed } from 'vue' import { computed, ref } from 'vue'
import { useAdminStore } from '../../stores/admin.js' import { useAdminStore } from '../../stores/admin.js'
import { usePreviewCommit } from '../../composables/usePreviewCommit.js'
import SampleNameGrid from '../common/SampleNameGrid.vue' import SampleNameGrid from '../common/SampleNameGrid.vue'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
const store = useAdminStore() const store = useAdminStore()
const preview = ref(null)
const loadingPreview = ref(false)
const committing = ref(false)
const deleted = ref(null)
// Bare-post prune: preview the count, then delete. After the apply the same const dupPreview = ref(null)
// predicate is empty, so collapse to count 0. const loadingDupPreview = ref(false)
const { const merging = ref(false)
previewData: preview, previewing: loadingPreview, committing, const merged = ref(null)
result: bareResult, runPreview: onPreview, runCommit: onCommit,
} = usePreviewCommit({
preview: () => store.pruneBarePosts({ dryRun: true }),
commit: () => store.pruneBarePosts({ dryRun: false }),
emptyPreview: { count: 0, sample_names: [] },
})
// Duplicate-post reconcile: same shape, collapse to zero groups after merge.
const {
previewData: dupPreview, previewing: loadingDupPreview, committing: merging,
result: dupResult, runPreview: onPreviewDupes, runCommit: onMergeDupes,
} = usePreviewCommit({
preview: () => store.reconcileDuplicatePosts({ dryRun: true }),
commit: () => store.reconcileDuplicatePosts({ dryRun: false }),
emptyPreview: { groups: 0, posts_to_merge: 0, sample: [] },
})
const dupSampleNames = computed(() => const dupSampleNames = computed(() =>
(dupPreview.value?.sample ?? []).map( (dupPreview.value?.sample ?? []).map(
(g) => `${g.title}${g.rows} rows`, (g) => `${g.title}${g.rows} rows`,
), ),
) )
async function onPreview() {
loadingPreview.value = true
deleted.value = null
try {
preview.value = await store.pruneBarePosts({ dryRun: true })
} finally {
loadingPreview.value = false
}
}
async function onCommit() {
committing.value = true
try {
const result = await store.pruneBarePosts({ dryRun: false })
deleted.value = result.deleted ?? 0
// Reflect the completed sweep — the predicate is identical to the preview,
// so after a commit there is nothing left to delete.
preview.value = { count: 0, sample_names: [] }
} finally {
committing.value = false
}
}
async function onPreviewDupes() {
loadingDupPreview.value = true
merged.value = null
try {
dupPreview.value = await store.reconcileDuplicatePosts({ dryRun: true })
} finally {
loadingDupPreview.value = false
}
}
async function onMergeDupes() {
merging.value = true
try {
const result = await store.reconcileDuplicatePosts({ dryRun: false })
merged.value = result.merged ?? 0
// Same predicate as the preview — after the merge there are no dup groups left.
dupPreview.value = { groups: 0, posts_to_merge: 0, sample: [] }
} finally {
merging.value = false
}
}
</script> </script>
@@ -1,303 +0,0 @@
<template>
<MaintenanceTile
icon="mdi-flask-outline"
title="Tagging eval (heads vs centroid)"
blurb="Measure whether a trained head beats the old centroid on your own tags — and whether tagging more sharpens it."
:open="!!run"
>
<p class="fc-muted text-body-2 mb-3">
Reuses the SigLIP embeddings already stored on your images (no re-embed, no
GPU). For each concept it trains a logistic-regression <strong>head</strong>
on your positives + negatives and compares it to the old single
<strong>centroid</strong>, with cross-validated AP/F1 and a learning curve.
Runs as a background task; the result is saved and reloads here.
</p>
<v-textarea
v-model="conceptsText" label="Concepts (comma-separated)"
rows="2" auto-grow density="compact" hide-details class="mb-3"
:disabled="running"
/>
<div class="d-flex mb-3" style="gap: 12px;">
<v-text-field
v-model.number="autoTopN" label="+ auto-add top-N concepts"
type="number" min="0" max="200" density="compact" hide-details
:disabled="running" style="max-width: 220px;"
/>
<v-text-field
v-model.number="precisionTarget" label="Auto-apply precision target"
type="number" min="0.5" max="0.999" step="0.01" density="compact" hide-details
:disabled="running" style="max-width: 220px;"
/>
</div>
<v-btn
v-if="!running"
color="accent" variant="flat" rounded="pill"
prepend-icon="mdi-play" :loading="busy" @click="onStart"
>Run eval</v-btn>
<div v-if="running" class="mt-3">
<v-progress-linear indeterminate color="accent" />
<div class="text-body-2 mt-2 fc-muted">Running (started {{ startedAgo }})</div>
</div>
<v-alert
v-if="run && run.status === 'error'"
type="error" variant="tonal" density="compact" class="mt-3"
>Eval failed: {{ run.error }}</v-alert>
<div v-if="report" class="mt-4">
<div class="fc-muted text-caption mb-2">
Ran {{ formatTime(report.generated_at) }} ·
{{ report.concepts.length }} concept(s) ·
neg ratio {{ report.params.neg_ratio }}, {{ report.params.cv_folds }}-fold CV
</div>
<div v-for="c in report.concepts" :key="c.name" class="fc-cc">
<div class="fc-cc__head">
<span class="fc-cc__name">{{ c.name }}</span>
<span v-if="c.skipped" class="fc-muted text-caption"> skipped: {{ c.skipped }}</span>
<span v-else class="fc-muted text-caption">
{{ c.n_pos }} pos · {{ c.n_neg }} neg<span v-if="c.n_rejected"> ({{ c.n_rejected }} rejected)</span>
</span>
</div>
<template v-if="!c.skipped">
<table class="fc-metrics">
<thead>
<tr><th></th><th>AP</th><th>F1</th><th>Prec</th><th>Rec</th></tr>
</thead>
<tbody>
<tr>
<td class="fc-metrics__lbl">Head</td>
<td class="fc-num fc-win">{{ c.head.ap }}</td>
<td class="fc-num">{{ c.head.f1 }}</td>
<td class="fc-num">{{ c.head.precision }}</td>
<td class="fc-num">{{ c.head.recall }}</td>
</tr>
<tr>
<td class="fc-metrics__lbl fc-muted">Centroid</td>
<td class="fc-num fc-muted">{{ c.centroid.ap }}</td>
<td class="fc-num fc-muted">{{ c.centroid.f1 }}</td>
<td class="fc-num fc-muted">{{ c.centroid.precision }}</td>
<td class="fc-num fc-muted">{{ c.centroid.recall }}</td>
</tr>
</tbody>
</table>
<div class="text-caption mb-2" :class="apDelta(c) >= 0 ? 'fc-up' : 'fc-down'">
Δ AP {{ apDelta(c) >= 0 ? '+' : '' }}{{ apDelta(c).toFixed(3) }}
(head centroid)
</div>
<div class="text-caption mb-2">
<span class="fc-muted">Auto-apply:</span>
<template v-if="c.head.auto_apply">
<span class="fc-up">ready</span> at P{{ c.head.auto_apply.target }}
catches recall <strong>{{ c.head.auto_apply.recall }}</strong>
(thr {{ c.head.auto_apply.threshold }})
</template>
<span v-else class="fc-down">not reachable at P{{ report.params.precision_target }}</span>
</div>
<div v-if="c.curve && c.curve.length" class="fc-curve">
<span class="fc-muted text-caption">Learning curve (AP @ N positives):</span>
<span v-for="p in c.curve" :key="p.n_pos" class="fc-curve__pt">
{{ p.n_pos }}<strong>{{ p.ap }}</strong>
</span>
</div>
<div v-if="c.examples" class="fc-ex">
<div
v-for="grp in [
{ dir: 'suggest', items: c.examples.head_would_suggest,
label: `Head would suggest — ✓ tag it, ✗ not ${c.name}` },
{ dir: 'doubts', items: c.examples.head_doubts_positive,
label: `Head doubts your tag — ✓ keep, ✗ remove (not ${c.name})` },
]" :key="grp.dir" class="fc-ex__row"
>
<div class="fc-muted text-caption mb-1">{{ grp.label }}</div>
<div class="fc-ex__thumbs">
<div
v-for="it in grp.items" :key="`${grp.dir}${it.id}`"
class="fc-ex__item"
:class="actedLabel(c, grp.dir, it) ? 'fc-ex__item--acted' : ''"
>
<button
type="button" class="fc-ex__thumb"
:title="`#${it.id} — click to enlarge`" @click="modal.open(it.id)"
>
<img :src="it.thumbnail_url" loading="lazy" />
</button>
<div v-if="actedLabel(c, grp.dir, it)" class="fc-ex__badge">
{{ actedLabel(c, grp.dir, it) }}
</div>
<div v-else class="fc-ex__acts">
<button
class="fc-act fc-act--yes" type="button"
:title="`Yes — it is ${c.name}`" @click="act(c, it, grp.dir, 'yes')"
><v-icon size="15">mdi-check</v-icon></button>
<button
class="fc-act fc-act--no" type="button"
:title="`No — not ${c.name}`" @click="act(c, it, grp.dir, 'no')"
><v-icon size="15">mdi-close</v-icon></button>
</div>
</div>
</div>
</div>
</div>
</template>
</div>
</div>
</MaintenanceTile>
</template>
<script setup>
import { toast } from '../../utils/toast.js'
import { computed, onMounted, onUnmounted, ref } from 'vue'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import { useTagEvalStore } from '../../stores/tagEval.js'
import { useModalStore } from '../../stores/modal.js'
const DEFAULT_CONCEPTS =
'glasses, cat, dog, horse, goblin, cum, lactation, fellatio, xray, stomach bulge'
const store = useTagEvalStore()
const modal = useModalStore()
const run = ref(null)
const conceptsText = ref(DEFAULT_CONCEPTS)
const autoTopN = ref(0)
const precisionTarget = ref(0.97)
const busy = ref(false)
let pollTimer = null
const running = computed(() => run.value?.status === 'running')
const report = computed(() => (run.value?.status === 'ready' ? run.value.report : null))
const startedAgo = computed(() =>
run.value?.started_at ? formatTime(run.value.started_at) : '')
// Rehydrate the persisted run on mount so the report survives navigation — the
// task runs backend-side regardless; we just reconnect to its row.
onMounted(async () => {
try {
const latest = await store.latest()
if (latest) {
run.value = await store.getRun(latest.id)
if (run.value.status === 'running') startPoll(latest.id)
}
} catch { /* non-fatal — card still works for a fresh run */ }
})
onUnmounted(stopPoll)
function startPoll(id) {
stopPoll()
pollTimer = setInterval(async () => {
try {
run.value = await store.getRun(id)
if (run.value.status !== 'running') stopPoll()
} catch (e) {
stopPoll()
toast({ text: `Eval poll failed: ${e.message}`, type: 'error' })
}
}, 5000)
}
function stopPoll() {
if (pollTimer) { clearInterval(pollTimer); pollTimer = null }
}
async function onStart() {
busy.value = true
try {
const concepts = conceptsText.value.split(',').map(s => s.trim()).filter(Boolean)
const res = await store.start({
concepts,
auto_top_n: Number(autoTopN.value) || 0,
precision_target: Number(precisionTarget.value) || 0.97,
})
run.value = await store.getRun(res.run_id)
startPoll(res.run_id)
} catch (e) {
const msg = e.body?.running_id
? 'An eval is already running.'
: e.message
toast({ text: `Could not start eval: ${msg}`, type: 'error' })
} finally {
busy.value = false
}
}
function apDelta(c) { return (c.head?.ap ?? 0) - (c.centroid?.ap ?? 0) }
function formatTime(iso) {
if (!iso) return ''
try { return new Date(iso).toLocaleString() } catch { return iso }
}
// Acting on an example writes the SAME tables the head trains on, so a re-run
// reflects the correction. Keyed per (concept, list, image); the report ids are
// frozen at run time, so we just grey out what's been handled in this view.
const acted = ref({})
const actedKey = (c, dir, it) => `${c.tag_id}:${dir}:${it.id}`
const actedLabel = (c, dir, it) => acted.value[actedKey(c, dir, it)] || ''
async function act(c, it, dir, verdict) {
const key = actedKey(c, dir, it)
let call, label
if (dir === 'suggest' && verdict === 'yes') { call = store.applyTag(it.id, c.tag_id); label = 'tagged' }
else if (dir === 'suggest' && verdict === 'no') { call = store.rejectTag(it.id, c.tag_id); label = 'rejected' }
else if (dir === 'doubts' && verdict === 'no') { call = store.removeTag(it.id, c.tag_id); label = 'removed' }
else { call = store.confirmTag(it.id, c.tag_id); label = 'kept' } // doubt + yes = keep (confirm)
try {
await call
acted.value[key] = label
} catch (e) {
toast({ text: `Action failed: ${e.message}`, type: 'error' })
}
}
</script>
<style scoped>
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
.fc-cc {
padding: 12px 0;
border-top: 1px solid rgb(var(--v-theme-surface-light));
}
.fc-cc__head { display: flex; align-items: baseline; gap: 8px; margin-bottom: 6px; }
.fc-cc__name { font-weight: 600; }
.fc-metrics { width: 100%; max-width: 360px; border-collapse: collapse; font-size: 13px; }
.fc-metrics th { text-align: right; font-weight: 600; color: rgb(var(--v-theme-on-surface-variant)); padding: 0 8px; }
.fc-metrics__lbl { text-align: left; }
.fc-num { text-align: right; font-variant-numeric: tabular-nums; padding: 1px 8px; }
.fc-win { color: rgb(var(--v-theme-accent)); font-weight: 600; }
.fc-up { color: rgb(var(--v-theme-success)); }
.fc-down { color: rgb(var(--v-theme-error)); }
.fc-curve { margin-bottom: 8px; }
.fc-curve__pt { margin-left: 10px; font-size: 13px; font-variant-numeric: tabular-nums; }
.fc-ex__row { margin-top: 8px; }
.fc-ex__thumbs { display: flex; flex-wrap: wrap; gap: 6px; }
.fc-ex__item { position: relative; width: 120px; height: 120px; }
.fc-ex__item--acted { opacity: 0.45; }
.fc-ex__thumb {
display: block; width: 100%; height: 100%; border-radius: 6px;
overflow: hidden; background: rgb(var(--v-theme-surface-light));
outline: 1px solid transparent; transition: outline-color 0.12s;
border: none; padding: 0; cursor: pointer;
}
.fc-ex__thumb:hover { outline-color: rgb(var(--v-theme-accent)); }
.fc-ex__thumb img { width: 100%; height: 100%; object-fit: cover; display: block; }
.fc-ex__acts { position: absolute; top: 4px; right: 4px; display: flex; gap: 4px; }
.fc-act {
width: 26px; height: 26px; border-radius: 50%; border: none; cursor: pointer;
display: flex; align-items: center; justify-content: center; color: #fff;
opacity: 0.9; box-shadow: 0 1px 3px rgba(0, 0, 0, 0.4); transition: transform 0.1s;
}
.fc-act:hover { opacity: 1; transform: scale(1.1); }
.fc-act--yes { background: rgb(var(--v-theme-success)); }
.fc-act--no { background: rgb(var(--v-theme-error)); }
.fc-ex__badge {
position: absolute; bottom: 4px; left: 4px; right: 4px; text-align: center;
font-size: 10px; text-transform: uppercase; letter-spacing: 0.05em;
background: rgba(0, 0, 0, 0.65); color: #fff; border-radius: 3px; padding: 1px 0;
}
</style>
@@ -186,11 +186,11 @@
<v-btn <v-btn
color="error" variant="flat" rounded="pill" color="error" variant="flat" rounded="pill"
prepend-icon="mdi-format-letter-case" prepend-icon="mdi-format-letter-case"
:disabled="!normPreview.total_changes || !!normResult" :disabled="!normPreview.total_changes || normResult === 'queued'"
:loading="normCommitting" :loading="normCommitting"
@click="onNormCommit" @click="onNormCommit"
>Standardize {{ normPreview.total_changes }} tag group(s)</v-btn> >Standardize {{ normPreview.total_changes }} tag group(s)</v-btn>
<span v-if="normResult" class="ml-3 text-caption text-success"> <span v-if="normResult === 'queued'" class="ml-3 text-caption text-success">
Queued ✓ — runs in the background (you can leave this page). It Queued ✓ — runs in the background (you can leave this page). It
processes in chunks, so re-run “Preview” later to confirm it's all done. processes in chunks, so re-run “Preview” later to confirm it's all done.
</span> </span>
@@ -199,53 +199,106 @@
</template> </template>
<script setup> <script setup>
import { ref } from 'vue'
import { useAdminStore } from '../../stores/admin.js' import { useAdminStore } from '../../stores/admin.js'
import { usePreviewCommit } from '../../composables/usePreviewCommit.js'
import SampleNameGrid from '../common/SampleNameGrid.vue' import SampleNameGrid from '../common/SampleNameGrid.vue'
import MaintenanceTile from '../common/MaintenanceTile.vue' import MaintenanceTile from '../common/MaintenanceTile.vue'
const store = useAdminStore() const store = useAdminStore()
const preview = ref(null)
const loadingPreview = ref(false)
const committing = ref(false)
const kindPreview = ref(null)
const loadingKindPreview = ref(false)
const kindCommitting = ref(false)
const resetPreview = ref(null)
const loadingResetPreview = ref(false)
const resetCommitting = ref(false)
const normPreview = ref(null)
const loadingNormPreview = ref(false)
const normCommitting = ref(false)
const normResult = ref(null)
// Unused-tag prune. Collapse to count 0 but carry the apply's sample_names. async function onPreview() {
const { loadingPreview.value = true
previewData: preview, previewing: loadingPreview, committing, try {
runPreview: onPreview, runCommit: onCommit, preview.value = await store.pruneUnusedTags({ dryRun: true })
} = usePreviewCommit({ } finally {
preview: () => store.pruneUnusedTags({ dryRun: true }), loadingPreview.value = false
commit: () => store.pruneUnusedTags({ dryRun: false }), }
emptyPreview: (r) => ({ count: 0, sample_names: r.sample_names || [] }), }
})
// Legacy migration-tag purge. async function onCommit() {
const { committing.value = true
previewData: kindPreview, previewing: loadingKindPreview, try {
committing: kindCommitting, runPreview: onKindPreview, runCommit: onKindCommit, const result = await store.pruneUnusedTags({ dryRun: false })
} = usePreviewCommit({ preview.value = { count: 0, sample_names: result.sample_names || [] }
preview: () => store.purgeLegacyTags({ dryRun: true }), } finally {
commit: () => store.purgeLegacyTags({ dryRun: false }), committing.value = false
emptyPreview: { count: 0, by_kind: {}, by_prefix: {}, sample_names: [] }, }
}) }
// Reset content tagging (general + character). async function onKindPreview() {
const { loadingKindPreview.value = true
previewData: resetPreview, previewing: loadingResetPreview, try {
committing: resetCommitting, runPreview: onResetPreview, runCommit: onResetCommit, kindPreview.value = await store.purgeLegacyTags({ dryRun: true })
} = usePreviewCommit({ } finally {
preview: () => store.resetContentTagging({ dryRun: true }), loadingKindPreview.value = false
commit: () => store.resetContentTagging({ dryRun: false }), }
emptyPreview: { count: 0, by_kind: {}, applications: 0, sample_names: [] }, }
})
// Standardize casing. The apply DISPATCHES a self-resuming background task (no async function onKindCommit() {
// poll-until-done — that would falsely report complete after the first chunk), kindCommitting.value = true
// so there's no emptyPreview: leave the projection up; a truthy normResult means try {
// "queued". The operator re-runs Preview later to confirm it's all done. await store.purgeLegacyTags({ dryRun: false })
const { kindPreview.value = { count: 0, by_kind: {}, by_prefix: {}, sample_names: [] }
previewData: normPreview, previewing: loadingNormPreview, } finally {
committing: normCommitting, result: normResult, kindCommitting.value = false
runPreview: onNormPreview, runCommit: onNormCommit, }
} = usePreviewCommit({ }
preview: () => store.normalizeTags({ dryRun: true }),
commit: () => store.normalizeTags({ dryRun: false }), async function onResetPreview() {
}) loadingResetPreview.value = true
try {
resetPreview.value = await store.resetContentTagging({ dryRun: true })
} finally {
loadingResetPreview.value = false
}
}
async function onResetCommit() {
resetCommitting.value = true
try {
await store.resetContentTagging({ dryRun: false })
resetPreview.value = { count: 0, by_kind: {}, applications: 0, sample_names: [] }
} finally {
resetCommitting.value = false
}
}
async function onNormPreview() {
loadingNormPreview.value = true
normResult.value = null
try {
normPreview.value = await store.normalizeTags({ dryRun: true })
} finally {
loadingNormPreview.value = false
}
}
async function onNormCommit() {
normCommitting.value = true
normResult.value = null
try {
// Fire-and-forget: the task is time-boxed and self-resuming across chunks
// (a large back-catalog can't finish in one run), so we DON'T poll-until-
// done — that would falsely report "complete" after the first chunk. Just
// confirm it's queued; the operator can re-run Preview later to verify.
await store.normalizeTags({ dryRun: false })
normResult.value = 'queued'
} finally {
normCommitting.value = false
}
}
</script> </script>
@@ -1,10 +1,7 @@
<template> <template>
<MaintenanceTile <v-card>
icon="mdi-image-refresh" <v-card-title>Thumbnail backfill</v-card-title>
title="Thumbnail backfill" <v-card-text>
blurb="Regenerate missing or broken thumbnails."
:open="busy"
>
<p class="text-body-2 mb-3"> <p class="text-body-2 mb-3">
Scan the library for images with no thumbnail, or whose thumbnail file Scan the library for images with no thumbnail, or whose thumbnail file
is missing or corrupt on disk. Repair candidates are re-enqueued for is missing or corrupt on disk. Repair candidates are re-enqueued for
@@ -22,14 +19,14 @@
· {{ result.ok }} ok · {{ result.ok }} ok
</span> </span>
<QueueStatusBar queue="thumbnail" queue-label="Thumbnail" /> <QueueStatusBar queue="thumbnail" queue-label="Thumbnail" />
</MaintenanceTile> </v-card-text>
</v-card>
</template> </template>
<script setup> <script setup>
import { toast } from '../../utils/toast.js' import { toast } from '../../utils/toast.js'
import { ref } from 'vue' import { ref } from 'vue'
import { useThumbnailsStore } from '../../stores/thumbnails.js' import { useThumbnailsStore } from '../../stores/thumbnails.js'
import MaintenanceTile from '../common/MaintenanceTile.vue'
import QueueStatusBar from './QueueStatusBar.vue' import QueueStatusBar from './QueueStatusBar.vue'
const store = useThumbnailsStore() const store = useThumbnailsStore()
const busy = ref(false) const busy = ref(false)
+1 -10
View File
@@ -15,16 +15,7 @@ async function request(method, url, { body, params, signal } = {}) {
if (params) { if (params) {
const search = new URLSearchParams() const search = new URLSearchParams()
for (const [k, v] of Object.entries(params)) { for (const [k, v] of Object.entries(params)) {
if (v === undefined || v === null) continue if (v !== undefined && v !== null) search.set(k, String(v))
// Array values serialize as a REPEATED key (k=a&k=b) — e.g. the
// gallery's tag_or OR-groups, read server-side via request.args.getlist.
if (Array.isArray(v)) {
for (const item of v) {
if (item !== undefined && item !== null) search.append(k, String(item))
}
} else {
search.set(k, String(v))
}
} }
const qs = search.toString() const qs = search.toString()
if (qs) fullUrl += (url.includes('?') ? '&' : '?') + qs if (qs) fullUrl += (url.includes('?') ? '&' : '?') + qs
@@ -1,67 +0,0 @@
import { computed, ref } from 'vue'
import { useHeadsStore } from '../stores/heads.js'
import { toast } from '../utils/toast.js'
// Shared "(re)train the concept heads" behaviour (#114): trigger + poll status,
// with toasts on start/finish. Backs both the Settings card and the Explore
// view's inline button so the active retrain works the same everywhere.
// Call start() on mount and stop() on unmount to manage the poll timer.
export function useHeadTraining() {
const store = useHeadsStore()
const summary = ref(null)
const busy = ref(false) // the trigger POST is in flight
let timer = null
const running = computed(() => summary.value?.running_id != null)
const headCount = computed(() => summary.value?.head_count ?? 0)
async function refresh() {
try {
summary.value = await store.status()
} catch { /* non-fatal — the button still offers a fresh train */ }
}
function startPoll() {
stopPoll()
timer = setInterval(async () => {
const wasRunning = running.value
await refresh()
// Announce completion once, on the running → done transition.
if (wasRunning && !running.value) {
toast({
text: `Heads retrained — ${headCount.value} concept${headCount.value === 1 ? '' : 's'}`,
type: 'success',
})
}
if (!running.value) stopPoll()
}, 5000)
}
function stopPoll() {
if (timer) { clearInterval(timer); timer = null }
}
// Reflect an already-running run (e.g. the nightly one) on mount.
async function start() {
await refresh()
if (running.value) startPoll()
}
async function train() {
busy.value = true
try {
await store.train()
toast({ text: 'Head training started…', type: 'info' })
await refresh()
startPoll()
} catch (e) {
const msg = e.body?.running_id ? 'Training is already running.' : e.message
toast({ text: `Could not start training: ${msg}`, type: 'error' })
} finally {
busy.value = false
}
}
return { summary, running, busy, headCount, refresh, train, start, stop: stopPoll }
}
@@ -1,52 +0,0 @@
import { ref } from 'vue'
// Canonical sync preview→commit flow for a maintenance tile: the operator
// previews (dry-run) → sees the projection → commits (apply), where the apply
// returns its result inline (no long Celery task). Owns the shared lifecycle —
// the two loading flags, the preview payload, and the apply result — so each
// tile stops hand-rolling its own `preview`/`loading`/`committing` refs +
// onPreview/onCommit handlers (this was duplicated across 6 maintenance flows;
// DRY pass #753).
//
// The tile supplies the `preview`/`commit` thunks (which call the right store
// action with dryRun true/false) and, optionally, `emptyPreview`: the shape to
// collapse the preview to after a successful apply — the apply uses the SAME
// backend predicate as the preview, so afterward there's nothing left. Pass a
// function to derive it from the apply result; omit it entirely for a commit
// that dispatches a background task and should leave the preview in place.
//
// Errors surface via the store's own lastError (shown in the tile's alert), so
// they're intentionally NOT captured here. For an apply that runs as a LONG
// Celery task the operator can navigate away from, use useMaintenanceTask.
export function usePreviewCommit ({ preview, commit, emptyPreview = null }) {
const previewData = ref(null)
const previewing = ref(false)
const committing = ref(false)
const result = ref(null)
async function runPreview () {
previewing.value = true
result.value = null // clear any prior apply badge when re-previewing
try {
previewData.value = await preview()
} finally {
previewing.value = false
}
}
async function runCommit () {
committing.value = true
try {
result.value = await commit()
if (emptyPreview !== null) {
previewData.value = typeof emptyPreview === 'function'
? emptyPreview(result.value)
: emptyPreview
}
} finally {
committing.value = false
}
}
return { previewData, previewing, committing, result, runPreview, runCommit }
}

Some files were not shown because too many files have changed in this diff Show More