51749e05db9ba71aba5e6e5a82c9714ca8138ced
109 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
6104452d2e |
feat(deps): vuetify 3→4 (batch 4 phase B, #1449)
Vuetify 4 is a Material-Design-3 styling refresh with revert snippets, not a component-API overhaul. FC's exposure was small: - Bump vuetify ^4, vite-plugin-vuetify ^2.1.0, vue ^3.5, engines node>=24. - Restore the dropped global CSS reset (minimal reset from the upgrade guide) in Vuetify's own low-precedence reset layer, so FC's margin-zeroing assumptions hold. - v-row prop→utility: 'dense' → density=compact (×3), align=center → class=align-center. - v-snackbar: multi-line removed → min-height=68. - v-autocomplete #item slot: item→internalItem (item now aliases raw) in TagPicker + GalleryFilterBar (item.raw.* → internalItem.raw.*). ACCEPTED (cosmetic, operator reviews live per plan #158): MD3 typography (text-body-2 ×73), non-uppercase buttons (v4 dropped the uppercase default), MD3 elevation. CI verifies BUILD only — the LOOK is the live-review pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
af0d39ed52 |
feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474)
Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d9a14e890d |
feat(system-tags): process auto-tag settings UI + mode-aware review strip (#1464)
Frontend for the system-tag refactor (milestone #157 step 6). - HeadsCard: new 'Auto-tag work-in-progress' section (enable + tag-confidence + conflict knobs) for wip/editor process auto-apply, mirroring the chrome card; copy notes they stay VISIBLE and the head only learns from titles/manual (no runaway). Presentation copy narrowed to banner-only. - HiddenReviewStrip: mode-aware — chrome flags read 'hidden as X / Keep hidden / Un-hide'; process flags read 'auto-tagged X / Keep tag / Remove tag'. Same endpoints (the backend returns mode), different words. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
571938781a |
feat(tagging): title-based WIP auto-tagging (#1458)
Auto-apply the `wip` system tag to posts whose TITLE explicitly declares
work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision
complement to the image-based ML `wip` head. WIP images are excluded from the
Explore/gallery browse, so honouring the artist's own label keeps unfinished
pieces out of the main browse.
- services/wip_title.py: precision-first token-anchored matcher (swipe/wiped
never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO
NOTHING, chunked under the psycopg param ceiling).
- importer: live hook on FRESH import only (never on deep-scan/supersede), so a
manually-removed WIP tag is never re-applied by a routine re-scan.
- maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep
(coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a
beat — a periodic re-run would silently undo manual removals.
- ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating
the live hook; GET/PATCH + POST /settings/wip-title/scan.
- Settings UI: toggle + "Scan existing posts" button.
- Tests: pure matcher unit tests + integration (apply idempotency, backfill
precision).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
aea2701c28 |
feat(translation): tunable acceptance floor (0.90) + per-post sticky override (#155)
The gate at a fixed 0.80 couldn't catch the real pain: Interpreter (fresh == cached, verified by probe) confidently mis-detects short ASCII English like "... WIP Part 1" as German at 0.86 — above the floor — so it was accepted and a re-translate reproduced it. Confidence alone can't separate the 0.86 collision (genuine German lands there too), and single-word mis-flags sit at a confident 1.0 no floor catches. Two operator-approved levers: - Acceptance floor is now a live Settings value (ImportSettings. translation_min_confidence, default 0.90; surfaced in the Translation card), so it's tunable without a redeploy. _accept takes the threshold as a parameter. - Per-post sticky override (Post.translation_override: auto/force/original). 'force' stores a translation even below the floor (rescue a skipped legit-foreign title); 'original' keeps the original and clears any stored translation (kill a confident mis-flag no floor catches). The sweep honors it on every run and _reset_translations skips 'original', so the choice survives a Re-translate-all. POST /api/posts/<id>/translation-override applies it immediately (translate now when the service is up, else queue for the sweep). UI: PostTranslationControl on the posts-feed card. Migration 0084 (both columns + a CHECK on the override). The feed + provenance serializers expose translation_override. With a stricter floor the rollback finally works: raise it -> Re-translate all -> the 0.86 mis-flags are rejected and restored to the original; force / keep-original handle the residual either way. Tests: gate thresholds against the param (0.86 rejected at 0.90, explicit-floor cases); sweep force/original + re-translate-skips-original; override endpoint (validation, original clears, force queues when disabled, feed exposes it); settings min_confidence default/save/validate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy |
||
|
bvandeusen
|
ffdbdbaf07 |
feat(translation): 8h sweep + drain button + detection probe (#1376)
Throughput: translate_posts now runs every 8h (was daily) as the steady-state cadence for newly-imported posts, and the Settings "Translate now" button runs it in drain mode (run-until-done, no reset) so one press clears the whole untranslated backlog instead of a single 300-post chunk. The interrupt/backoff re-enqueue now preserves the drain flag so a bulk drain resumes cleanly after an Interpreter restart. Misdetection groundwork: surface the detector's confidence from the Interpreter client (it was in the detectedLanguage payload but discarded) and add a read-only "Test translation" box — POST /settings/translation/ probe + TranslationCard UI — that shows detected language + confidence + engine + result for pasted text, without saving. Lets the operator see why a short/abbreviation-heavy English title gets mis-detected so the detection guard (min-length + confidence floor) can be tuned from real numbers. The guard itself follows once the mis-detected cases are probed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy |
||
|
bvandeusen
|
4371ddb7e7 |
feat(translation): live re-translate progress in the Settings card
translation/status now reports `active` (a translate/retranslate sweep is running, from the TaskRun table) and `last_run` (the most recent finished run's task + status). The Settings card polls live while a sweep runs, showing a spinner + "Translating… N remaining" that ticks down, and flags a last run that ended in error/timeout. No migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
1f6d94f51d |
feat(translation): re-translate on model change — artist-scoped + global re-run (#146)
retranslate_posts resets the 5 translation columns to NULL for a scoped set of posts (all, or WHERE artist_id IN ids) then reuses the untranslated sweep to re-run them, chasing the tail until drained (run-until-done). Interpreter cache keys on engine_version so a changed model re-translates, an unchanged one is cache-fast. Reset only happens when the service is configured+healthy so translations are never wiped when they can't be rebuilt. New POST /settings/translation/retranslate (artist_id | all=true). UI: per-artist 'Re-translate posts' on the Artist Management tab + 'Re-translate all' in the Settings Translation card, both with confirm dialogs. No migration (reuses m143 columns). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6a255482ea |
feat(translation): "Test connection" button — on-demand Interpreter health check (#143)
New POST /api/settings/translation/test pings /v1/health for a GIVEN base URL (not the saved one), so the operator can verify a URL before enabling it. TranslationCard gains a Test-connection button that reports reachable/unreachable inline and updates the status dot. Endpoint test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
83c1745fd0 |
feat(ui): translation-forward post text + Settings card (#143 step 5)
PostCard + modal ProvenancePanel show the English title/description by default when a translation exists, with a per-card "show original (<lang>)" toggle — translated bodies render as plain text, originals keep their sanitized HTML. New TranslationCard in Settings → Ingestion & filters: enable switch, Interpreter base URL (generic placeholder, no default host), target language, a reachability indicator + untranslated-posts count + "Translate now". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
1f548d8a7b |
feat(ui): presentation-chrome auto-hide Settings controls (#141 step 4)
HeadsCard gains a "Hide presentation chrome" section: on/off switch + "Hide confidence" (presentation_auto_apply_threshold) + "Flag if content ≥" (presentation_conflict_threshold), wired to MLSettings via patchSettings and loaded on mount. Makes the step-4 sweep's thresholds operator-tunable (config-in-UI). wip is called out as never auto-hidden. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
ab362bc79c |
feat(ml): Settings → Tagging 'Crop proposers' card (#134 step 3)
Exposes the detector config (per-proposer enable + weights + confidence, caps, dedupe IoU) in Settings → Tagging, backed by MLSettings via /api/ml/settings. ml_admin adds the detector fields to _EDITABLE + GET payload + validation (conf 0..1, caps >=1, IoU 0..1). New CropProposersCard.vue (mirrors HeadsCard) with working defaults pre-filled, per-field live-save (no restart — the agent picks changes up on its next lease), weights-format help, switch-revert on error. Closes milestone #134: all three proposers are on out-of-the-box and tunable in the UI. Test: detector defaults GET + patch round-trip + range validation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
e039689eff |
feat(ia): wave 2 — Activity becomes the whole-app pulse; Overview gets the health strip
The Activity tab only knew Celery — the GPU agent (the majority of processing) and the download pipeline were invisible there. Two new self-polling panels: - GpuActivityPanel: queue depths + triage verdicts (defects / file-ok / unprobed, top reason buckets) with a jump to Maintenance -> Failed processing. The triage detail refetches only when the error count moves. - DownloadsActivityPanel: 24h stat chips + failing-source names with a jump into Subscriptions. Both panels join the Activity tab under Queues+workers AND double as the Overview health strip (side-by-side grid under the Celery summary) — one component set, so Overview answers 'is everything healthy?' across all systems. SystemStatsCards reviewed: content still accurate, left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
5b34c9221c |
feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home
Settings IA per the approved A3 design (the old layout was the two-app merge fossilized): - Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual /import scans stay API-level; imports arrive via downloads/extension, heal via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm moves to Maintenance → 'Ingestion & filters' and loads its own settings; the import store shrinks to settings-only (no remaining consumers of the scan/task-list machinery). Overview's pending banner now points at Activity. - Maintenance regrouped: Ingestion & filters / GPU agent & embeddings (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders, Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive re-extract demoted last) / Storage. - One extension home: BrowserExtensionCard moves from Settings → Overview to Subscriptions → Settings, above the API key bar it authenticates. - Single-color import filter WIRED: skip_single_color/threshold existed since FC-2 but nothing read them (the audit module's docstring said as much) — now enforced on both import paths via the audit's canonical predicate (tolerance 30, matching the Cleanup card default; animated images exempt like the transparency check). Default stays off; test added. - Dead weight: PlaceholderView (zero refs) and the permanently-disabled 'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed (celery queue docstring, threshold comment citing retired tasks, ml package docstring, HeadsCard 'replaces Camie' blurb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
19b962f1a7 |
feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
7c19ad91ed |
feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback)
Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint says 'holding at the bandwidth cap' and /status reports bw_capped, so the behavior is legible without tests that need the ML stack. Reset content tagging: stays a FULL-instance reset (operator's call), but now lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale counts are rejected server-side). Copy no longer claims suggestions repopulate: it says plainly the heads' training examples are deleted and re-tagging starts fresh. Moved out of TagMaintenanceCard into DangerZoneCard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
eaea4308fc |
chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved)
The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
a7abcc41ca |
feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
|
||
|
bvandeusen
|
1f27189b8f |
chore: retire ml-backfill-daily beat + the spent purge-legacy action (operator-approved)
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's daily embed backfill for the same NULL-embedding images at ~100x the cost (B1 audit verdict, milestone #124). The backfill TASK stays — the manual /api/ml/backfill button remains the deliberate CPU fallback pending B3. - purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on the live library before removal (A2 audit, milestone #123). Fully retired per rule 22: tile, store action, route, service fn, tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
09e2772628 |
fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards
The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).
One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.
Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
reason instead of another release; strikes never count while stopping, and
clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
GpuJobService so they can't drift) converts expired leases with >=5 lease
grants and pending jobs with >=10 to 'error', preserving the last stored
failure reason. Backstops old agent builds.
Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
|
||
|
bvandeusen
|
3d6201734c |
fix(settings): maintenance tiles start collapsed; remember manual open state
GpuAgentCard was hardcoded :open=true, HeadsCard opened whenever any head existed, TagEvalCard whenever a persisted run existed — so a fresh Settings load greeted the operator with several tiles already expanded. All three now force-open only while their task is actually running (the #877 resurface behavior on the busy-driven tiles is untouched). MaintenanceTile additionally persists MANUAL expand/collapse per tile in localStorage, so the section reloads the way the operator left it; a forced open while a task runs stays transient and is never saved as a preference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusen
|
686808d3f3 |
feat(gpu): "Retry errored jobs" — scoped requeue of errors only
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
bvandeusen
|
359bc5a283 |
feat(ml): default to SigLIP 2 (new installs) + model dropdown, no free-text (#1203)
- Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in) — UPDATE applies ONLY where no image is embedded yet (fresh install), so an existing library is NOT silently invalidated; it switches deliberately via the dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2. - GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512 recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types. - GpuAgentCard: the two name/version text fields → a single model dropdown; Save sets name+version from the picked option (the current model is always selectable even if off-list). - embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1) to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the ml-worker's persistent HF_HOME. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
80f8eb4756 |
feat(gpu): re-process trigger to apply new crop detectors to the existing library (#1202)
The siglip/ccip backfills skip images that already have current-version regions, so adding crop detectors only affected NEW images — the back-catalogue would never be re-cropped. Add a reprocess trigger that resets every done/error job of a task back to pending, so the agent re-runs the FULL pipeline (figure detection + CCIP + concept/panel crops) over the whole library under the current detectors. - reprocess_gpu_jobs(task='ccip') task + POST /api/gpu/reprocess. - gpu store reprocess() + GpuAgentCard "Re-process library (re-detect + re-crop)" button with a confirm (it's heavy). - Test: a done job resets to pending (attempts cleared). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
bc6d43d3f2 |
refactor(ml): drop dead tagger/suggestion settings + columns (#1199)
Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
485387ff0b |
refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189)
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
3d77a38a25 |
refactor(ml): remove the dead per-tag centroid subsystem (#1189)
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
4daa3f2790 |
feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
c6f38b0dac |
feat(tagging): SigLIP concept crops + max-over-bag scoring (#114)
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray, lactation) that the whole-image SigLIP vector washes out: the GPU agent now embeds figure crops with SigLIP too, stored as kind='concept' regions, and the suggestion rail scores each image as a BAG (whole-image + every concept crop), taking each head's MAX over the bag. The whole-image vector is always in the bag, so this can never score lower than before. Model-agnostic by construction: the server ANNOUNCES the embedding model (HF name + version) in the lease, so the agent loads whatever the heads were trained in and stays in lock-step — a model swap is a server setting + a re-embed migration, never an agent change. - agent: model-agnostic CropEmbedder (torch/transformers get_image_features, fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the back-catalogue backfill never churns figure/CCIP regions; torch cu124 + transformers in the image. - server: lease announces embed_model_name/embed_version; score_image is max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill 'siglip' gates on a missing concept region (drains the back-catalogue, retries failures, no double-enqueue); daily siglip-backfill beat; UI button; /api/ccip/overview reports images_with_concept_siglip. - v1 scope: suggestion rail only — auto-apply stays whole-image (conservative; heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
b91a230f12 |
feat(ccip): automation + reference quality — keep identity flowing hands-free (#114)
Works through the optional CCIP ideas + the "keep moving even if I forget" ask:
AUTOMATION (no button needed):
- Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored
ones retried) on their own; the queue never goes idle waiting for a click.
- CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so
identity tags keep flowing. ON by default (opt-out, like head auto-apply);
ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut),
migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips
already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH
/api/ml/settings; auto_applied count in /api/ccip/overview.
REFERENCE QUALITY (the over-fire root cause):
- character_references now draws ONLY from single-character images — on a
multi-character image the tag is image-level, so every figure would otherwise
pollute each character's prototypes (a 2-char image tagged 'Velma' made
Daphne's figure a Velma reference). This is the contamination behind residual
over-firing.
- Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the
reference load isn't redone on every modal open.
Tests: multi-character image not used as a reference; auto-apply tags a confident
match as ccip_auto.
NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot
interesting content").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
625336b6b4 |
feat(ccip): tunable match threshold, default 0.85 (#114)
Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got
3-10 character guesses dominated by the most-referenced characters (a 27-ref
character clears a low bar on many images). A sweep showed 0.85 collapses the
noise (noisy multi-matches 47→3) while keeping the confident single-character
matches.
- ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image
reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85.
- Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999).
- Slider in the GPU agent card ("Character-match strictness") — tune live, no
redeploy, same observe-and-tune loop as auto-apply.
Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
d91eef7a4b |
feat(gpu): GPU agent admin card — token, queue, backfill (#114)
The FC-side control surface the operator asked for: Settings → Tagging → "GPU agent". Generate/reveal/copy/rotate the agent bearer token (with the FC URL to point the agent at), see the live job-queue depth (pending/in-flight/done/ errored, polled), and a "Queue character embedding (CCIP)" button that triggers the library backfill. Plain-HTTP-safe copy (copyText resolves on success, throws on fail). Closes the "how do I get the token in the UI" gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
1463794778 |
feat(heads): auto-apply UI on the Concept-heads card (#114 auto-apply C)
Surfaces earned auto-apply + its observability in Settings → Tagging → Concept heads: - Auto-apply section: an on/off switch (writes head_auto_apply_enabled), the precision-target + min-examples-to-fire tuning inputs, a Preview (dry-run → "would apply N", per-concept chips) and Apply-now button, with live run state. - "How auto-apply is landing": per-concept table from /api/heads/metrics — applied volume, misfires, realized misfire rate (green/amber/red), and missed (under-fires) — the signal to tune the precision target from. store: autoApply(dryRun) / autoApplyStatus() / metrics(). Card polls the sweep to completion, then refreshes counts + metrics. Completes the auto-apply task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
06d5e83da4 |
feat(heads): admin card to train + inspect concept heads (#114 B)
The UI for the heads subsystem: Settings → Tagging → "Concept heads". Shows head count, auto-apply-ready count, and last-trained; a Train/Retrain button (one run at a time, polls while running, surfaces a failed run's error); an empty state guiding the operator to tag first; and a per-concept table (name, category, +tags, AP, P, R, auto-apply ⚡) sorted strongest-first so weak/under- tagged concepts are obvious. Rehydrates status from GET /api/heads on mount so it survives navigation. Pulls head_min_positives from ML settings for copy. Slice C (swap the rail's suggestions to heads, remove Camie + centroid) is next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
b69c70ab2b |
feat(tag-eval): "keep" records a confirmation so doubts stop resurfacing
"Keep" on a doubted positive was a no-op, so the same confirmed-correct images came back in "head doubts" every run (operator-flagged: reinforcement keeps surfacing the same images). Add tag_positive_confirmation (mirror of tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and the eval excludes confirmed positives from the doubts list — exactly as rejected items already drop out of the suggest list. The tag stays a positive either way (confirmation is a "reviewed" marker, not a training change). - model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent). - tag_eval: _confirmed_ids + exclude from head_doubts_positive examples. - store.confirmTag + card "keep" calls it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5143f4c34f |
feat(tag-eval): auto-apply operating point + server-side top-N concept discovery
Two additions driven by "what's the commit threshold?" + "find more tags":
1. High-precision operating point (Bar 4). Per concept, report the threshold that
maximizes recall while holding precision >= a target (default 0.97, configurable
via `precision_target`) — i.e. "could this fire without a human, and how much
would it catch?" `head.auto_apply` = {target, threshold, precision, recall} or
null if the target is unreachable. Surfaced on the card.
2. Server-side concept auto-discovery. `auto_top_n` param unions the explicit
concept list with the N most-tagged general tags (one fast DB query) so the
eval can broaden itself without hand-listing — replaces the slow HTTP directory
paging. Card gains "+ auto-add top-N" and precision-target inputs.
No migration; numpy/sklearn stay lazy. Existing _normalize_params test still
holds (new keys additive; None still falls back to DEFAULT_CONCEPTS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
fc64f130b8 |
fix(tag-eval): thumbnail click opens the view modal, not Explore
Clicking an example in the maintenance card navigated to /explore/<id> — heavier than wanted (operator: just want a bigger look). Open the existing app-wide ImageViewer modal via modal.open(id) instead: bigger image + tags in place, no navigation away from Settings. The ✓/✗ actions are unaffected (separate overlay buttons). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
13d297b881 |
feat(tag-eval): inline confirm/reject actions on example thumbnails
Closes the learn-from-tags loop directly on the eval lists (operator-flagged: no surface to confirm/refine the head's suggestions). Each thumbnail gets a green ✓ / red ✗ that writes the SAME tables the head trains on: - suggest + ✓ → apply tag (new positive, POST /images/<id>/tags) - suggest + ✗ → record rejection (hard negative, suggestions/dismiss) - doubt + ✗ → remove tag + record rejection (kill bad positive, add negative) - doubt + ✓ → keep (stays a positive, no write) Acted thumbs grey out with a badge; re-run to see the head sharpen. Thumb still links to /explore/<id>. All endpoints already existed — no backend change. Inline is the starting point; longer-term the modal Suggestions rail gets the red "No" (negative) so per-image rejection is native there too (next slice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4974b7cf77 |
feat(tag-eval): bigger, clickable example thumbnails (label-review queue)
The 56px example thumbs were too small to judge a label (operator-flagged). Bump to 120px and wrap each in a link to /explore/<id> (new tab) so the "head doubts / would suggest" galleries double as a review-and-fix queue — click a doubted positive, land on it in Explore, correct the tag, re-run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6cd7281af5 |
feat(settings): tag-eval admin card — trigger + persisted report (survives nav)
Frontend for #1130. A maintenance tile in Settings → Tagging: - Editable concept list + "Run eval" → POST /api/tag-eval (one running at a time). - Rehydrates on mount via the persisted run (getRun by latest id) and polls while running — so the report SURVIVES navigation (operator-flagged); the task runs backend-side regardless and the card reconnects to its row. - Renders the saved report: per-concept head-vs-centroid metrics table (AP/F1/ precision/recall) with Δ AP, the learning curve (AP @ N positives), and thumbnail galleries (head-would-suggest / head-doubts-positive) for eyeballing. Backend: _examples now stores thumbnail_urls (not just ids) so the report is a self-contained artifact that renders without per-id lookups on reload. No new top-level surface — slots into the existing maintenance area. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
958378312c |
fix(settings): sticky headers on the virtual data tables
Allowlist / Alias / ImportTask tables scroll their bodies (height=360/480) but the column headers scrolled away with the rows, so you lost the column labels (operator-flagged 2026-06-27). Add Vuetify `fixed-header` so the header row stays pinned while the body scrolls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e49cea3eba |
feat(tagging): allowlist tuning dashboard + post-accept toast + merge preview UI (#7c/#7d/#8b)
Cluster B frontend, milestone #99. #7c: AllowlistTable gains Applied + Covers columns and a live 'covers ~N at T' projection as the operator drags a row's threshold (debounced coverage call, then commits the threshold). allowlist store gains coverage(tagId, threshold) and refreshes coverage_count after a save. #7d: suggestions store surfaces a non-blocking toast when accept/alias newly allowlists a tag — '<verb>: <tag> — allowlisted, auto-applying to ~N images' (N is the projection; apply runs async). Falls back to the plain toast when the tag was already allowlisted. #8b: TagsView merge picker now previews the merge via usePreviewCommit before committing — shows images moving / already-on-target / series pages / alias-or- delete / a thumbnail sample, blocks the Merge button on an incompatible kind/fandom. adminStore.mergeTags gains a dryRun option. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
7c94d99b9f |
refactor(settings): canonical usePreviewCommit for maintenance preview→commit tiles (#753)
Frontend pattern-consistency sweep (note #1026, the last DRY-thread item). TagMaintenanceCard (4 flows) + PostMaintenanceCard (2 flows) each hand-rolled the same sync preview→commit state machine: a previewData/previewing/committing triple + onPreview/onCommit that dry-run-previews, then applies and collapses the projection (the apply shares the backend predicate, so afterward it's empty). Extract usePreviewCommit({preview, commit, emptyPreview}) owning that lifecycle. The 6 flows become declarative: supply the two thunks + the collapse shape. The normalize flow (commit dispatches a self-resuming background task, not a sync apply) omits emptyPreview so the projection stays and a truthy result = queued. Composable returns are aliased to the cards' existing local names, so the templates only change where they read the apply result (the success badges). Long-Celery-task cards (GatedPurge/VideoDedup) keep useMaintenanceTask — a different pattern (navigable-away task lifecycle), deliberately not merged. Exhaustiveness: no card hand-rolls the refs anymore; the only dryRun:false callers are these two cards, both via the composable. Added a vitest spec for the primitive (collapse static + fn, dispatch-variant, re-preview clears result). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
311fe0ee9c |
feat(settings): tidy Maintenance tab into compact tiles + center the views (pass 2)
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled viewport; expanding a tile to read into it is the only reason to scroll. - Every Maintenance card converted to the collapsible MaintenanceTile (collapsed by default = icon + short title + one-line blurb). Task cards (ML backfill, centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance) sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion thresholds, allowlist, aliases) grouped in one Tagging section as collapsible tiles; Backup is its own collapsible tile. - Three labeled sections mirror the Cleanup tab: Backfills and reprocessing / Tagging / Storage. - Center the whole Settings surface: SettingsView is now a centered, width-capped (1140px) column so the tab strip and every panel sit in a tidy centered measure (was full-width). CleanupView drops its own left-aligned max-width to fill it. All card logic unchanged - only the chrome. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
3b435dc0ba |
feat(settings): tidy Cleanup tab into sectioned compact tiles (pass 1)
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles (2026-06-18: keep both tabs, group inside, compact tiles in a grid). New common/MaintenanceTile.vue: a compact expandable tile (icon + short title + one-line blurb collapsed; click the header to expand the full controls/preview/ result inline; keyboard-accessible button + focus ring; tints the icon, keeps a running task expanded). Cleanup tab (this pass) restructured into 3 sections — Import-filter audits (Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts, Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy, Reset content tagging, Standardize casing) — each a responsive grid of tiles. PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are destructive content cleanup). All card logic unchanged — only the chrome. Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
eff64275fc |
feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
e287802ecb |
fix(maint): resurface dedup/gated-purge results after navigate-away (#877)
Long-running maintenance tasks must survive navigating away or reloading
the page. VideoDedupCard + GatedPurgeCard held the in-flight Celery task id
only in component refs and polled task-result inline, so leaving the page
mid-run lost the id and the result was never shown — even though the task
finished on the worker.
New shared composable useMaintenanceTask: persists {taskId, mode, startedAt}
to localStorage on dispatch, re-attaches on mount, and re-shows the result
when the task finishes (the celery result backend retains the summary well
under result_expires). Stale-guard skips resume past 3h. Both cards refactored
onto it; card-specific computeds + confirm dialog kept.
Also fixed the QueueStatusBar lane: both cards watched queue="maintenance"
but tasks.admin.* routes to maintenance_long, so the bar never reflected
their own task — now queue="maintenance_long".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
540151290b |
feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
A one-shot Maintenance action to remove the blurred locked-preview images the ingester downloaded from tier-gated Patreon posts before #874. current_user_can_view was never persisted, so the cleanup re-walks each enabled Patreon source (read-only) to re-derive which posts are gated now and the blurred filehashes Patreon serves for them, then matches by CONTENT HASH against stored source_filehash. Because the hash is content-addressed, a real file downloaded when access existed has a different hash and can never match — regained-then-lost-access content is provably spared (operator's hard requirement). NULL source_filehash => unverifiable, kept + reported. On apply: delete matched ImageRecords + files (provenance cascades), clear seen/dead-letter ledger rows for those hashes so the real media re-ingests if access returns, and delete gated posts left bare. Shares one match predicate between preview and apply (rule 93). - cleanup_service: collect_gated_previews + purge_gated_previews - tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed) - api.admin: POST /maintenance/purge-gated-previews - GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply) - tests: collect predicate, hash-match delete/spare/unverifiable, ledger clear, bare-post removal, no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
369e3de684 |
feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)
Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag firing on one frame survived at peak confidence, and a fixed count under-samples long multi-scene videos so real scene-local tags looked like noise. Redesign (operator-steered): - Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds` (default 4) across the 5–95% window — so a tag's frame-presence reflects real screen time independent of video length. Capped at `video_max_frames` (default 64): a long video stretches the spacing instead of exploding into hundreds of inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg timeout also cut 60s→30s). - Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in >= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on screen — duration-independent noise rejection), with confidence = MEAN over the frames it appears in (not max). Clamps the threshold to the sample count so a 1–2-frame short video still tags. - All three knobs are DB-backed ml_settings (migration 0053), patchable via /api/ml/settings + sliders in the ML settings card — replaces the VIDEO_ML_FRAMES env var (product-not-project). Tests: aggregation drops one-frame noise + means corroborated tags + clamps on short videos; settings round-trip + min>max validation. Replaced the _maxpool_predictions unit test. NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs CPU-only — is GPU enablement, tracked separately in #872. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
41652db20f |
feat(maintenance): retroactive video-dedup action — preview + apply (#871)
Phase 2 of #871: clean up the duplicate videos already in the library (the #859 "same video from multiple sources" clutter). Import-time dedup (Phase 1) only prevents NEW dups; this is the operator-triggered cleanup of existing ones. cleanup_service.dedup_videos(dry_run): - backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the existing library participates; idempotent (only NULL rows), writes a negative sentinel for un-probeable files so they're neither re-probed forever nor matched. - find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect, anchored per cluster to bound the span (no chain drift); keeper = highest pixel area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances. - apply: re-point each loser's post links to the keeper (so no post loses the video) THEN delete the redundant records + files via delete_images (cascade). dry_run shares the same discovery predicate and returns the projection only (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated). - dedup_videos_task (maintenance queue; summary → task_run.metadata). - POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so the card shows the dry-run projection before the destructive apply. - VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind a confirm dialog. Mounted in the Maintenance panel. Tests: dedup collapses + re-links the loser's post to the keeper + removes the file; dry-run deletes nothing; distinct durations aren't grouped; task registered. (Migration 0052 for duration_seconds already shipped with Phase 1.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |