7e74fa767c41f3f3068f12aea80ec53cb2b55fa4
87 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
359bc5a283 |
feat(ml): default to SigLIP 2 (new installs) + model dropdown, no free-text (#1203)
- Migration 0069: new installs default to SigLIP 2 (so400m, 512px, 1152-d drop-in) — UPDATE applies ONLY where no image is embedded yet (fresh install), so an existing library is NOT silently invalidated; it switches deliberately via the dropdown → Re-embed → Retrain. Column server_defaults moved to SigLIP 2. - GET /api/ml/embedder-models: server-authoritative supported list (SigLIP 2 512 recommended / 384 faster / SigLIP 1 384 original) so the UI never free-types. - GpuAgentCard: the two name/version text fields → a single model dropdown; Save sets name+version from the picked option (the current model is always selectable even if off-list). - embedder.py DEFAULT_MODEL_NAME unchanged (stays the baked local-dir SigLIP 1) to avoid a local-dir/weights mismatch; SigLIP 2 loads by HF name, cached on the ml-worker's persistent HF_HOME. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
80f8eb4756 |
feat(gpu): re-process trigger to apply new crop detectors to the existing library (#1202)
The siglip/ccip backfills skip images that already have current-version regions, so adding crop detectors only affected NEW images — the back-catalogue would never be re-cropped. Add a reprocess trigger that resets every done/error job of a task back to pending, so the agent re-runs the FULL pipeline (figure detection + CCIP + concept/panel crops) over the whole library under the current detectors. - reprocess_gpu_jobs(task='ccip') task + POST /api/gpu/reprocess. - gpu store reprocess() + GpuAgentCard "Re-process library (re-detect + re-crop)" button with a confirm (it's heavy). - Test: a done job resets to pending (attempts cleared). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
bc6d43d3f2 |
refactor(ml): drop dead tagger/suggestion settings + columns (#1199)
Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
485387ff0b |
refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189)
Heads + CCIP are the tag source and head auto-apply is the earned propagation.
The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction
rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply
path firing in parallel with heads on every accept — exactly the un-earned spray
the v2 pivot replaced. Retire both.
Behavior change: accepting a suggestion now applies the tag to THAT image only
(source='ml_accepted', a head-training positive) — it no longer allowlists +
fans the tag across the library via Camie. Propagation is heads' earned
auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was
un-earned and bypassed the precision gate.)
- tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction
writes); backfill enqueues it for images with no embedding.
- Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat +
every enqueue caller (accept/alias/merge/per-image), api/allowlist.py +
blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067),
AllowlistTable.vue + allowlist store, the accept coverage-projection payload.
- AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store
the rail still needs); accept returns nothing, API returns {accepted, tag_id}.
- tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on
ML-applied image_tag sources (incl. head_auto) instead of the allowlist.
- UI: MLBackfillCard relabelled to embedding-only; accept toast simplified;
MaintenancePanel drops the allowlist tile.
Left for a follow-up hygiene pass (now-inert, harmless): the dead settings
columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*,
video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders
trim, and the Camie model download in download_models.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
3d77a38a25 |
refactor(ml): remove the dead per-tag centroid subsystem (#1189)
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
4daa3f2790 |
feat(ml): operator model swap — GPU re-embed + embedder as a setting (#1190)
Make the SigLIP embedder an operator choice (drop-in to SigLIP 2:
google/siglip2-so400m-patch16-512 is a verified 1152-d model at 512px → no
schema change, better small-cue fidelity). A swap = set model + re-embed +
retrain, all operator-driven; the GPU agent does the re-embed so it's fast.
- settings: embedder_model_name is now a setting (migration 0065) alongside the
existing embedder_model_version; both editable + validated (non-empty) in the
ml admin API. The server embedder loads by HF name (AutoImageProcessor/Model,
model-agnostic), preferring the pre-downloaded local dir for the default so
existing deploys don't re-download; rebuilds on a name change.
- agent: new 'embed' job = whole-image SigLIP embedding (mean-pool video frames)
under the lease-announced model → POST /jobs/submit_embedding writes
image_record.siglip_embedding + siglip_model_version. The lease now announces
the model FROM THE SETTING (not a constant).
- re-embed routing: enqueue_gpu_backfill('embed') selects unembedded + stale-
version images; 'siglip' now re-embeds concept crops whose version != current
(so a swap re-triggers crops, not just the never-embedded back-catalogue). The
CPU ml-worker backfill no longer re-embeds on a version mismatch (it can't
churn the library at 512px) — the GPU agent owns version re-embeds. Daily
'embed' + 'siglip' beats self-heal.
- scoring: score_image only bags embeddings in the CURRENT model's space (whole-
image gated by siglip_model_version, concept regions by embedding_version) so a
mid-swap stale vector isn't scored by new-space heads; legacy NULL = current.
- UI: GpuAgentCard "Embedding model (advanced)" — edit name/version, Save, and
"Re-embed library (GPU)" (queues embed + siglip); points at SigLIP 2.
Tests: lease announces model + submit_embedding round-trip; enqueue 'embed'
selects stale/unembedded; stale-version excluded from scoring; embedder model
settable + empty rejected; siglip gate updated to current-version concept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
c6f38b0dac |
feat(tagging): SigLIP concept crops + max-over-bag scoring (#114)
Lift recall on small/local concepts (glasses, cum, stomach-bulge, xray, lactation) that the whole-image SigLIP vector washes out: the GPU agent now embeds figure crops with SigLIP too, stored as kind='concept' regions, and the suggestion rail scores each image as a BAG (whole-image + every concept crop), taking each head's MAX over the bag. The whole-image vector is always in the bag, so this can never score lower than before. Model-agnostic by construction: the server ANNOUNCES the embedding model (HF name + version) in the lease, so the agent loads whatever the heads were trained in and stays in lock-step — a model swap is a server setting + a re-embed migration, never an agent change. - agent: model-agnostic CropEmbedder (torch/transformers get_image_features, fp16 on CUDA, inference-locked); worker branches on job.task — 'ccip' emits figure(CCIP)+concept(SigLIP) in one pass, 'siglip' emits concept-only so the back-catalogue backfill never churns figure/CCIP regions; torch cu124 + transformers in the image. - server: lease announces embed_model_name/embed_version; score_image is max-over-bag (version-filtered region embeddings); enqueue_gpu_backfill 'siglip' gates on a missing concept region (drains the back-catalogue, retries failures, no double-enqueue); daily siglip-backfill beat; UI button; /api/ccip/overview reports images_with_concept_siglip. - v1 scope: suggestion rail only — auto-apply stays whole-image (conservative; heads' thresholds were calibrated on whole-image). Bulk-apply bag = follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
b91a230f12 |
feat(ccip): automation + reference quality — keep identity flowing hands-free (#114)
Works through the optional CCIP ideas + the "keep moving even if I forget" ask:
AUTOMATION (no button needed):
- Hourly beat auto-enqueues CCIP backfill — new images get embedded (and errored
ones retried) on their own; the queue never goes idle waiting for a click.
- CCIP auto-apply: a daily sweep tags confident matches (source='ccip_auto') so
identity tags keep flowing. ON by default (opt-out, like head auto-apply);
ml_settings.ccip_auto_apply_enabled + _threshold (0.92, above the suggest cut),
migration 0064. Vectorized (one matmul + reduceat per image), reversible, skips
already-applied/rejected. Switch + threshold in the GPU agent card; GET/PATCH
/api/ml/settings; auto_applied count in /api/ccip/overview.
REFERENCE QUALITY (the over-fire root cause):
- character_references now draws ONLY from single-character images — on a
multi-character image the tag is image-level, so every figure would otherwise
pollute each character's prototypes (a 2-char image tagged 'Velma' made
Daphne's figure a Velma reference). This is the contamination behind residual
over-firing.
- Cached on a cheap signature (char-tag count + ccip-region count/max-id) so the
reference load isn't redone on every modal open.
Tests: multi-character image not used as a reference; auto-apply tags a confident
match as ccip_auto.
NEXT (not done, confirmed): comic-panel cropping + SigLIP concept crops ("spot
interesting content").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
625336b6b4 |
feat(ccip): tunable match threshold, default 0.85 (#114)
Live data showed the v1 flat 0.75 cosine over-fired — ~64% of matched images got
3-10 character guesses dominated by the most-referenced characters (a 27-ref
character clears a low bar on many images). A sweep showed 0.85 collapses the
noise (noisy multi-matches 47→3) while keeping the confident single-character
matches.
- ml_settings.ccip_match_threshold (migration 0063, default 0.85); match_image
reads it (override still accepted). DEFAULT_SIM_THRESHOLD fallback 0.75→0.85.
- Exposed in GET/PATCH /api/ml/settings (validated 0.5–0.999).
- Slider in the GPU agent card ("Character-match strictness") — tune live, no
redeploy, same observe-and-tune loop as auto-apply.
Test: a ~0.9-cosine figure matches at 0.85, dropped at 0.95.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
|
||
|
bvandeusen
|
d91eef7a4b |
feat(gpu): GPU agent admin card — token, queue, backfill (#114)
The FC-side control surface the operator asked for: Settings → Tagging → "GPU agent". Generate/reveal/copy/rotate the agent bearer token (with the FC URL to point the agent at), see the live job-queue depth (pending/in-flight/done/ errored, polled), and a "Queue character embedding (CCIP)" button that triggers the library backfill. Plain-HTTP-safe copy (copyText resolves on success, throws on fail). Closes the "how do I get the token in the UI" gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
1463794778 |
feat(heads): auto-apply UI on the Concept-heads card (#114 auto-apply C)
Surfaces earned auto-apply + its observability in Settings → Tagging → Concept heads: - Auto-apply section: an on/off switch (writes head_auto_apply_enabled), the precision-target + min-examples-to-fire tuning inputs, a Preview (dry-run → "would apply N", per-concept chips) and Apply-now button, with live run state. - "How auto-apply is landing": per-concept table from /api/heads/metrics — applied volume, misfires, realized misfire rate (green/amber/red), and missed (under-fires) — the signal to tune the precision target from. store: autoApply(dryRun) / autoApplyStatus() / metrics(). Card polls the sweep to completion, then refreshes counts + metrics. Completes the auto-apply task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
06d5e83da4 |
feat(heads): admin card to train + inspect concept heads (#114 B)
The UI for the heads subsystem: Settings → Tagging → "Concept heads". Shows head count, auto-apply-ready count, and last-trained; a Train/Retrain button (one run at a time, polls while running, surfaces a failed run's error); an empty state guiding the operator to tag first; and a per-concept table (name, category, +tags, AP, P, R, auto-apply ⚡) sorted strongest-first so weak/under- tagged concepts are obvious. Rehydrates status from GET /api/heads on mount so it survives navigation. Pulls head_min_positives from ML settings for copy. Slice C (swap the rail's suggestions to heads, remove Camie + centroid) is next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa |
||
|
bvandeusen
|
b69c70ab2b |
feat(tag-eval): "keep" records a confirmation so doubts stop resurfacing
"Keep" on a doubted positive was a no-op, so the same confirmed-correct images came back in "head doubts" every run (operator-flagged: reinforcement keeps surfacing the same images). Add tag_positive_confirmation (mirror of tag_suggestion_rejection): keep → POST /images/<id>/tags/<tag_id>/confirm, and the eval excludes confirmed positives from the doubts list — exactly as rejected items already drop out of the suggest list. The tag stays a positive either way (confirmation is a "reviewed" marker, not a training change). - model TagPositiveConfirmation + migration 0057; confirm endpoint (idempotent). - tag_eval: _confirmed_ids + exclude from head_doubts_positive examples. - store.confirmTag + card "keep" calls it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5143f4c34f |
feat(tag-eval): auto-apply operating point + server-side top-N concept discovery
Two additions driven by "what's the commit threshold?" + "find more tags":
1. High-precision operating point (Bar 4). Per concept, report the threshold that
maximizes recall while holding precision >= a target (default 0.97, configurable
via `precision_target`) — i.e. "could this fire without a human, and how much
would it catch?" `head.auto_apply` = {target, threshold, precision, recall} or
null if the target is unreachable. Surfaced on the card.
2. Server-side concept auto-discovery. `auto_top_n` param unions the explicit
concept list with the N most-tagged general tags (one fast DB query) so the
eval can broaden itself without hand-listing — replaces the slow HTTP directory
paging. Card gains "+ auto-add top-N" and precision-target inputs.
No migration; numpy/sklearn stay lazy. Existing _normalize_params test still
holds (new keys additive; None still falls back to DEFAULT_CONCEPTS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
fc64f130b8 |
fix(tag-eval): thumbnail click opens the view modal, not Explore
Clicking an example in the maintenance card navigated to /explore/<id> — heavier than wanted (operator: just want a bigger look). Open the existing app-wide ImageViewer modal via modal.open(id) instead: bigger image + tags in place, no navigation away from Settings. The ✓/✗ actions are unaffected (separate overlay buttons). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
13d297b881 |
feat(tag-eval): inline confirm/reject actions on example thumbnails
Closes the learn-from-tags loop directly on the eval lists (operator-flagged: no surface to confirm/refine the head's suggestions). Each thumbnail gets a green ✓ / red ✗ that writes the SAME tables the head trains on: - suggest + ✓ → apply tag (new positive, POST /images/<id>/tags) - suggest + ✗ → record rejection (hard negative, suggestions/dismiss) - doubt + ✗ → remove tag + record rejection (kill bad positive, add negative) - doubt + ✓ → keep (stays a positive, no write) Acted thumbs grey out with a badge; re-run to see the head sharpen. Thumb still links to /explore/<id>. All endpoints already existed — no backend change. Inline is the starting point; longer-term the modal Suggestions rail gets the red "No" (negative) so per-image rejection is native there too (next slice). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4974b7cf77 |
feat(tag-eval): bigger, clickable example thumbnails (label-review queue)
The 56px example thumbs were too small to judge a label (operator-flagged). Bump to 120px and wrap each in a link to /explore/<id> (new tab) so the "head doubts / would suggest" galleries double as a review-and-fix queue — click a doubted positive, land on it in Explore, correct the tag, re-run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
6cd7281af5 |
feat(settings): tag-eval admin card — trigger + persisted report (survives nav)
Frontend for #1130. A maintenance tile in Settings → Tagging: - Editable concept list + "Run eval" → POST /api/tag-eval (one running at a time). - Rehydrates on mount via the persisted run (getRun by latest id) and polls while running — so the report SURVIVES navigation (operator-flagged); the task runs backend-side regardless and the card reconnects to its row. - Renders the saved report: per-concept head-vs-centroid metrics table (AP/F1/ precision/recall) with Δ AP, the learning curve (AP @ N positives), and thumbnail galleries (head-would-suggest / head-doubts-positive) for eyeballing. Backend: _examples now stores thumbnail_urls (not just ids) so the report is a self-contained artifact that renders without per-id lookups on reload. No new top-level surface — slots into the existing maintenance area. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
958378312c |
fix(settings): sticky headers on the virtual data tables
Allowlist / Alias / ImportTask tables scroll their bodies (height=360/480) but the column headers scrolled away with the rows, so you lost the column labels (operator-flagged 2026-06-27). Add Vuetify `fixed-header` so the header row stays pinned while the body scrolls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e49cea3eba |
feat(tagging): allowlist tuning dashboard + post-accept toast + merge preview UI (#7c/#7d/#8b)
Cluster B frontend, milestone #99. #7c: AllowlistTable gains Applied + Covers columns and a live 'covers ~N at T' projection as the operator drags a row's threshold (debounced coverage call, then commits the threshold). allowlist store gains coverage(tagId, threshold) and refreshes coverage_count after a save. #7d: suggestions store surfaces a non-blocking toast when accept/alias newly allowlists a tag — '<verb>: <tag> — allowlisted, auto-applying to ~N images' (N is the projection; apply runs async). Falls back to the plain toast when the tag was already allowlisted. #8b: TagsView merge picker now previews the merge via usePreviewCommit before committing — shows images moving / already-on-target / series pages / alias-or- delete / a thumbnail sample, blocks the Merge button on an incompatible kind/fandom. adminStore.mergeTags gains a dryRun option. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX |
||
|
bvandeusen
|
7c94d99b9f |
refactor(settings): canonical usePreviewCommit for maintenance preview→commit tiles (#753)
Frontend pattern-consistency sweep (note #1026, the last DRY-thread item). TagMaintenanceCard (4 flows) + PostMaintenanceCard (2 flows) each hand-rolled the same sync preview→commit state machine: a previewData/previewing/committing triple + onPreview/onCommit that dry-run-previews, then applies and collapses the projection (the apply shares the backend predicate, so afterward it's empty). Extract usePreviewCommit({preview, commit, emptyPreview}) owning that lifecycle. The 6 flows become declarative: supply the two thunks + the collapse shape. The normalize flow (commit dispatches a self-resuming background task, not a sync apply) omits emptyPreview so the projection stays and a truthy result = queued. Composable returns are aliased to the cards' existing local names, so the templates only change where they read the apply result (the success badges). Long-Celery-task cards (GatedPurge/VideoDedup) keep useMaintenanceTask — a different pattern (navigable-away task lifecycle), deliberately not merged. Exhaustiveness: no card hand-rolls the refs anymore; the only dryRun:false callers are these two cards, both via the composable. Added a vitest spec for the primitive (collapse static + fn, dispatch-variant, re-preview clears result). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
311fe0ee9c |
feat(settings): tidy Maintenance tab into compact tiles + center the views (pass 2)
Goal (operator 2026-06-18): the overview of a Settings tab fits one unscrolled viewport; expanding a tile to read into it is the only reason to scroll. - Every Maintenance card converted to the collapsible MaintenanceTile (collapsed by default = icon + short title + one-line blurb). Task cards (ML backfill, centroids, thumbnails, archive re-extract, missing-file repair, DB maintenance) sit in a responsive grid; running tasks auto-expand. Tagging config (suggestion thresholds, allowlist, aliases) grouped in one Tagging section as collapsible tiles; Backup is its own collapsible tile. - Three labeled sections mirror the Cleanup tab: Backfills and reprocessing / Tagging / Storage. - Center the whole Settings surface: SettingsView is now a centered, width-capped (1140px) column so the tab strip and every panel sit in a tidy centered measure (was full-width). CleanupView drops its own left-aligned max-width to fill it. All card logic unchanged - only the chrome. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
3b435dc0ba |
feat(settings): tidy Cleanup tab into sectioned compact tiles (pass 1)
The Cleanup + Maintenance sections had ~17 full-width stacked cards with long descriptions — a hunt to scan. Operator wants compact, sectioned, scannable tiles (2026-06-18: keep both tabs, group inside, compact tiles in a grid). New common/MaintenanceTile.vue: a compact expandable tile (icon + short title + one-line blurb collapsed; click the header to expand the full controls/preview/ result inline; keyboard-accessible button + focus ring; tints the icon, keeps a running task expanded). Cleanup tab (this pass) restructured into 3 sections — Import-filter audits (Min dimensions, Transparency, Single-color) / Duplicates & posts (Bare posts, Duplicate posts, Deduplicate videos, Gated-post previews) / Tags (Unused, Legacy, Reset content tagging, Standardize casing) — each a responsive grid of tiles. PostMaintenanceCard split into 2 tiles, TagMaintenanceCard into 4. Moved VideoDedupCard + GatedPurgeCard from the Maintenance tab here (both are destructive content cleanup). All card logic unchanged — only the chrome. Maintenance tab tiling is pass 2 (TODO noted in MaintenancePanel). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
eff64275fc |
feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
e287802ecb |
fix(maint): resurface dedup/gated-purge results after navigate-away (#877)
Long-running maintenance tasks must survive navigating away or reloading
the page. VideoDedupCard + GatedPurgeCard held the in-flight Celery task id
only in component refs and polled task-result inline, so leaving the page
mid-run lost the id and the result was never shown — even though the task
finished on the worker.
New shared composable useMaintenanceTask: persists {taskId, mode, startedAt}
to localStorage on dispatch, re-attaches on mount, and re-shows the result
when the task finishes (the celery result backend retains the summary well
under result_expires). Stale-guard skips resume past 3h. Both cards refactored
onto it; card-specific computeds + confirm dialog kept.
Also fixed the QueueStatusBar lane: both cards watched queue="maintenance"
but tasks.admin.* routes to maintenance_long, so the bar never reflected
their own task — now queue="maintenance_long".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusen
|
540151290b |
feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
A one-shot Maintenance action to remove the blurred locked-preview images the ingester downloaded from tier-gated Patreon posts before #874. current_user_can_view was never persisted, so the cleanup re-walks each enabled Patreon source (read-only) to re-derive which posts are gated now and the blurred filehashes Patreon serves for them, then matches by CONTENT HASH against stored source_filehash. Because the hash is content-addressed, a real file downloaded when access existed has a different hash and can never match — regained-then-lost-access content is provably spared (operator's hard requirement). NULL source_filehash => unverifiable, kept + reported. On apply: delete matched ImageRecords + files (provenance cascades), clear seen/dead-letter ledger rows for those hashes so the real media re-ingests if access returns, and delete gated posts left bare. Shares one match predicate between preview and apply (rule 93). - cleanup_service: collect_gated_previews + purge_gated_previews - tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed) - api.admin: POST /maintenance/purge-gated-previews - GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply) - tests: collect predicate, hash-match delete/spare/unverifiable, ledger clear, bare-post removal, no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
369e3de684 |
feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)
Video tag noise root cause: frames were a FIXED count (6) max-pooled — a tag firing on one frame survived at peak confidence, and a fixed count under-samples long multi-scene videos so real scene-local tags looked like noise. Redesign (operator-steered): - Sample at a fixed CADENCE — one frame every `video_frame_interval_seconds` (default 4) across the 5–95% window — so a tag's frame-presence reflects real screen time independent of video length. Capped at `video_max_frames` (default 64): a long video stretches the spacing instead of exploding into hundreds of inferences, bounding per-video cost on the single ml-worker (per-frame ffmpeg timeout also cut 60s→30s). - Aggregate with `_aggregate_video_predictions`: keep a tag only if it appears in >= `video_min_tag_frames` sampled frames (≈ that many × interval seconds on screen — duration-independent noise rejection), with confidence = MEAN over the frames it appears in (not max). Clamps the threshold to the sample count so a 1–2-frame short video still tags. - All three knobs are DB-backed ml_settings (migration 0053), patchable via /api/ml/settings + sliders in the ML settings card — replaces the VIDEO_ML_FRAMES env var (product-not-project). Tests: aggregation drops one-frame noise + means corroborated tags + clamps on short videos; settings round-trip + min>max validation. Replaced the _maxpool_predictions unit test. NOTE: this is the QUALITY half of #747. The perf half — the ml-worker runs CPU-only — is GPU enablement, tracked separately in #872. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
41652db20f |
feat(maintenance): retroactive video-dedup action — preview + apply (#871)
Phase 2 of #871: clean up the duplicate videos already in the library (the #859 "same video from multiple sources" clutter). Import-time dedup (Phase 1) only prevents NEW dups; this is the operator-triggered cleanup of existing ones. cleanup_service.dedup_videos(dry_run): - backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the existing library participates; idempotent (only NULL rows), writes a negative sentinel for un-probeable files so they're neither re-probed forever nor matched. - find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect, anchored per cluster to bound the span (no chain drift); keeper = highest pixel area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances. - apply: re-point each loser's post links to the keeper (so no post loses the video) THEN delete the redundant records + files via delete_images (cascade). dry_run shares the same discovery predicate and returns the projection only (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated). - dedup_videos_task (maintenance queue; summary → task_run.metadata). - POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so the card shows the dry-run projection before the destructive apply. - VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind a confirm dialog. Mounted in the Maintenance panel. Tests: dedup collapses + re-links the loser's post to the keeper + removes the file; dry-run deletes nothing; distinct durations aren't grouped; task registered. (Migration 0052 for duration_seconds already shipped with Phase 1.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
949c9abcc6 |
fix(external): path-safe unlink + per-link staging + orphan repair (#859)
External downloads import IN PLACE, so the post-attach dedup-skip unlink could delete a file that IS an ImageRecord's backing file — orphaning the record and 404-ing on playback. Two sources of that: - Two links on the same post (same film from mega + gdrive) emitted the same filename into one external/<post_id>/ dir; the second overwrote the first. Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path. - The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it path-safe: only unlink when `f` is NOT the existing record's canonical file. Plus an operator-triggered orphan-repair maintenance task (prune_missing_file_records_task) to clean up records already orphaned by the bug: scans ImageRecords, deletes those whose file is gone (cascade), with an NFS-stall guard that aborts without deleting if a large sample is mostly missing. Wired through POST /api/admin/maintenance/prune-missing-files and a MissingFileRepairCard in the Maintenance panel. Tests: refetch-same-link keeps the canonical file; orphan repair deletes only real orphans and aborts on the mostly-missing guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusen
|
3610ba495f |
feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)
Read cutover verified in prod (suggestions + allowlist read image_prediction; backfill complete at 908k rows / 51k images). Removes the old JSON column and everything that fed it: - ImageRecord.tagger_predictions column removed; migration 0046 DROPs it. tagger_model_version kept as the "tagged / current?" signal the backfill sweep reads (needs-tagging check switched to tagger_model_version IS NULL). - tag_and_embed no longer dual-writes the JSON — image_prediction is the only write path. - importer re-import reset drops the JSON line (image_prediction rows are already deleted on re-import). - Retired the one-time #768 backfill task + the #764 prune task, their admin endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard). - Tests seed/assert via image_prediction; stale column refs removed. Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run `VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's the source of truth now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
65211a3f2f |
fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
The inline INSERT…SELECT backfill in migration 0045 wrapped the table creation and a ~100 GB pass over image_record.tagger_predictions in one transaction: nothing committed until the end, it was unmonitorable, and an earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A deploy got stuck on it for ~2h with image_prediction never appearing. Split the concerns: - 0045 now creates ONLY the table + indexes (instant DDL → web boots). - New backend.app.tasks.admin.backfill_image_predictions_task copies the >= store-floor predictions from the JSON into image_prediction, batched by id window and committed per chunk: live progress, resumable (re-enqueues from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each stays in the DB executor streaming each window — no Python-side 100 GB load, no materialization. - POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card to trigger the one-time run after upgrading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d55e52ae9b |
feat(admin): prune_low_confidence_predictions backfill task + UI (#764)
The one-time backfill that actually shrinks the DB: drops stored tagger_predictions entries below ml_settings.tagger_store_floor from every image_record row, and clamps any allowlist min_confidence below the floor up to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate so backfilled rows match new imports. Keyset by id ASC, idempotent, self-resumes on the soft time limit; runs on the maintenance_long lane. pg_dump copies live data only, so this alone fixes the #739 backup timeout — the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional disk-return step, brief because post-prune the live data is tiny. - admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions - PrunePredictionsCard in the Maintenance panel (shows the current floor) - tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c8b815afe6 |
feat(ml): clamp allowlist min_confidence to the tagger store floor
Consumer #4 of the store-floor change (#764). An allowlist tag can't auto-apply more permissively than the ingest floor — predictions below tagger_store_floor aren't stored, so a lower min_confidence behaves identically to the floor. update_threshold now clamps to max(value, floor); the AllowlistTable confidence input min-binds to the live floor and clamps on edit. Keeps the stored threshold honest about actual apply behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
3f92669f12 |
feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor
Promotes the prediction store-floor from the TAGGER_STORE_FLOOR env (default 0.05) to a DB-backed, Settings-UI-tunable ml_settings column (default 0.70). Storing every tag down to 0.05 from a ~10k-tag tagger is what grew image_record's TOAST to ~100 GB; the suggestion path already filters at 0.70 and the centroid/learned path covers lower-confidence preferred tags, so the sub-0.70 tail is redundant. Foundation for plan-task #764 (backfill + reclaim land next; this only changes the write gate for NEW imports). - ml_settings.tagger_store_floor (migration 0044, default 0.70) - tagger.Tagger.infer(store_floor=...); ml task passes settings.tagger_store_floor - ML admin GET/PATCH expose it; PATCH rejects a category suggestion threshold below the floor (nothing below the floor is stored, so the gap surfaces nothing) — server backstop for the UI slider clamp - Settings → ML: store-floor slider + caption; category sliders min-bound to it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
70d4017cf6 |
feat(activity): search/filter on both Activity-tab panes
Recent failures gains a client-side search over the already-loaded 24h rows (task/queue/target/error), shown as a filtered/total count alongside the existing error-type chips. All recent activity gains a debounced server-side task-name search (new `task` ILIKE param on /runs) so it spans the full history, not just the loaded page. LIKE wildcards are escaped so task names' literal underscores match literally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
9deebfa133 |
refactor(ui): CardHeading primitive for icon+title card/dialog headings (DRY pattern sweep)
The icon+title v-card-title heading (d-flex align-center + gap + <v-icon size=small> + <span>) was hand-rolled identically in 13 cards/dialogs (15 heading instances). Consolidate to <CardHeading icon title> (components/common) with an iconColor prop (error headings) and a default slot for trailing content (spacer+actions, inline status chip). Adopted everywhere the pattern appears — all-or-nothing per the hardened DRY process. Over-DRY guard: plain text-only <v-card-title> one-liners are NOT this pattern and stay; DownloadDetailModal leads with a status CHIP (not an icon), a different concept, left alone. §8b: the only remaining d-flex align-center v-card-title is that intentional variant. Catalog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4854d74c5a |
refactor(ui): SampleNameGrid primitive for maintenance-card previews (DRY pattern sweep)
The preview sample-name grid (scrollable monospace chip grid) was hand-rolled 5 times with verbatim-duplicated markup + CSS — TagMaintenanceCard (×4) and PostMaintenanceCard. Consolidate to <SampleNameGrid> (components/common): pass :names for the plain case, default slot for the normalize from→to chips (styled via :slotted .fc-name). Removed the duplicated .fc-name-grid/.fc-name CSS from both cards. Over-DRY guard: only the verbatim-duplicated grid is merged — each card's preview/commit logic and result-count lines genuinely differ and stay put; MinDimensionCard's typed-token confirm is a separate variant, untouched. §8b: fc-name-grid now lives only in SampleNameGrid. Catalog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
4e83b4225a |
refactor(ui): single global .fc-muted token (DRY pattern sweep)
The muted-text token was redefined identically in 12 component <style scoped> blocks. Consolidate to one global utility in styles/app.css; remove the 12 copies. Keeps the explicit on-surface-variant (vellum) token, NOT Vuetify's opacity-based text-medium-emphasis (per the muted-text-token rule). Behavior- preserving: every class=fc-muted usage now resolves to the single source. §8b exhaustiveness caught (and I fixed) my own sed clobbering the new app.css rule — now exactly one .fc-muted definition exists, zero component-local. Catalog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
c774042a85 |
refactor(ui): consolidate 7 hand-rolled kebabs into one KebabMenu (DRY pattern sweep)
First pattern-consistency DRY pass (process #594). The overflow kebab was hand-rolled 7 ways in two divergent activator strategies — Pattern A (#activator + v-bind) which silently breaks inside the teleported image modal (#711), and Pattern B (manual v-model + activator=parent + open-on-click=false + z-index 2400) the modal kebabs needed as a workaround. New <KebabMenu> (components/common) bakes in the modal-safe strategy UNIVERSALLY, so every kebab works in modal and non-modal contexts — folding the latent #711-class bug fix into all five Pattern-A sites. Menu items go in the default slot; variations (size/variant/location/label/min-width) are props. Adopted across all 7: TagChip, SuggestionItem, TagCard, SeriesView card, SeriesManageView, BackupRunsTable, SourceActions. Exhaustiveness (§8b): mdi-dots-vertical now lives only in KebabMenu. Labeled dropdowns / nav menus / filter popovers are a different concept and left alone. Seeded the pattern catalog so new code reuses the primitive. Test: KebabMenu renders slot items + trigger label/glyph + presentational props. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a8f624a0f1 |
fix(posts): link duplicate items to every post + prune bare shells
The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
daaa7543a8 |
fix(backup,tags): unwedge backups on NFS (#739) + tag-standardize "0 groups" (#740)
#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the concurrency-1 maintenance_long lane for hours — starving normalize_tags, re-extract, audits, and the new series rescan (which is why #740 "never applied"). Three fixes: - _run_bounded: Popen + bounded post-kill reap; if the child is unkillable (D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The orphan is reaped by the OS once its syscall clears. - backup_db dumps to a LOCAL temp file then moves the finished .sql to the (NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local writes (killable) instead of an NFS write that hangs. backup_images keeps bounded-kill (too big to stage locally). - recover_stalled_backup_runs: split the stall window — db 40 min (was sharing images' 7h), so a hung DB backup is flipped to error promptly. #740 — Standardize tag casing showed "0 groups to change" the instant it was clicked: onNormCommit overwrote the preview with zeros. Keep the real preview visible and disable the button while queued; backend apply was already correct. Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp target; per-kind stall sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
d9d502a60d |
fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
3c89223dcb |
feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714)
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps whatever casing it was created with. This adds a maintenance action that Title-Cases every existing tag (collapsing whitespace) and merges case/whitespace-variant duplicates into one. Backend: - tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor (prefer an already-canonical member → no rename/self-alias; else the best-connected tag → fewest FK repoints; else lowest id), merges the variants INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/ aliases/series_page repoints + protective ML aliases), then renames the survivor to canonical. Losers are deleted before the rename so there's no transient unique-index clash; commits per group and isolates failures per group. Idempotent — an already-canonical lone tag is a no-op. - normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i. - POST /api/admin/tags/normalize: dry_run=true returns a projection inline (group/collision/rename counts + sample); dry_run=false enqueues the task. Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup tab) — preview → apply (polls the activity dashboard to terminal status), behind a back-up-first warning. admin store gains normalizeTags(). Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept separate, ML-known loser keeps a protective alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a497104661 |
feat(maintenance): re-extract archive attachments + link to post — #713 part 2
Existing PostAttachments that are actually archives (filed opaquely before the magic-byte gate) need extracting retroactively. cleanup_service. reextract_archive_attachments scans PostAttachments, magic-detects the archives, and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place in a temp dir — so the members extract and re-link to the SAME post via find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by sha256). Enqueues thumbnail+ML for new members. Wired as a maintenance-queue Celery task (tasks/admin) + POST /api/admin/maintenance/reextract-archives (202) + a "Re-extract archive attachments" card in Settings → Maintenance. Test: a zip stored under a mangled extension-less name extracts + links its member to the post via ImageProvenance, and a second run is a no-op (idempotent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
a5b3702863 |
feat(settings): surface the near-duplicate (pHash) control + reorder import tab
The phash_threshold knob (controls whether edits/variants of an image are dropped as near-duplicates on import) was buried at the bottom of the import filters form and labelled opaquely, so it read as 'missing'. Hoist it to the TOP of the form as a 'Near-duplicate sensitivity' section: a labelled slider (Exact / Strict / Default / Loose stops, 0-16) for the gist + the precise number field, both bound to phash_threshold, with copy that says plainly to lower it if variants are being dropped. Also swap the import-tab order to filters → trigger → recent-tasks (filters on top per operator); the task list stays directly under the trigger for hit/miss feedback adjacency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
91b0145bc8 |
feat(tags): 'Reset content tagging' admin action
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
ef3ee5aceb |
feat(maintenance): DB maintenance UI card + fix ruff I001
- Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats. - dbMaintenance store (loadStats / runVacuum) + test. - Fix ruff I001: combine the two _sync_engine imports onto one line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
98673d4dca |
fix(audit-g5a): small architectural cleanups bundle
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.
- download_service: replace hardcoded ('discord','pixiv') tuple with
auth_type_for(platform) == 'token'. A 7th token-platform now picks
up the right credential path without touching this site.
- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
merge so the target's centroid reflects its new image set
immediately. Daily list_drifted catches it within 24h, but eager
recompute closes the suggestion-quality dip in the meantime.
- backfill_thumbnails added to beat_schedule (daily). The task
docstring claimed periodic Beat but the entry was never registered,
so the library got no self-healing thumbnail repair; only the
manual admin-UI button fired it.
- modal.createAndAdd pushes a kind='fandom' tag into
tagsStore.fandomCache so FandomPicker sees the new fandom on next
open. Was: cache-gated load (length===0) skipped refetch, new
fandom invisible until full page reload.
- cleanup cluster:
- Drop .webp from cleanup_service.unlink — thumbnailer only writes
.jpg/.png; the third tuple member was dead code.
- Drop effective_date from /api/gallery/scroll response — no FE
consumer reads it. Service still computes the attribute for
timeline ordering; this just trims the JSON.
- Rename store.recentMinute → store.recentRuns across the
systemActivity store + three consumers (SystemActivitySummary,
QueuesTable, SystemActivityTab). The data is the last 200 runs
(not actually "last minute"), so the name lied.
NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
|
||
|
bvandeusen
|
9cbdb70e13 |
fix(thumbnails): surface backfill results + tighten validity check
Two coupled problems, operator-flagged 2026-06-01: "missing thumbnails
but triggering backfill found nothing."
1. **Backfill UI was a black box.** `POST /api/thumbnails/backfill`
returned just `{celery_task_id}` and the admin card said
"Enqueued." with no counts. There was no way to tell whether the
scan found 0 candidates, 5000 candidates, or whether the worker
even picked up the task. "Found nothing" was indistinguishable
from a broken queue.
Fix: refactor the scan into a sync helper (`_run_backfill_scan`)
shared by the Celery task and the API endpoint. The API now runs
the scan in an executor and returns `{scanned, enqueued, ok,
regenerated}`. The actual thumbnail generation work still goes
to the thumbnail Celery queue per row via
`generate_thumbnail.delay()` — the scan itself is fast
(SELECT id+thumbnail_path + a file.stat() per row).
2. **`_thumb_is_valid` accepted header-only corrupt files.** The
magic-byte check passed for any 8-byte file starting with a JPEG
or PNG header, including empty/truncated/zero-pad files that
browsers render as broken. Backfill counted these as `ok` and
never regenerated.
Fix: also require file size ≥ MIN_THUMB_BYTES (256). Real
thumbnails are minimum ~2KB even on solid-color sources; header-
only corrupt files top out around 12 bytes. 256 is well above
the corrupt floor and well below any legitimate thumbnail.
Plus the admin card now shows the per-run counts instead of
"Enqueued.":
Scanned 5,432 · enqueued 3 (2 regenerated) · 5,429 ok
|
||
|
bvandeusen
|
af7b5c95e9 |
feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories
Four coupled operator-asked changes to the view modal (Scribe plan #509): 1. **Autofocus tag entry on modal open** — TagAutocomplete grabs focus in onMounted/nextTick so the caret is in the input the moment the modal renders. No click needed to start typing. 2. **General suggestions expanded by default** — SuggestionsPanel's general-category group now mounts with `:default-open="true"`. Operator can collapse if too noisy, but the v1 frame shows them. 3. **Lower general threshold default 0.95 → 0.50** — MLSettings. suggestion_threshold_general default matches character. Alembic 0029 also bumps the existing singleton row's value if it's still at the old 0.95. Operator can re-tune from Settings → ML. 4. **Retire `copyright` + `artist` as ML suggestion categories** — neither feeds a Tag.kind (`artist` retired in FC-2d-vii-c, never really existed as a copyright tag-kind). They were surfaced in the suggestions pipeline + threshold settings UI but had no follow- through. Drop from SURFACED_CATEGORIES, suggestions._threshold_for, ml_admin GET/PATCH allowlist, MLSettings columns (alembic 0029 drops the two columns), frontend CATEGORY_ORDER + CATEGORY_LABELS, SuggestionsPanel.peopleCats, AliasPickerDialog kind-check, and MLThresholdSliders rows. Out of scope (intentional): `tag_kind` Postgres enum still includes `artist` for historic Tag row queryability (per the model comment); no operator pain reported, no enum-shrink needed. Tests: - test_surfaced_categories asserts {character, general}, excludes artist + copyright. - test_threshold_for_artist_is_unsurfaced extended to cover copyright. - test_get_and_patch_settings asserts new 0.50 default and the absent artist + copyright keys in the GET payload. |