feat: ML tag suggestions, character/fandom integrity, underscores, modal polish
Consolidated merge of feat/tag-suggestions branch. Original 64-commit history was lost to git-object corruption in a Nextcloud-synced checkout; this single commit captures the equivalent diff. Includes: - pgvector-backed tag suggestion infra (WD14 + SigLIP centroids, ml-worker container, Celery tasks, suggestion service, accept/reject endpoints + modal UI with green/red chip buttons) - Character/fandom integrity: title-case normalization on every write path, fandom-id backfill, maintenance task + settings button, migrations g26041901 + h26041901 to canonicalize legacy rows with case-only duplicate merging - Tag-underscores + modal polish: WD14 name canonicalization at emit + accept + add/bulk-add paths, migration i26041901 for legacy-row rename-or-merge across character/fandom/NULL kinds, suggestion-accept refresh parity via awaited loadTags, persistent chip tint
This commit is contained in:
@@ -0,0 +1,318 @@
|
||||
# Clickable Post Tag Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** In the gallery/showcase image modal, move the system-generated `post` tag out of the editable tag list into a dedicated, visually distinct provenance section whose chip links to the tag-filtered gallery view.
|
||||
|
||||
**Architecture:** Frontend-only (Jinja template + vanilla JS + CSS). No backend or API changes — the existing `/image/<id>/tags` and `/api/post-metadata/by-tag-name/<name>` endpoints cover everything. Tags returned by the image-tags endpoint are partitioned client-side into editable vs. provenance lists and rendered into separate containers.
|
||||
|
||||
**Tech Stack:** Jinja2 templates, vanilla JavaScript, single CSS file (`app/static/style.css`)
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-04-18-clickable-post-tag-design.md`
|
||||
|
||||
**Testing:** This repo has no automated test harness for the frontend. Verification is a manual browser check at the end of the plan. Each earlier task ends with a commit; the browser check is gated to after all code is in place.
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
| File | Task(s) | What changes |
|
||||
|------|---------|--------------|
|
||||
| `app/templates/_gallery_modal.html` | 1 | New `modal-provenance-section` between Series and Tags sections |
|
||||
| `app/static/style.css` | 2 | `.modal-provenance-section`, `.provenance-chip`, platform accent selectors |
|
||||
| `app/static/js/view-modal.js` | 3 | Cache provenance DOM refs; partition tags in `loadTags`; add `renderProvenance`, `getProvenanceDisplayName`, `upgradeProvenanceChip`; clear provenance in `closeModal` |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Add provenance section markup to the modal
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/templates/_gallery_modal.html` (insert between line 48 and line 50)
|
||||
|
||||
### Step 1.1: Open the modal template
|
||||
|
||||
- [ ] Open `app/templates/_gallery_modal.html`.
|
||||
- [ ] Locate the end of the Series section — the `</div>` on line 48 that closes `modal-series-section`, immediately followed by `<!-- SECTION: Tags -->` on line 50.
|
||||
|
||||
### Step 1.2: Insert the provenance section between Series and Tags
|
||||
|
||||
- [ ] Insert this block on a new line between the closing `</div>` of the Series section and the `<!-- SECTION: Tags -->` comment:
|
||||
|
||||
```html
|
||||
<!-- SECTION: Provenance (system-generated tags: post; future: source, archive) -->
|
||||
<div class="modal-sidebar-section modal-provenance-section" id="modalProvenanceSection" style="display: none;">
|
||||
<div id="modalProvenanceList" class="provenance-list"></div>
|
||||
</div>
|
||||
|
||||
```
|
||||
|
||||
The `style="display: none;"` matches the inline-style pattern already used by `modalSeriesInfo` and `modalAddToSeries` in this same file (per the project's "inline style over CSS class toggling" convention from commit `e37ce27`).
|
||||
|
||||
### Step 1.3: Verify the insertion
|
||||
|
||||
- [ ] Run: `grep -n "modalProvenanceSection\|modal-tags-section" app/templates/_gallery_modal.html`
|
||||
- [ ] Expected: `modalProvenanceSection` appears on an earlier line than `modal-tags-section`.
|
||||
|
||||
### Step 1.4: Commit
|
||||
|
||||
```bash
|
||||
git add app/templates/_gallery_modal.html
|
||||
git commit -m "add empty provenance section to gallery modal sidebar"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Add CSS for provenance section and chip
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/static/style.css` (append new block)
|
||||
|
||||
### Step 2.1: Find the end of the modal sidebar / tag chip section in the stylesheet
|
||||
|
||||
- [ ] Open `app/static/style.css`.
|
||||
- [ ] Run: `grep -n "Tag chips in modal - with remove button" app/static/style.css`
|
||||
- [ ] Navigate to that block (around line 816). Scroll to the end of the rule block that styles `.tag-editor .tag-chip` — the last closing `}` of that rule group.
|
||||
|
||||
### Step 2.2: Append the provenance CSS
|
||||
|
||||
- [ ] Immediately after that closing `}`, append:
|
||||
|
||||
```css
|
||||
|
||||
/* Modal provenance section — system-generated origin tags (post; future: source/archive) */
|
||||
.modal-provenance-section .provenance-list {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
|
||||
.provenance-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.4rem;
|
||||
padding: 2px 8px 2px 10px;
|
||||
border-radius: 999px;
|
||||
background: transparent;
|
||||
color: var(--text-dim);
|
||||
text-decoration: none;
|
||||
border: 1px solid rgba(255, 255, 255, 0.15);
|
||||
border-left-width: 2px;
|
||||
border-left-color: rgba(255, 255, 255, 0.4);
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
font: 12px/1.6 system-ui, sans-serif;
|
||||
max-width: clamp(8rem, 22vw, 14rem);
|
||||
transition: border-color 0.15s ease, background 0.15s ease, color 0.15s ease;
|
||||
}
|
||||
|
||||
.provenance-chip:hover {
|
||||
border-color: rgba(255, 255, 255, 0.35);
|
||||
background: rgba(255, 255, 255, 0.04);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.provenance-chip[data-platform="patreon"] {
|
||||
border-left-color: #f96854;
|
||||
}
|
||||
.provenance-chip[data-platform="subscribestar"] {
|
||||
border-left-color: #009cde;
|
||||
}
|
||||
.provenance-chip[data-platform="hentaifoundry"] {
|
||||
border-left-color: #c8232c;
|
||||
}
|
||||
|
||||
.provenance-chip .provenance-arrow {
|
||||
opacity: 0.6;
|
||||
font-size: 0.85em;
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2.3: Commit
|
||||
|
||||
```bash
|
||||
git add app/static/style.css
|
||||
git commit -m "style provenance chip with platform-tinted ghost treatment"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Partition tags and render the provenance chip in view-modal.js
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/static/js/view-modal.js`
|
||||
|
||||
This task has several edits inside one file. Make them in order; commit once at the end.
|
||||
|
||||
### Step 3.1: Cache the new DOM references
|
||||
|
||||
- [ ] Open `app/static/js/view-modal.js`.
|
||||
- [ ] Find the block that caches tag editor elements (around lines 11-16), beginning with `const tagEditor = document.getElementById('modalTagEditor');`.
|
||||
- [ ] Immediately after the line `const tagActionFeedback = document.getElementById('tagActionFeedback');`, insert:
|
||||
|
||||
```javascript
|
||||
const provenanceSection = document.getElementById('modalProvenanceSection');
|
||||
const provenanceList = document.getElementById('modalProvenanceList');
|
||||
```
|
||||
|
||||
### Step 3.2: Add the provenance display-name helper
|
||||
|
||||
- [ ] Find the `renderTags` function (starts around line 85).
|
||||
- [ ] Immediately **above** `function renderTags(tags) {`, insert this helper:
|
||||
|
||||
```javascript
|
||||
function getProvenanceDisplayName(name) {
|
||||
// Post tag names are "post:platform:artist:id". Strip the "post:" prefix only.
|
||||
if (name.startsWith('post:')) return name.slice(5);
|
||||
return name;
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### Step 3.3: Add renderProvenance and upgradeProvenanceChip
|
||||
|
||||
- [ ] Directly **below** the existing `renderTags` function (after its closing `}` around line 96), insert:
|
||||
|
||||
```javascript
|
||||
|
||||
function renderProvenance(tags) {
|
||||
if (!provenanceSection || !provenanceList) return;
|
||||
provenanceList.innerHTML = '';
|
||||
if (!tags || tags.length === 0) {
|
||||
provenanceSection.style.display = 'none';
|
||||
return;
|
||||
}
|
||||
tags.forEach(t => {
|
||||
const chip = document.createElement('a');
|
||||
chip.className = 'provenance-chip';
|
||||
chip.href = `/gallery?tag=${encodeURIComponent(t.name)}`;
|
||||
chip.title = t.name;
|
||||
chip.dataset.tagName = t.name;
|
||||
const placeholder = escapeHtml(getProvenanceDisplayName(t.name));
|
||||
chip.innerHTML = `📌 <span class="provenance-label">${placeholder}</span> <span class="provenance-arrow">↗</span>`;
|
||||
provenanceList.appendChild(chip);
|
||||
upgradeProvenanceChip(chip, t.name);
|
||||
});
|
||||
provenanceSection.style.display = '';
|
||||
}
|
||||
|
||||
async function upgradeProvenanceChip(chip, tagName) {
|
||||
try {
|
||||
const r = await fetch(`/api/post-metadata/by-tag-name/${encodeURIComponent(tagName)}`);
|
||||
const j = await r.json();
|
||||
if (!j.ok || !j.metadata) return;
|
||||
const pm = j.metadata;
|
||||
if (pm.platform) chip.dataset.platform = pm.platform;
|
||||
const label = pm.artist || pm.title;
|
||||
if (label) {
|
||||
const labelEl = chip.querySelector('.provenance-label');
|
||||
if (labelEl) labelEl.textContent = label;
|
||||
}
|
||||
} catch {
|
||||
// Fetch failed — chip keeps its placeholder label and neutral accent.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3.4: Partition tags in loadTags
|
||||
|
||||
- [ ] Find the current `loadTags` function (around lines 97-106):
|
||||
|
||||
```javascript
|
||||
async function loadTags(imageId) {
|
||||
if (!imageId) { renderTags([]); return; }
|
||||
try {
|
||||
const r = await fetch(`/image/${imageId}/tags`);
|
||||
const j = await r.json();
|
||||
if (j.ok) renderTags(j.tags);
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] Replace it with:
|
||||
|
||||
```javascript
|
||||
async function loadTags(imageId) {
|
||||
if (!imageId) { renderTags([]); renderProvenance([]); return; }
|
||||
try {
|
||||
const r = await fetch(`/image/${imageId}/tags`);
|
||||
const j = await r.json();
|
||||
if (j.ok) {
|
||||
const all = j.tags || [];
|
||||
const postTags = all.filter(t => t.kind === 'post');
|
||||
const editableTags = all.filter(t => t.kind !== 'post');
|
||||
renderTags(editableTags);
|
||||
renderProvenance(postTags);
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3.5: Clear provenance on modal close
|
||||
|
||||
- [ ] Find the `closeModal` function (around lines 615-634).
|
||||
- [ ] Locate the line `renderTags([]);` inside it (around line 625).
|
||||
- [ ] Immediately **after** that line, add:
|
||||
|
||||
```javascript
|
||||
renderProvenance([]);
|
||||
```
|
||||
|
||||
### Step 3.6: Commit
|
||||
|
||||
```bash
|
||||
git add app/static/js/view-modal.js
|
||||
git commit -m "split post tags into provenance section in image modal"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Manual browser verification
|
||||
|
||||
**No code changes. This is a gate before declaring the feature done.**
|
||||
|
||||
### Step 4.1: Start the app
|
||||
|
||||
- [ ] Follow whatever start procedure this project uses locally (e.g., `docker compose up` or `python run.py` — consult `README.md` / `entrypoint.sh` if unsure).
|
||||
- [ ] Open the showcase in a browser.
|
||||
|
||||
### Step 4.2: Verify with an image that has a post tag
|
||||
|
||||
- [ ] In the Tags nav, pick any post-backed image — or open the showcase and click any image known to come from an imported post (Patreon / SubscribeStar / HentaiFoundry).
|
||||
- [ ] Confirm:
|
||||
- A new chip-only section appears in the modal sidebar, **above** the editable tag list and **below** the Series section.
|
||||
- The chip displays 📌, a text label, and a trailing ↗.
|
||||
- The chip's **left border** is tinted with the platform color (orange / teal / red).
|
||||
- The label starts as the raw `platform:artist:id` placeholder, then updates to the artist (or title, if no artist) once the platform metadata request resolves.
|
||||
- The post tag is **not** present in the editable tag list below.
|
||||
- Hovering the chip brightens the border and background.
|
||||
- Clicking the chip navigates to `/gallery?tag=<post tag name>` and the gallery page shows the post's title/description/source URL block.
|
||||
|
||||
### Step 4.3: Verify with an image that has NO post tag
|
||||
|
||||
- [ ] Open an image with no post tag (e.g., a manually uploaded image without import metadata).
|
||||
- [ ] Confirm:
|
||||
- The provenance section is **not visible** — no empty heading, no empty chip row, no stray divider.
|
||||
- The editable tag list still renders unchanged.
|
||||
|
||||
### Step 4.4: Verify the metadata-fetch failure path (optional smoke test)
|
||||
|
||||
- [ ] In DevTools, block requests to `/api/post-metadata/by-tag-name/*` and reopen the modal on a post-backed image.
|
||||
- [ ] Confirm the chip renders with the neutral-grey left border (no platform tint) and the raw `platform:artist:id` label, and clicking it still navigates to the filtered gallery.
|
||||
|
||||
### Step 4.5: Navigate between images with arrow keys
|
||||
|
||||
- [ ] From a post-backed image modal, press the right-arrow key to advance through several images of mixed types (with and without post tags).
|
||||
- [ ] Confirm the provenance section shows and hides correctly per image, with no stale chip left over from the previous image.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- Spec coverage: placement between Series and Tags ✓ (Task 1), hidden when empty ✓ (Task 3.3, 3.4, 3.5), platform-tinted ghost chip ✓ (Task 2), click → filtered gallery ✓ (Task 3.3), partition so post tag is removed from editable list ✓ (Task 3.4), placeholder + metadata upgrade ✓ (Task 3.3), fallback accent on fetch failure ✓ (Task 4.4 verifies).
|
||||
- No placeholders in the plan: every code step shows the full snippet to insert; no "similar to" references.
|
||||
- Type consistency: `renderProvenance`, `upgradeProvenanceChip`, `getProvenanceDisplayName` are named consistently across steps; DOM ids (`modalProvenanceSection`, `modalProvenanceList`) match between template and JS.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,100 @@
|
||||
# Clickable Post Tag in Gallery Modal
|
||||
|
||||
**Status:** Approved, ready for implementation planning
|
||||
**Date:** 2026-04-18
|
||||
**Scope:** Gallery/showcase image view modal (`_gallery_modal.html`)
|
||||
|
||||
## Problem
|
||||
|
||||
When viewing an image in the gallery/showcase modal, the user frequently wants to revisit the original post (Patreon/SubscribeStar/HentaiFoundry) the image came from, to re-read context. Today the post tag is rendered as a plain, non-clickable `<span>` in the editable tag list — there is no path from the modal to the post.
|
||||
|
||||
Secondary concern: the post tag is system-generated and shouldn't appear inside the editable tag area, where the `×` remove button invites deletion of metadata the user didn't author.
|
||||
|
||||
## Goal
|
||||
|
||||
Make the post tag in the image modal a clickable chip that navigates to the gallery filtered by that post tag. The filtered gallery view already surfaces the post's title, description, and source URL (`main.py:143-157`), so click-to-filter is the shortest path to the original post.
|
||||
|
||||
Move the post tag out of the editable tag list into a dedicated "provenance" area so it reads as metadata, not a user tag.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Opening the external post URL directly from the modal. Navigation goes to the filtered gallery, which is where `source_url` is already presented.
|
||||
- Making `source` (🌐) or `archive` (🗜️) tags clickable in the modal. Will reconsider after the post-tag treatment is validated.
|
||||
- Changes to `tag-editor.js` / the tags-list page's edit modal. Different modal.
|
||||
- New backend endpoints.
|
||||
|
||||
## Design
|
||||
|
||||
### Placement
|
||||
|
||||
A new section `modal-provenance-section` is inserted into the modal sidebar **between** the existing Series section and Tags section.
|
||||
|
||||
- Section name is forward-looking: "provenance" accommodates source/archive later without renaming.
|
||||
- The section is rendered only when the current image has at least one post tag. When empty, it is fully hidden — no placeholder, no heading.
|
||||
|
||||
### Chip content
|
||||
|
||||
One chip per post tag (in practice zero or one). Chip structure:
|
||||
|
||||
- 📌 icon prefix
|
||||
- Display label — see rules below
|
||||
- Trailing `↗` affordance to signal the chip navigates away from the modal
|
||||
|
||||
**Display label rules.** `getTagDisplayName` today only strips prefixes for `artist|character|series|fandom|rating`; post tags fall through and yield `platform:artist:id`, which is ugly. Label resolution for the provenance chip:
|
||||
|
||||
1. **Before metadata resolves:** show the raw tag name split once on `:` (the `platform:artist:id` portion). Ugly but functional placeholder.
|
||||
2. **After metadata resolves:** prefer `pm.artist`; if absent, fall back to `pm.title`; if both absent, keep the placeholder.
|
||||
|
||||
The upgrade from placeholder to metadata-driven label happens in the same render pass that applies the platform accent color.
|
||||
|
||||
### Chip behavior
|
||||
|
||||
- Click target: the whole chip, as an `<a href="/gallery?tag=<encoded post tag name>">`
|
||||
- Navigation: same tab (consistent with other tag chips in the app)
|
||||
- No `×` remove button — system tag, not user-editable
|
||||
|
||||
### Chip styling — platform-tinted ghost chip
|
||||
|
||||
Visually distinct from the standard editable tag chip (which is light-on-light with a `×` on hover). The provenance chip is a ghost chip with a platform-colored accent.
|
||||
|
||||
- **Base:** transparent background, 1px border `rgba(255, 255, 255, 0.15)`, light text
|
||||
- **Platform accent:** a 2px colored left-border driven by `PostMetadata.platform`:
|
||||
- `patreon` → Patreon orange (`#f96854`)
|
||||
- `subscribestar` → teal (`#009cde`)
|
||||
- `hentaifoundry` → red (`#c8232c`)
|
||||
- Unknown / missing metadata → neutral fallback, `rgba(255, 255, 255, 0.4)`
|
||||
- **Hover:** border brightens to `rgba(255, 255, 255, 0.35)`, background `rgba(255, 255, 255, 0.04)`
|
||||
- Platform color is applied via a `data-platform="<platform>"` attribute on the chip plus CSS attribute selectors — no inline styles.
|
||||
|
||||
### Data flow
|
||||
|
||||
1. `view-modal.js` already calls `loadTags(imageId)` when the modal opens and when navigating between images.
|
||||
2. Extend the tag load: partition the returned tags into
|
||||
- `postTags` — `tag.kind === 'post'`
|
||||
- `editableTags` — everything else
|
||||
3. Render `editableTags` into the existing `#modalTagList` container (unchanged behavior, minus the post chip).
|
||||
4. Render `postTags` into a new `#modalProvenanceList` container inside `modal-provenance-section`. Show the section iff `postTags.length > 0`; hide it otherwise.
|
||||
5. For each post tag, resolve its platform by calling the existing endpoint `/api/post-metadata/by-tag-name/<tag_name>`, which returns `{ platform, source_url, ... }`. The chip renders immediately with the neutral fallback accent, then upgrades to the platform-tinted accent when metadata resolves. This keeps the modal responsive even on a slow metadata fetch.
|
||||
6. Clean up provenance rendering on modal close and on image navigation, the same way the tag list is cleaned up today.
|
||||
|
||||
### Components to touch
|
||||
|
||||
- `app/templates/_gallery_modal.html` — add the `modal-provenance-section` with a `#modalProvenanceList` container between the Series and Tags sections.
|
||||
- `app/static/js/view-modal.js` — partition tags on load, render provenance chips, fetch platform metadata per post tag, handle show/hide of the section, reset on close/navigate.
|
||||
- `app/static/style.css` — `.modal-provenance-section`, `.provenance-chip`, platform-specific `[data-platform="..."]` accent rules, hover state.
|
||||
|
||||
### Error handling
|
||||
|
||||
- Metadata endpoint fails, returns 404, or returns no metadata record → chip renders with neutral fallback accent. The chip remains clickable; the filter-by-tag navigation works without platform data.
|
||||
- Post tag exists but its name is malformed → chip still renders with `getTagDisplayName` output and fallback accent; click still navigates to `/gallery?tag=<raw name>`.
|
||||
|
||||
## Risks and considerations
|
||||
|
||||
- Tag-kind drift: if another part of the codebase ever assumes every tag in the modal is user-editable (has a `×` button), splitting the list could surprise it. Mitigation: grep for `#modalTagList` consumers and confirm only view-modal.js reads it.
|
||||
- Duplicate clicks during metadata fetch: chip is clickable before metadata arrives — fine, because click navigation doesn't depend on metadata.
|
||||
- Future extension to source/archive: section and class names are generic (`provenance`), so adding them later is additive and doesn't require renames.
|
||||
|
||||
## Future work (deferred)
|
||||
|
||||
- Consider moving `source` (🌐) and `archive` (🗜️) tags into the same provenance section once the post-tag treatment is polished.
|
||||
- Consider a direct "open original post" action (using `source_url`) as a secondary affordance — only if click-to-filter proves insufficient.
|
||||
@@ -0,0 +1,252 @@
|
||||
# Tag Suggestions — ImageRepo Integration Spec
|
||||
|
||||
**Parent spec:** `docs/superpowers/tag-suggestion-system-spec.md`
|
||||
**Status:** Design approved 2026-04-18. Ready for implementation planning.
|
||||
**Scope this phase:** Scan pipeline + modal suggestion UI. Multi-edit / gallery bulk suggestions deferred.
|
||||
|
||||
This document layers ImageRepo-specific integration decisions on top of the parent spec. Read the parent spec first for model choice, schema rationale, threshold config, and model versioning; this spec assumes all of that is still in effect.
|
||||
|
||||
---
|
||||
|
||||
## 1. System layout
|
||||
|
||||
A new Docker Swarm service joins the existing stack:
|
||||
|
||||
- `ml-worker` — Celery worker on a new `ml` queue. Shares the existing Redis broker and Postgres database. Runs **CPU inference only** (Swarm does not expose GPUs cleanly and this is a single-user system; per-upload latency is async, backfill runs once overnight).
|
||||
|
||||
Existing services are unchanged. The `worker` container stays on its current queues (`import`, `thumbnail`, `sidecar`, `default`); the `scheduler` keeps `maintenance` and `scan`.
|
||||
|
||||
### Image
|
||||
|
||||
`ml-worker` uses its own Dockerfile (`Dockerfile.ml` or similar) so the main `worker` image stays lean:
|
||||
|
||||
- Base: `python:3.12-slim`
|
||||
- System deps: `libgl1`, `libglib2.0-0` (for Pillow / OpenCV if used for preprocessing)
|
||||
- Python deps (new `requirements-ml.txt`): `onnxruntime` (CPU build), `numpy`, `Pillow`, `huggingface_hub` (to pull models at build time), plus everything in the base `requirements.txt` that the Celery app imports
|
||||
- Models baked into the image at build time under `/models/` (WD14 ONNX + SigLIP-SO400M ONNX). Fetching at build time avoids cold-start latency and network dependency at runtime. Image size will be ~2-3 GB; acceptable for a persistent service.
|
||||
|
||||
### Resource sizing
|
||||
|
||||
- CPU: at least 4 cores pinned via `deploy.resources.limits.cpus`
|
||||
- RAM: 6-8 GB (ONNX graph + model weights + image batch preprocessing headroom)
|
||||
- Celery concurrency: **1**. ONNX Runtime will use multiple CPU threads internally via `intra_op_num_threads`; multi-worker concurrency just fights for the same cores.
|
||||
|
||||
---
|
||||
|
||||
## 2. Database
|
||||
|
||||
Follows the parent spec's schema, adapted to ImageRepo conventions:
|
||||
|
||||
- Primary keys use `Integer`, not `BIGSERIAL` (consistent with `ImageRecord.id`, `Tag.id`).
|
||||
- FKs target the actual ImageRepo table names: `image_record.id`, `tag.id`. The parent spec's `images(id)` references become `image_record(id)`.
|
||||
- Table names stay singular per ImageRepo convention: `image_tag_prediction`, `image_embedding`, `character_reference_embedding`, `suggestion_feedback`, `tag_suggestion_config`.
|
||||
- `tag_suggestion_config` could be folded into the existing `AppSettings` key/value table, but the parent spec's dedicated table keeps threshold tuning separate from general settings and makes the thresholds discoverable. Stick with the dedicated table.
|
||||
|
||||
### Alembic migration
|
||||
|
||||
A new revision in `migrations/versions/` does:
|
||||
|
||||
1. `CREATE EXTENSION IF NOT EXISTS vector;`
|
||||
2. Create the five new tables with SQLAlchemy-`pgvector` column types.
|
||||
3. Create the HNSW index on `image_embedding.embedding` using `vector_cosine_ops`.
|
||||
4. Seed `tag_suggestion_config` with the initial threshold rows from the parent spec.
|
||||
|
||||
The `pgvector` Python package must be added to the base `requirements.txt` (not just ml-worker's) so the web process can read embeddings for suggestion generation.
|
||||
|
||||
### Embedding dimension
|
||||
|
||||
Locked at **1152** (SigLIP-SO400M). If the model is swapped later, a new `model_version` coexists in the same tables per the parent spec; the schema does not change.
|
||||
|
||||
---
|
||||
|
||||
## 3. Task flow
|
||||
|
||||
New Celery module `app/tasks/ml.py`, tasks routed to the `ml` queue.
|
||||
|
||||
### Per-upload: chained from the import pipeline
|
||||
|
||||
`app.tasks.import_file.import_media_file` currently ends by committing an `ImageRecord` row. After commit, it enqueues:
|
||||
|
||||
```python
|
||||
from app.tasks.ml import tag_and_embed
|
||||
tag_and_embed.apply_async(args=[image_id], queue='ml')
|
||||
```
|
||||
|
||||
`tag_and_embed(image_id)`:
|
||||
1. Load the image from disk (or from the path stored on `ImageRecord.filepath`)
|
||||
2. Run WD14 preprocessing → WD14 inference → write rows into `image_tag_prediction`
|
||||
3. Run SigLIP preprocessing → SigLIP inference → write row into `image_embedding`
|
||||
4. Commit both in one DB transaction
|
||||
|
||||
If either model fails the task raises and Celery will retry per its default policy. A failure does not block the upload — the image is already in the DB and visible in the gallery; suggestions simply will not appear for it until the task succeeds.
|
||||
|
||||
### Backfill: resumable batch job
|
||||
|
||||
`app.tasks.ml.backfill(batch_size=50)`:
|
||||
|
||||
1. Query images missing predictions **or** embeddings for the current model versions
|
||||
2. For each, enqueue `tag_and_embed.apply_async(args=[image_id], queue='ml')`
|
||||
3. Process in batches so the enqueue side doesn't dump 62k messages into Redis at once — sleep/yield between batches
|
||||
4. Track progress via a row in the existing `ImportBatch` / `Task` pattern (whichever is conventional for long-running scan jobs — follow `scan_directory` for precedent)
|
||||
|
||||
Resumability comes from the query itself: it only enqueues images without current rows. Re-running after a crash picks up exactly where it left off.
|
||||
|
||||
An admin trigger for `backfill` is exposed via a new button on the Settings → Maintenance tab (the tab already exists from the UI/UX redesign). No automated scheduling — the user kicks it off manually once after deploy, and again after a model version change.
|
||||
|
||||
### On accept: centroid recompute
|
||||
|
||||
When a user accepts a `character`-kind suggestion on an image:
|
||||
|
||||
1. Apply the tag as normal (existing tag-add endpoint flow)
|
||||
2. Write a `suggestion_feedback` row with `decision='accepted'`
|
||||
3. Check `character_reference_embedding.reference_count` for this tag. If the current count of confirmed images minus `reference_count` at last computation exceeds `centroid_recompute_delta` (default 3), enqueue `recompute_centroid.apply_async(args=[character_tag_name], queue='ml')`
|
||||
|
||||
`recompute_centroid(tag_name)` fetches all embeddings for images that currently carry that character tag, computes the mean vector, upserts `character_reference_embedding` with the new centroid and the current reference count.
|
||||
|
||||
This runs on the `ml` queue because it touches the pgvector tables; it does **not** need the ONNX models, so it completes in milliseconds.
|
||||
|
||||
### On reject: feedback only
|
||||
|
||||
Rejecting a suggestion writes a `suggestion_feedback` row with `decision='rejected'`. No other side effect. The rejected tag will reappear on the next modal open — intentional, per the parent spec's rationale that feedback is informational and we should not lose tags the user might later want.
|
||||
|
||||
---
|
||||
|
||||
## 4. Web integration
|
||||
|
||||
### Flask endpoints
|
||||
|
||||
All under the existing image blueprint. Permissions inherit from however the modal's existing `/image/<id>/tags` endpoints gate access.
|
||||
|
||||
**`GET /image/<id>/suggestions`**
|
||||
|
||||
Returns the merged suggestion list grouped by category. Body:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"suggestions": {
|
||||
"character": [
|
||||
{"name": "<tag>", "confidence": 0.91, "source": "embedding_similarity", "exists_in_db": true}
|
||||
],
|
||||
"copyright": [ ... ],
|
||||
"rating": [ ... ],
|
||||
"general": [ ... ]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Suggestion generation logic lives in a new `app/services/tag_suggestions.py` module and is invoked by the route. It:
|
||||
|
||||
1. Fetches this image's `image_tag_prediction` rows for the current WD14 model version, filters by per-category thresholds from `tag_suggestion_config`, drops tags already applied to the image.
|
||||
2. Fetches this image's `image_embedding` for the current SigLIP model version, computes cosine similarity against every `character_reference_embedding` whose `reference_count >= min_reference_images_per_character`, keeps matches above `threshold_embedding_character`, drops characters already applied.
|
||||
3. Merges: WD14-character and embedding-character lists dedupe by tag name; keep the higher normalized confidence and record both sources if desired (for now, single `source` is fine).
|
||||
4. Annotates each suggestion with `exists_in_db: true|false` by looking up each name in the `tag` table.
|
||||
5. Groups by category, sorts by confidence descending within each group.
|
||||
|
||||
Categories returned in the response map 1:1 to WD14's categories. `meta` is omitted from the modal for now (noisy, low value).
|
||||
|
||||
**`POST /image/<id>/suggestions/accept`**
|
||||
|
||||
Body: `{"tag_name": "...", "source": "wd14"|"embedding_similarity", "category": "general"|"character"|"copyright"|"rating"}`.
|
||||
|
||||
Server applies the hybrid vocabulary policy:
|
||||
|
||||
- `character`, `copyright`, `rating` → if the tag doesn't exist in the `tag` table, create it with the appropriate `kind`. Then associate with the image. Kind mapping:
|
||||
- WD14 `character` → `kind='character'`
|
||||
- WD14 `copyright` → `kind='fandom'` (ImageRepo's IP/franchise kind — series is reserved for sequential readers)
|
||||
- WD14 `rating` → `kind='rating'`
|
||||
- `general` → if the tag exists, associate with the image. If not, return `{"ok": false, "error": "unknown_general_tag"}` (the UI renders this branch as a disabled chip in advance; this error is the safety net for race conditions).
|
||||
|
||||
After applying, writes `suggestion_feedback` with `decision='accepted'` and triggers centroid recompute if the accepted tag is a character.
|
||||
|
||||
**`POST /image/<id>/suggestions/reject`**
|
||||
|
||||
Body: `{"tag_name": "...", "source": "..."}`. Writes `suggestion_feedback` with `decision='rejected'`. Returns `{"ok": true}`.
|
||||
|
||||
### Modal UI
|
||||
|
||||
New section in the gallery modal sidebar (`app/templates/_gallery_modal.html`) **below** the existing Tags section, with id `modalSuggestionsSection` and a `display: none` inline style that JavaScript toggles (matches the project's inline-style-over-class convention per commit `e37ce27`).
|
||||
|
||||
Structure:
|
||||
|
||||
```html
|
||||
<div class="modal-sidebar-section modal-suggestions-section" id="modalSuggestionsSection" style="display: none;">
|
||||
<div class="section-header">
|
||||
<span class="section-title">Suggestions</span>
|
||||
<button class="suggestions-refresh" type="button" aria-label="Refresh suggestions">↻</button>
|
||||
</div>
|
||||
<div id="modalSuggestionsList" class="suggestions-list"></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
Frontend logic goes in `app/static/js/view-modal.js` next to the existing `loadTags` pattern. A new `loadSuggestions(imageId)` fetches `/image/<id>/suggestions`, renders a flat DOM structure with subtle category-heading dividers between the groups, and wires up the accept / reject click handlers.
|
||||
|
||||
**Chip rendering:**
|
||||
|
||||
- Layout: `[+ label · XX%]` with a hover-revealed `✕` on the right.
|
||||
- Accept click: disable chip, `POST .../accept`, on success animate-out and call the existing `refreshItemTags(imageId)` hook so the gallery thumbnail overlay updates. Also append the new tag to the editable tag list in the modal.
|
||||
- Reject click: disable chip, `POST .../reject`, animate-out. No other side effect.
|
||||
- `exists_in_db: false` and policy says general-must-exist: render the chip dimmed, accept button disabled, tooltip "create this tag first in Tags admin."
|
||||
- Platform-style accents: not needed here. Keep chips visually distinct from the editable tags (maybe a dashed border or slightly lower opacity) so users don't confuse suggestions with applied tags.
|
||||
|
||||
**Feedback:**
|
||||
|
||||
Reuse the `tagActionFeedback` element the modal already owns for "applied" / "rejected" / error messages. No toast system.
|
||||
|
||||
**Lifecycle:**
|
||||
|
||||
- Open modal → `loadSuggestions(imageId)` called alongside `loadTags` / `renderProvenance`.
|
||||
- Close modal → clear the suggestions list (the existing `closeModal` hook).
|
||||
- Arrow-key navigation between images re-runs `loadSuggestions` per image.
|
||||
|
||||
---
|
||||
|
||||
## 5. Tag vocabulary policy (hybrid)
|
||||
|
||||
Confirmed from brainstorming:
|
||||
|
||||
- **Character / copyright / rating** suggestions → auto-create the tag on accept if it doesn't exist in the DB. The `tag.kind` is set as follows:
|
||||
- WD14 `character` → `kind='character'`
|
||||
- WD14 `copyright` → `kind='fandom'` (ImageRepo's IP/franchise kind — `series` is reserved for sequential readers, not franchises)
|
||||
- WD14 `rating` → `kind='rating'`
|
||||
- **General** suggestions → filter to tags that already exist in the DB. Unknown general tags are rendered dimmed (disabled) in the UI; the accept endpoint rejects them as a safety net.
|
||||
- **Embedding-similarity** suggestions are always characters; they suggest only characters that already have a reference set, so vocabulary never grows via that path.
|
||||
|
||||
---
|
||||
|
||||
## 6. Scope boundaries
|
||||
|
||||
### In scope this phase
|
||||
|
||||
- `ml-worker` Docker service + image + requirements
|
||||
- Alembic migration for pgvector extension, five tables, HNSW index, seed config rows
|
||||
- SQLAlchemy models for all five tables (in `app/models.py` or a dedicated `app/models_ml.py` — follow whichever the codebase prefers for locality)
|
||||
- `app/tasks/ml.py` with `tag_and_embed`, `backfill`, `recompute_centroid`
|
||||
- Hook into `import_media_file` to chain `tag_and_embed` after commit
|
||||
- `app/services/tag_suggestions.py` suggestion-generation module
|
||||
- Three new Flask endpoints under the existing image blueprint
|
||||
- Modal UI additions: template block, CSS, `view-modal.js` suggestion logic
|
||||
- Settings → Maintenance tab: "Run ML backfill" button
|
||||
- Threshold values read from `tag_suggestion_config` (DB-editable via psql for now)
|
||||
|
||||
### Deferred
|
||||
|
||||
- Bulk-edit gallery multi-select suggestions (explicit next phase)
|
||||
- Admin UI for editing `tag_suggestion_config` thresholds
|
||||
- Admin panel showing character reference-count stats
|
||||
- Reranker / feedback-driven threshold tuning
|
||||
- Automatic backfill scheduling (manual trigger only for now)
|
||||
- Multi-centroid-per-character for characters with high intra-class variance (see parent spec §"Why centroids instead of querying all reference images")
|
||||
|
||||
---
|
||||
|
||||
## 7. Verification
|
||||
|
||||
Per the ImageRepo project norm, there is no automated test harness. Verification at implementation time is a manual browser check:
|
||||
|
||||
- Upload a new image; confirm `image_tag_prediction` and `image_embedding` rows appear after a few seconds.
|
||||
- Open the modal on an image that has predictions; confirm suggestions appear grouped by category with plausible confidences.
|
||||
- Accept a character suggestion; confirm the tag is applied, a `suggestion_feedback` row is written, and (if count delta is met) a `recompute_centroid` task runs and `character_reference_embedding` is updated.
|
||||
- Reject a suggestion; confirm a feedback row is written and the suggestion reappears on modal reopen.
|
||||
- Kick off a backfill from Settings → Maintenance; confirm it processes images in batches, is resumable after a restart, and does not stall the ml-worker.
|
||||
- Accept a general tag that doesn't exist in the DB (forcing the disabled-chip path); confirm the UI prevents submission and the endpoint returns `unknown_general_tag` if attempted.
|
||||
@@ -0,0 +1,177 @@
|
||||
# Character-Tag Integrity Design
|
||||
|
||||
**Date:** 2026-04-19
|
||||
**Branch:** `feat/tag-suggestions` (same branch as the tag-suggestion feature; this is a separate concern but bundles naturally).
|
||||
|
||||
## Problem
|
||||
|
||||
Two related defects erode the integrity of the `Tag` table's character/fandom data:
|
||||
|
||||
1. **Stale image↔fandom associations.** When a character tag gains a `fandom_id` via `POST /api/tag/<id>/set-fandom` (or via a rename that introduces `(Fandom)`), existing images already tagged with that character do not receive the fandom tag. Only *future* attachments through `add_tag` / `add_bulk_tag` auto-apply the fandom, because those paths explicitly append it. The result: images tagged before the association is set miss the fandom and won't show up under the fandom in filters or suggestions.
|
||||
|
||||
2. **Inconsistent casing on character/fandom display names.** The user (solo dev) introduces typos like `character:misty` alongside `character:Misty`. There is no normalization on create or rename, so the DB accumulates duplicates that diverge by case.
|
||||
|
||||
## Goals
|
||||
|
||||
- Every image tagged with a character whose tag has a `fandom_id` also has that fandom tag attached.
|
||||
- Character and fandom tag display names always start each whitespace-separated word with an upper-case letter.
|
||||
- Existing lowercase names are renamed one-time on migration, merging into any already-correctly-cased tag of the same kind.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Subtractive reconciliation (removing fandom tags from images when the character's fandom changes or is cleared). The DB does not record *why* a fandom was attached, so removal risks deleting user-intended tags.
|
||||
- Smart handling of apostrophes, hyphens, or Roman numerals in names. Users who need `"O'Brien"` or `"Mary-Kate"` can still type the exact casing via `set-fandom`; the normalizer only guarantees first-letter-of-each-word.
|
||||
- Normalization for other kinds (`artist`, `series`, `user`, `post`, `rating`). Scope stays on the two kinds the user asked about.
|
||||
- Retroactive removal of image↔fandom links. Only additive.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Normalization helper
|
||||
|
||||
New module `app/utils/tag_names.py`:
|
||||
|
||||
```python
|
||||
def normalize_display_name(s: str) -> str:
|
||||
"""Title-case each whitespace-separated word. Preserves internal punctuation.
|
||||
|
||||
Examples:
|
||||
"misty" -> "Misty"
|
||||
"misty ketchum" -> "Misty Ketchum"
|
||||
"o'brien" -> "O'brien"
|
||||
"mary-kate" -> "Mary-kate"
|
||||
"""
|
||||
return ' '.join(w[:1].upper() + w[1:] for w in s.split(' ') if w != '') \
|
||||
if s else s
|
||||
```
|
||||
|
||||
Called from every character/fandom write path:
|
||||
|
||||
- `add_tag` (`/image/<id>/tags/add`) — after parsing `(Fandom)` suffix, normalize both `char_name` and `fandom_name` before storing.
|
||||
- `add_bulk_tag` (in `app/main.py` around line 2240) — same.
|
||||
- `set_tag_fandom` (`/api/tag/<id>/set-fandom`) — normalize `fandom_name` when the `fandom_name` path is taken, normalize the character display base before reassembling the tag name.
|
||||
- `accept_image_suggestion` (WD14 accept path in `app/main.py`) — when auto-creating a new character or fandom tag from a WD14 suggestion, normalize first.
|
||||
- `_ensure_fandom_tag` (helper in `app/main.py`) — normalize internally so every caller benefits.
|
||||
- `_parse_character_fandom` (helper in `app/main.py`) — unchanged; parsing is orthogonal to normalization.
|
||||
|
||||
### Retroactive fandom backfill on `set_tag_fandom`
|
||||
|
||||
After the existing commit that renames the character tag and sets `fandom_id`, the endpoint runs a second transaction that attaches the fandom tag to every image already tagged with the character. The operation is a single SQL statement on the `image_tags` association table:
|
||||
|
||||
```sql
|
||||
INSERT INTO image_tags (image_id, tag_id)
|
||||
SELECT image_id, :fandom_tag_id
|
||||
FROM image_tags
|
||||
WHERE tag_id = :character_tag_id
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM image_tags it2
|
||||
WHERE it2.image_id = image_tags.image_id
|
||||
AND it2.tag_id = :fandom_tag_id
|
||||
);
|
||||
```
|
||||
|
||||
Running it in a second transaction after the rename commits means a failed backfill never rolls back the rename. A failed backfill leaves stale state that the sweep button will correct.
|
||||
|
||||
### "Sync character fandoms to images" sweep
|
||||
|
||||
New Celery task `sync_character_fandoms` in `app/tasks/maintenance.py` (new module) on the default queue:
|
||||
|
||||
```python
|
||||
@celery.task(name='app.tasks.maintenance.sync_character_fandoms',
|
||||
soft_time_limit=120, time_limit=180)
|
||||
def sync_character_fandoms():
|
||||
"""For every character tag with fandom_id, attach fandom to images
|
||||
that have the character but not the fandom."""
|
||||
with app.app_context():
|
||||
total_added = 0
|
||||
chars = Tag.query.filter_by(kind='character').filter(Tag.fandom_id.isnot(None)).all()
|
||||
for char in chars:
|
||||
inserted = _attach_fandom_to_images(char.id, char.fandom_id)
|
||||
total_added += inserted
|
||||
db.session.commit()
|
||||
log.info("sync_character_fandoms: added %d image↔fandom links across %d characters",
|
||||
total_added, len(chars))
|
||||
return {'characters_scanned': len(chars), 'links_added': total_added}
|
||||
```
|
||||
|
||||
New route `POST /settings/maintenance/sync-character-fandoms` that calls `sync_character_fandoms.apply_async()` (default queue, no `queue='ml'`).
|
||||
|
||||
Settings → Maintenance gets a third button "Sync character fandoms to images".
|
||||
|
||||
### One-time migration: rename + merge
|
||||
|
||||
New Alembic migration `h26041901_normalize_character_fandom_tag_names.py` (revision id follows the existing date-coded pattern; adjust `down_revision` to whatever `alembic current` is at deploy time).
|
||||
|
||||
Upgrade algorithm, entirely in raw SQL within a single transaction:
|
||||
|
||||
1. Build a working set: `SELECT id, name, kind, fandom_id FROM tag WHERE kind IN ('character', 'fandom')`.
|
||||
2. For each row, compute `normalized_name` by:
|
||||
- Splitting the name at the first `:` into `prefix` + `rest`.
|
||||
- For character tags: further split `rest` at ` (` to separate `char_base` from `fandom_part)`.
|
||||
- Normalize `char_base`.
|
||||
- If a fandom part exists, normalize it too and reassemble `{norm_char} ({norm_fandom})`.
|
||||
- For fandom tags: normalize `rest` directly.
|
||||
- Reassemble `{prefix}:{normalized_rest}`.
|
||||
3. For each `(id, old_name, new_name)` where `old_name != new_name`:
|
||||
- Check for a target row with `name = new_name AND kind = current_row.kind AND id != current_row.id`.
|
||||
- **No collision:** `UPDATE tag SET name = :new_name WHERE id = :id`.
|
||||
- **Collision (merge):**
|
||||
- Winner: the existing correctly-cased target (the one the current row would collide with).
|
||||
- Loser: the row being renamed.
|
||||
- `INSERT INTO image_tags(image_id, tag_id) SELECT DISTINCT image_id, :winner_id FROM image_tags WHERE tag_id = :loser_id ON CONFLICT DO NOTHING`.
|
||||
- `DELETE FROM image_tags WHERE tag_id = :loser_id`.
|
||||
- `UPDATE tag SET fandom_id = :winner_id WHERE fandom_id = :loser_id`.
|
||||
- `UPDATE tag_reference_embedding SET tag_name = :winner_name WHERE tag_name = :loser_name AND NOT EXISTS (SELECT 1 FROM tag_reference_embedding WHERE tag_name = :winner_name AND model_version = tag_reference_embedding.model_version)` — keeps winner's centroid if both have one.
|
||||
- `DELETE FROM tag_reference_embedding WHERE tag_name = :loser_name`.
|
||||
- `DELETE FROM tag WHERE id = :loser_id`.
|
||||
4. Log counts of renames vs merges to migration output.
|
||||
|
||||
Downgrade: no-op for merges (destructive), plain name-revert for pure renames is possible but not worth the bookkeeping. The downgrade function raises `NotImplementedError` with an explanatory message.
|
||||
|
||||
## Data flow — summary
|
||||
|
||||
```
|
||||
Write paths (add_tag, add_bulk_tag, accept_suggestion, set_tag_fandom)
|
||||
→ normalize_display_name(…)
|
||||
→ store {prefix}:{Normalized Name}
|
||||
→ attach to image (and auto-attach fandom if character has one)
|
||||
|
||||
set_tag_fandom (specifically)
|
||||
→ normalize + rename + set fandom_id (commit #1)
|
||||
→ SELECT image_ids tagged with this character
|
||||
→ INSERT (image_id, fandom_tag_id) for rows missing the fandom (commit #2)
|
||||
→ return JSON
|
||||
|
||||
Sweep button
|
||||
→ enqueue sync_character_fandoms on default queue
|
||||
→ iterate character tags with fandom_id
|
||||
→ per-tag: batch-insert missing links, commit per tag
|
||||
→ log summary
|
||||
|
||||
Migration (one-time)
|
||||
→ enumerate character:* and fandom:* tags
|
||||
→ compute normalized names
|
||||
→ rename survivors; merge collisions (reassign image_tags, fandom_id, tag_reference_embedding; delete loser)
|
||||
→ one transaction; no downgrade for merges
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- **Normalization of empty or whitespace-only string:** returns the input unchanged; callers already strip and reject empty via `abort(400)`.
|
||||
- **`set_tag_fandom` rename collision with differently-cased target:** post-normalization, both names converge, so the 409 check at `app/main.py:940` fires correctly. No behavior change.
|
||||
- **`set_tag_fandom` backfill DB error:** rename commits first (commit #1); backfill fails (commit #2 rolled back). Response still returns 200 with the renamed tag. Sweep button later recovers.
|
||||
- **Sweep task per-tag error:** try/except around each tag; log `sync_character_fandoms: tag %s failed: %s`, continue, return partial summary.
|
||||
- **Migration mid-flight error:** full rollback. User investigates via migration log.
|
||||
|
||||
## Testing plan
|
||||
|
||||
Manual verification on dev DB:
|
||||
|
||||
1. **Migration:** dev DB has `character:emelie`. Run migration → verify tag is renamed to `character:Emelie`, row count in `tag` unchanged, all `image_tags` links preserved.
|
||||
2. **Create with normalization:** via add-tag, submit `character:misty (pokemon)`. Verify tag lands as `character:Misty (Pokemon)`; fandom tag `fandom:Pokemon` is auto-created; both attached to the image.
|
||||
3. **Retroactive backfill on `set-fandom`:** pick an existing character with no fandom, attach it to 3 images. Call `set-fandom` with a new fandom_name. Verify all 3 images now have the fandom tag (even though `set-fandom` was the trigger).
|
||||
4. **Merge on migration:** manually seed `character:foo` and `character:Foo` with overlapping and distinct image links. Run migration. Verify one `character:Foo` survives with the union of image links; `character:foo` row is gone; `tag_reference_embedding` has one row (if either did); `fandom_id` references pointing at either row now point at the survivor.
|
||||
5. **Sweep button:** manually remove a fandom link from an image whose character tag has a fandom (simulating drift). Click "Sync character fandoms to images". Verify the fandom is re-attached and the task log shows `links_added: 1`.
|
||||
|
||||
## Open items
|
||||
|
||||
None. Design confirmed via brainstorming: additive-only backfill, title-case normalization via simple first-letter-per-word rule, covers character + fandom kinds, one-time migration with merge, sweep available as manual maintenance action.
|
||||
@@ -0,0 +1,269 @@
|
||||
## Tag Underscores + Modal Polish Design
|
||||
|
||||
**Date:** 2026-04-19
|
||||
**Branch:** `feat/tag-suggestions`
|
||||
**Builds on:** `2026-04-19-character-tag-integrity-design.md` (same branch, same normalizer module)
|
||||
|
||||
## Problem
|
||||
|
||||
Three related pain points around WD14-sourced suggestions and the suggestion modal:
|
||||
|
||||
1. **WD14 uses Danbooru-style underscored names.** Raw WD14 tags surface as `big_hair`, `long_hair`, `blue_sky`. The user's library uses spaces (`big hair`). Accepting a WD14 suggestion today writes the underscored form into the `Tag` table and attaches it to the image, so the library diverges from the user's naming convention.
|
||||
2. **Accepting a suggestion refreshes the attached-tags list later and more loosely than the manual "Add" button.** Manual add awaits `loadTags(imageId)` before showing feedback; accept fires `loadTags` un-awaited inside a 200 ms animation callback. The visible result: the modal's attached-tags list looks briefly stale after accept, and reject/accept ordering can race.
|
||||
3. **Accept (✓) and reject (✕) buttons have no persistent color cue.** Green/red only appear on hover, so the buttons read as decorative glyphs until the user hovers.
|
||||
|
||||
## Goals
|
||||
|
||||
- WD14 suggestion chips display and persist as space-separated names (`big hair`, not `big_hair`), regardless of kind.
|
||||
- Character and fandom names continue to be title-cased on write (`character:Misty`); general (null) names preserve the user's casing but gain underscore→space.
|
||||
- A one-time migration scrubs any already-persisted underscored names in the `Tag` table, merging into any existing space-separated duplicate.
|
||||
- Accepting a suggestion refreshes the modal's attached-tags list with the same timing semantics as manual add — the data refresh is awaited and not gated by animation.
|
||||
- Accept and reject buttons are persistently green and red, not only on hover.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing the WD14 model, thresholds, or suggestion logic. The transformation is cosmetic/naming.
|
||||
- Touching kinds beyond character, fandom, and null (general). Artist, post, series, rating, user, source, archive keep their current behavior.
|
||||
- Stripping other Danbooru-style punctuation (parentheses, backslashes). Underscore→space only.
|
||||
- Changing the reject endpoint behavior. It already records feedback without creating a tag; nothing to adjust on the backend.
|
||||
- Redesigning the chip layout. Only the button colors change.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Normalization helpers
|
||||
|
||||
`app/utils/tag_names.py` grows two additions:
|
||||
|
||||
```python
|
||||
def underscores_to_spaces(s: str) -> str:
|
||||
"""Replace underscores with spaces. Used for general/null-kind names and
|
||||
as a pre-step inside normalize_display_name for character/fandom."""
|
||||
return s.replace('_', ' ') if s else s
|
||||
|
||||
|
||||
def normalize_display_name(s: str) -> str:
|
||||
"""Title-case each whitespace-separated word. Underscores become spaces
|
||||
first so 'misty_ketchum' and 'Misty Ketchum' collapse onto the same name."""
|
||||
if not s:
|
||||
return s
|
||||
s = underscores_to_spaces(s)
|
||||
parts = s.split(' ')
|
||||
out: list[str] = []
|
||||
for p in parts:
|
||||
if p == '':
|
||||
out.append(p)
|
||||
else:
|
||||
out.append(p[:1].upper() + p[1:])
|
||||
return ' '.join(out)
|
||||
```
|
||||
|
||||
No new helper for "general" — write paths call `underscores_to_spaces` directly when handling null-kind. That keeps the module small (one concept per helper) and avoids a second name that does nothing more than call `.replace`.
|
||||
|
||||
### Write-path integration
|
||||
|
||||
#### `accept_image_suggestion` (`app/main.py:723`)
|
||||
|
||||
Accept is called only for suggestion categories `character`, `copyright`, and `general`. Kind mapping: `character → character`, `copyright → fandom`, `general → None`. Change:
|
||||
|
||||
- Before the `Tag.query.filter_by(name=tag_name)` lookup, canonicalize based on category:
|
||||
- `character` / `copyright`: split `prefix:rest`, apply `normalize_display_name` to `rest`, reassemble.
|
||||
- `general`: apply `underscores_to_spaces` to the whole name (no prefix expected).
|
||||
- Use the canonicalized `tag_name` for both the existence check and the new `Tag(...)` row. The existing character/fandom normalization block inside the "if tag is None" branch becomes redundant and is removed.
|
||||
|
||||
The net effect: `fandom:trigun_stampede` → `fandom:Trigun Stampede`; `big_hair` → `big hair`; `character:misty_ketchum` → `character:Misty Ketchum`. Accepting a WD14 chip produces the same Tag row the user would get by typing the clean name manually.
|
||||
|
||||
#### `add_tag` (`app/main.py:823`) and `bulk_add_tag` (`app/main.py:2279`)
|
||||
|
||||
These already call `normalize_display_name` on character/fandom. With `normalize_display_name` now stripping underscores internally, character and fandom paths are already covered. Add:
|
||||
|
||||
- For the no-prefix / `kind == "user"` path: `tag_name = underscores_to_spaces(tag_name)` before the Tag lookup.
|
||||
|
||||
Kinds that are **not** transformed: `artist`, `post`, `source`, `archive`, `series`, `rating`. Artists sometimes have `_` in their canonical identifier (e.g., `redjuice_redjuice`); post tags use `:` as the separator and may contain underscored platform IDs; source/archive/series are path- or identifier-like where underscores may be meaningful. Those paths pass their raw name through to the DB unchanged, matching current behavior.
|
||||
|
||||
Rule of thumb: the transform applies to exactly three kinds — `character`, `fandom`, and null (general) — the same set the migration touches.
|
||||
|
||||
### Suggestion-service transform
|
||||
|
||||
`app/services/tag_suggestions.py` — WD14 names are emitted to the modal still carrying underscores. Two choices:
|
||||
|
||||
- **Transform at emit time** (chosen): after `_wd14_suggestions` builds a row, rewrite `p.tag_name` using the same rule as the accept path. The chip displays `big hair` and the accept endpoint receives `big hair` in the POST body.
|
||||
- ~~Transform at paint time (in `view-modal.js`)~~ rejected — it would mean the display name and the POST body diverge, forcing the accept endpoint to re-derive the canonical form.
|
||||
|
||||
Implementation:
|
||||
|
||||
```python
|
||||
from app.utils.tag_names import normalize_display_name, underscores_to_spaces
|
||||
|
||||
_WD14_CATEGORY_TO_KIND = {
|
||||
'character': 'character',
|
||||
'copyright': 'fandom',
|
||||
}
|
||||
|
||||
def _canonicalize_wd14_name(raw: str, category: str) -> str:
|
||||
kind = _WD14_CATEGORY_TO_KIND.get(category)
|
||||
if ':' in raw:
|
||||
prefix, rest = raw.split(':', 1)
|
||||
transform = normalize_display_name if kind in ('character', 'fandom') else underscores_to_spaces
|
||||
return f"{prefix}:{transform(rest)}"
|
||||
return normalize_display_name(raw) if kind in ('character', 'fandom') else underscores_to_spaces(raw)
|
||||
```
|
||||
|
||||
The existing `already` set (tags already attached to the image) is also canonicalized — `{underscores_to_spaces(n) for n in already}` — so that an already-attached `big hair` correctly shadows a WD14 `big_hair` suggestion.
|
||||
|
||||
`_WD14_CATEGORY_TO_KIND` moves from `app/main.py` to the service module to avoid circular imports; `main.py` imports from the service.
|
||||
|
||||
### Migration `i26041901_normalize_underscores_and_general_tags.py`
|
||||
|
||||
One-time rename-or-merge covering every existing row in `tag` where the name contains `_` in the portion after the first `:` (or the whole name if there is no `:`). The down_revision is `h26041901` (the previous character/fandom normalization migration).
|
||||
|
||||
Algorithm (raw SQL, single transaction):
|
||||
|
||||
1. `SELECT id, name, kind, fandom_id FROM tag WHERE kind IN ('character', 'fandom') OR kind IS NULL` — widen the net to any kind that the write-paths now transform.
|
||||
2. For each row, compute `new_name`:
|
||||
- Split on first `:` into `prefix`, `rest` (or `rest = name` if no prefix).
|
||||
- For character: split `rest` at ` (` to isolate `char_base` and `fandom_part)`; run `normalize_display_name` on each; reassemble.
|
||||
- For fandom: `normalize_display_name(rest)`.
|
||||
- For null (general): `underscores_to_spaces(rest)`.
|
||||
- Reassemble with prefix (character/fandom always have prefixes; general usually has none).
|
||||
3. For each `(id, old_name, new_name)` where they differ:
|
||||
- Look for a `tag` row with `name = new_name AND kind = current.kind AND id != current.id`.
|
||||
- **No collision:** `UPDATE tag SET name = :new_name WHERE id = :id`.
|
||||
- **Collision (merge):** winner = existing correctly-named row, loser = current row.
|
||||
- `INSERT INTO image_tags(image_id, tag_id) SELECT DISTINCT image_id, :winner_id FROM image_tags WHERE tag_id = :loser_id ON CONFLICT DO NOTHING`
|
||||
- `DELETE FROM image_tags WHERE tag_id = :loser_id`
|
||||
- `UPDATE tag SET fandom_id = :winner_id WHERE fandom_id = :loser_id`
|
||||
- `UPDATE tag_reference_embedding SET tag_name = :winner_name WHERE tag_name = :loser_name AND NOT EXISTS (SELECT 1 FROM tag_reference_embedding t2 WHERE t2.tag_name = :winner_name AND t2.model_version = tag_reference_embedding.model_version)`
|
||||
- `DELETE FROM tag_reference_embedding WHERE tag_name = :loser_name`
|
||||
- `DELETE FROM tag WHERE id = :loser_id`
|
||||
4. Log rename count and merge count.
|
||||
|
||||
Downgrade: raises `NotImplementedError` (merges are destructive, same rationale as `h26041901`).
|
||||
|
||||
Code duplication note: this migration and `h26041901` share the "compute new name / rename-or-merge" shape but not the exact transform. Migrations are historical artifacts — they run once and are never edited — so repeating the scaffolding is preferred over extracting a shared helper module that the migrations would need to import.
|
||||
|
||||
### Modal JS: accept-path refresh parity
|
||||
|
||||
`app/static/js/view-modal.js:203-233` (`acceptSuggestion`):
|
||||
|
||||
```javascript
|
||||
async function acceptSuggestion(chip, imageId) {
|
||||
if (chip.dataset.disabled === 'true') return;
|
||||
chip.dataset.disabled = 'true';
|
||||
chip.style.pointerEvents = 'none';
|
||||
try {
|
||||
const r = await fetch(`/image/${imageId}/suggestions/accept`, { /* ...body as before... */ });
|
||||
const j = await r.json();
|
||||
if (!j.ok) {
|
||||
chip.dataset.disabled = 'false';
|
||||
chip.style.pointerEvents = '';
|
||||
if (tagActionFeedback) {
|
||||
tagActionFeedback.textContent = `Couldn't accept: ${j.error || 'error'}`;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (typeof loadTags === 'function') await loadTags(imageId);
|
||||
if (tagActionFeedback) tagActionFeedback.textContent = `Added ${j.tag.name}`;
|
||||
if (typeof window.refreshItemTags === 'function') window.refreshItemTags(imageId);
|
||||
animateChipOut(chip); // fire-and-forget; data is already refreshed
|
||||
} catch {
|
||||
chip.dataset.disabled = 'false';
|
||||
chip.style.pointerEvents = '';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key changes:
|
||||
- `await loadTags(imageId)` before setting feedback and before the chip animation runs. Matches the manual `addTag` ordering at `view-modal.js:600-602`.
|
||||
- `animateChipOut` no longer takes a completion callback (it's become a pure visual-only helper); its `setTimeout(...)` just removes the node. The signature becomes `animateChipOut(chip)` — callers that still pass a callback lose the callback argument cleanly.
|
||||
- The chip is disabled via `dataset.disabled` for duplicate-click protection.
|
||||
|
||||
### Modal CSS: persistent green/red chip buttons
|
||||
|
||||
`app/static/style.css:980-1006`:
|
||||
|
||||
```css
|
||||
.suggestion-chip .sugg-btn.sugg-accept {
|
||||
color: #8be78b;
|
||||
border-color: rgba(139, 231, 139, 0.35);
|
||||
}
|
||||
.suggestion-chip .sugg-btn.sugg-accept:hover {
|
||||
background: rgba(139, 231, 139, 0.12);
|
||||
border-color: rgba(139, 231, 139, 0.6);
|
||||
}
|
||||
.suggestion-chip .sugg-btn.sugg-reject {
|
||||
color: #e78b8b;
|
||||
border-color: rgba(231, 139, 139, 0.35);
|
||||
}
|
||||
.suggestion-chip .sugg-btn.sugg-reject:hover {
|
||||
background: rgba(231, 139, 139, 0.12);
|
||||
border-color: rgba(231, 139, 139, 0.6);
|
||||
}
|
||||
```
|
||||
|
||||
The base `.sugg-btn` rule keeps `color: inherit` for anything that isn't accept/reject — safe-by-default if a future button gets added.
|
||||
|
||||
## Data flow — summary
|
||||
|
||||
```
|
||||
Suggestion emit:
|
||||
WD14 prediction ("big_hair", "character:misty_ketchum")
|
||||
→ _wd14_suggestions builds row
|
||||
→ _canonicalize_wd14_name rewrites name ("big hair", "character:Misty Ketchum")
|
||||
→ merged with embedding rows (already canonical)
|
||||
→ returned to modal
|
||||
|
||||
Accept:
|
||||
POST /image/<id>/suggestions/accept {tag_name: "big hair", category: "general"}
|
||||
→ normalize tag_name (underscores_to_spaces on general, normalize_display_name on character/fandom)
|
||||
→ Tag.query.filter_by(name=...) — single lookup
|
||||
→ if missing, create with kind = _WD14_CATEGORY_TO_KIND[category] or None
|
||||
→ attach to image, record feedback, commit
|
||||
→ return {ok: true, tag}
|
||||
Modal:
|
||||
→ await loadTags(imageId) — tag list refreshes (same timing as manual add)
|
||||
→ set feedback text
|
||||
→ refresh gallery thumbnail overlay
|
||||
→ animate chip out (visual only)
|
||||
|
||||
Migration:
|
||||
enumerate kind in (character, fandom, NULL) tag rows
|
||||
compute new name per kind
|
||||
rename survivors; merge collisions (image_tags union, fandom_id reassign, embedding keep-winner, delete loser)
|
||||
one transaction, no downgrade
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- **Canonicalization of empty string:** returns input unchanged; caller's existing `if not tag_name` check short-circuits.
|
||||
- **Accept with name already canonical:** transforms are idempotent; second call is a no-op on the name.
|
||||
- **Migration mid-flight error:** full transaction rollback. User investigates via migration log.
|
||||
- **WD14 emits a name with no underscores but mixed case (e.g., `Copyright:OnePiece`):** `normalize_display_name` applies to character/fandom regardless of underscores; result is `copyright:Onepiece`. The user can still rename via `set-fandom` later if needed. General with no underscore: no change.
|
||||
- **Accept fetch rejects or throws:** chip is re-enabled via `chip.dataset.disabled = 'false'`; user can retry.
|
||||
|
||||
## Testing plan
|
||||
|
||||
Manual verification on dev DB:
|
||||
|
||||
1. **Migration:** seed `tag` rows `big_hair` (kind=NULL), `fandom:trigun_stampede`, and a pre-existing `big hair` (kind=NULL) to force a merge. Run migration. Verify:
|
||||
- `big_hair` is gone, `big hair` remains with the union of image links.
|
||||
- `fandom:trigun_stampede` is renamed to `fandom:Trigun Stampede` (rename, not merge — no collision).
|
||||
- `tag_reference_embedding` rows for merged losers are deleted.
|
||||
2. **WD14 chip display:** open an image modal whose WD14 predictions include `long_hair`. Verify the chip shows "long hair" (not "long_hair").
|
||||
3. **Accept refresh parity:** click the ✓ on a suggestion. Observe: attached-tags list updates *before* the chip animation completes (no stale frame). Feedback text matches manual-add timing.
|
||||
4. **Accept canonicalization roundtrip:** click ✓ on a `big_hair` suggestion. Verify:
|
||||
- `Tag.query.filter_by(name='big hair').first()` returns a row.
|
||||
- No `big_hair` row exists in `tag`.
|
||||
- The image's tags include `big hair`.
|
||||
5. **Button color:** without hovering, verify ✓ is green-tinted and ✕ is red-tinted on all chips.
|
||||
6. **Duplicate-click protection:** rapid double-click on ✓. Verify only one accept POST fires.
|
||||
7. **Accept error path:** force the backend to return `{ok: false}`; verify chip re-enables, feedback shows the error, chip stays on screen.
|
||||
|
||||
## Open items
|
||||
|
||||
None. Design confirmed via brainstorming:
|
||||
- WD14 is the only source introducing underscores.
|
||||
- Canonicalize at one layer (service for display, accept for persistence — matching transforms).
|
||||
- General keeps case; only character/fandom title-case.
|
||||
- Migration covers character + fandom + null kinds with merge-on-collision.
|
||||
- Modal accept mirrors manual-add refresh ordering.
|
||||
- Chip buttons persist green/red tint; hover intensifies rather than introduces.
|
||||
@@ -0,0 +1,157 @@
|
||||
# User-Tag Embedding-Similarity Suggestions Design
|
||||
|
||||
**Date:** 2026-04-19
|
||||
**Branch:** `feat/tag-suggestions` (same branch as the base feature; this is an incremental extension)
|
||||
**Supersedes:** Partially — extends the character-only centroid mechanism documented in `docs/superpowers/tag-suggestion-system-spec.md`.
|
||||
|
||||
## Problem
|
||||
|
||||
WD14's tag vocabulary is Danbooru-style (`1girl`, `long_hair`, `blue_sky`). The user's library uses their own naming conventions. Today, WD14 is the only source of general/copyright suggestions, so users rarely see their own tags surfaced even when an image visually matches others they've already tagged.
|
||||
|
||||
Character tags already bypass this problem because SigLIP centroid similarity suggests the user's character tags based on visual resemblance to previously-tagged images. This design extends the same mechanism to the other tag kinds the user actually uses: `fandom` (copyright/IP) and null (general/topic).
|
||||
|
||||
## Goals
|
||||
|
||||
- Extend embedding-similarity suggestions from `character` only to `character` + `fandom` + null.
|
||||
- Keep WD14 alongside — it still runs, still produces suggestions, merges with embedding suggestions via name dedup.
|
||||
- Drop the `rating` kind from the UI and accept path entirely (the user doesn't need content gating).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Replacing WD14. WD14 suggestions continue to surface for the three kinds in scope.
|
||||
- Per-kind tuning of thresholds or min-refs. Start with shared defaults; add knobs only if real usage shows a need.
|
||||
- Extending to `artist`, `series`, `post`, or `rating`. Scope stays minimal.
|
||||
- Changing the accept/reject UX.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Storage
|
||||
|
||||
The existing `character_reference_embedding` table becomes `tag_reference_embedding`:
|
||||
|
||||
- Rename table: `character_reference_embedding` → `tag_reference_embedding`.
|
||||
- Rename column: `character_tag` → `tag_name`.
|
||||
- Add column: `tag_kind TEXT NULL` (populated by recompute, nullable for general/topic).
|
||||
- Unique index migrates with the table: `(tag_name, model_version)`.
|
||||
- pgvector ivfflat index on `centroid` migrates with the table.
|
||||
|
||||
The SQLAlchemy model class renames from `CharacterReferenceEmbedding` to `TagReferenceEmbedding`.
|
||||
|
||||
### Eligible kinds
|
||||
|
||||
Embedding centroids are computed only for tags whose `kind` is in `{'character', 'fandom', None}`. A single module-level constant `ELIGIBLE_CENTROID_KINDS` lives in `app/services/tag_suggestions.py` and is imported where needed (recompute task, accept endpoint). Adding `artist` or `series` later is a one-tuple change.
|
||||
|
||||
`rating`, `post`, and any other kinds are skipped.
|
||||
|
||||
### Recompute task
|
||||
|
||||
`recompute_centroid(tag_name)` in `app/tasks/ml.py`:
|
||||
|
||||
1. Look up the Tag row to get its `kind`.
|
||||
2. If `kind` is not in `ELIGIBLE_CENTROID_KINDS`, log and return `{'status': 'ineligible_kind'}`.
|
||||
3. Query all embeddings for images tagged with `tag_name`. If empty, return `{'status': 'no_embeddings'}`.
|
||||
4. Compute mean vector.
|
||||
5. Upsert `TagReferenceEmbedding(tag_name=..., tag_kind=..., model_version=..., centroid=..., reference_count=...)` on the `(tag_name, model_version)` conflict.
|
||||
|
||||
### Batch recompute task
|
||||
|
||||
New task `recompute_all_centroids()`:
|
||||
|
||||
1. For each Tag where `kind IN (character, fandom, NULL)`, count attached images.
|
||||
2. For each tag with `count >= min_reference_images`, enqueue `recompute_centroid(tag_name)` on the `ml` queue.
|
||||
3. Logs progress per batch.
|
||||
|
||||
Exposed as a Settings → Maintenance button next to the existing "Run ML backfill" button.
|
||||
|
||||
### Suggestion service
|
||||
|
||||
In `app/services/tag_suggestions.py`:
|
||||
|
||||
- `_embedding_character_suggestions` renames to `_embedding_tag_suggestions`.
|
||||
- Query drops the character-only filter. Returns suggestions for all centroids above the similarity threshold with `reference_count >= min_reference_images`.
|
||||
- For each returned row, map `tag_kind` to a display category:
|
||||
- `character` → `character`
|
||||
- `fandom` → `copyright`
|
||||
- `None` → `general`
|
||||
|
||||
The existing `_merge` step dedups by name. If WD14 and embedding similarity both suggest the same tag, the higher-confidence one wins.
|
||||
|
||||
`get_suggestions` returns three groups: `character`, `copyright`, `general`. The `rating` group is removed.
|
||||
|
||||
### Accept endpoint
|
||||
|
||||
In `app/main.py`:
|
||||
|
||||
- `_WD14_CATEGORY_TO_KIND` drops the `'rating': 'rating'` entry. It becomes `{'character': 'character', 'copyright': 'fandom'}`.
|
||||
- The centroid-recompute trigger block currently guarded by `if tag.kind == 'character'` changes to `if tag.kind in ELIGIBLE_CENTROID_KINDS`. Because `ELIGIBLE_CENTROID_KINDS` includes `None`, null-kind general tags are covered by the same check.
|
||||
- Delta-triggered recompute stays at `centroid_recompute_delta = 3` globally.
|
||||
|
||||
### Config cleanup
|
||||
|
||||
The existing `tag_suggestion_config` rows are renamed/removed:
|
||||
|
||||
- `threshold_embedding_character` → `threshold_embedding` (same default: 0.85)
|
||||
- `min_reference_images_per_character` → `min_reference_images` (same default: 5)
|
||||
- `threshold_rating` → deleted
|
||||
- Other thresholds (`threshold_general`, `threshold_character_wd14`, `threshold_copyright`) stay unchanged since they still gate WD14 suggestions.
|
||||
|
||||
### UI
|
||||
|
||||
Modal template (`app/templates/_gallery_modal.html`) and `view-modal.js`:
|
||||
|
||||
- Suggestion category order becomes `['character', 'copyright', 'general']` — `rating` is removed.
|
||||
- Category label map drops the `rating` entry.
|
||||
- No visual changes to the chip layout (already updated in a prior change).
|
||||
|
||||
## Data flow on accept
|
||||
|
||||
```
|
||||
user clicks ✓
|
||||
→ POST /image/<id>/suggestions/accept
|
||||
→ tag created if missing (kind = WD14 category map, or None)
|
||||
→ tag attached to image
|
||||
→ if tag.kind in ELIGIBLE_CENTROID_KINDS or None, check delta:
|
||||
current_image_count - last_reference_count >= 3
|
||||
→ enqueue recompute_centroid(tag_name) on ml queue
|
||||
→ ml-worker runs recompute_centroid
|
||||
→ mean-pools embeddings, upserts tag_reference_embedding row
|
||||
→ next image's suggestion request benefits from the updated centroid
|
||||
```
|
||||
|
||||
## Migration plan for existing data
|
||||
|
||||
The database already has one row in `character_reference_embedding` per character tag touched so far. The migration:
|
||||
|
||||
1. `ALTER TABLE character_reference_embedding RENAME TO tag_reference_embedding`.
|
||||
2. `ALTER TABLE tag_reference_embedding RENAME COLUMN character_tag TO tag_name`.
|
||||
3. `ALTER TABLE tag_reference_embedding ADD COLUMN tag_kind TEXT NULL`.
|
||||
4. Backfill existing rows: `UPDATE tag_reference_embedding SET tag_kind = 'character'` (safe because every pre-existing row was written by the old character-only path).
|
||||
5. Drop `threshold_rating` row from `tag_suggestion_config`.
|
||||
6. Rename `threshold_embedding_character` → `threshold_embedding`.
|
||||
7. Rename `min_reference_images_per_character` → `min_reference_images`.
|
||||
|
||||
After migration, the user clicks "Recompute all centroids" once from Settings → Maintenance to populate centroids for any fandom/general tags they've already applied to 5+ images.
|
||||
|
||||
### Existing rating data is preserved
|
||||
|
||||
The migration does not delete any `Tag` rows with `kind='rating'` or their image-tag associations. WD14 continues to store rating predictions in `image_tag_prediction` (category='rating') on every tag_and_embed run. This data is simply no longer surfaced in the suggestions UI or accepted via the new endpoint paths. If the user ever wants rating gating back, re-adding it is a UI-only change.
|
||||
|
||||
## Error handling
|
||||
|
||||
- `recompute_centroid` for an ineligible kind: logs and returns `{'status': 'ineligible_kind'}`. No retry.
|
||||
- `recompute_centroid` for a tag with 0 images (e.g., all images that used this tag were deleted): logs and returns `{'status': 'no_embeddings'}`. No retry.
|
||||
- `recompute_all_centroids` failures in individual enqueues: logged, loop continues.
|
||||
- Accept endpoint centroid enqueue failure: swallowed (existing behavior — don't fail the accept).
|
||||
|
||||
## Testing plan
|
||||
|
||||
Manual verification after the implementation lands:
|
||||
|
||||
1. Migration runs cleanly on the dev DB. Existing character centroid survives with `tag_kind='character'`.
|
||||
2. Fresh suggestion request for an image returns three groups (no `rating`).
|
||||
3. Tag a fandom across 5 images, trigger recompute, verify new image shows that fandom as a suggestion.
|
||||
4. Tag a null-kind general topic across 5 images, trigger recompute, verify the general group now includes it.
|
||||
5. Accept a general-kind suggestion — verify `Tag(kind=None)` is created, image gets it, centroid recompute is enqueued after the delta threshold is met.
|
||||
6. Settings → Maintenance → "Recompute all centroids" button enqueues recomputes only for eligible-kind tags with ≥ 5 images.
|
||||
|
||||
No automated tests in this spec — the base feature didn't have any either, and the manual test plan in the parent spec still covers the integration surface.
|
||||
@@ -0,0 +1,265 @@
|
||||
# Tag Suggestion System - Technical Spec
|
||||
|
||||
## Overview
|
||||
|
||||
Add an ML-powered tag suggestion system to the existing gallery application. The system suggests tags for images in two ways:
|
||||
|
||||
1. **Direct classification via WD14** for general tags, ratings, popular characters, and series
|
||||
2. **Embedding similarity** for OC/niche character identification based on previously-tagged reference images
|
||||
|
||||
The system produces *suggestions only* - never auto-applies tags. All suggestions require user confirmation. Artists are explicitly out of scope; they are set at import time from source metadata.
|
||||
|
||||
## Scale & Context
|
||||
|
||||
- ~62,000 existing images (anime/drawn art exclusively)
|
||||
- ~300 tagged characters currently, expected to grow
|
||||
- Single-user or small-user tagging workflow
|
||||
- Target hardware: NVIDIA Tesla P4 GPU (8GB VRAM, compute-only, no NVDEC needed for this workload)
|
||||
- Existing app is homegrown; this is an addition, not a rewrite
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Artist identification (handled at import from source metadata)
|
||||
- Auto-applying tags without user confirmation
|
||||
- Training or fine-tuning models (inference only; "learning" happens via reference set growth)
|
||||
- Multi-user permission models for suggestions
|
||||
|
||||
## Models
|
||||
|
||||
### Tagger: WD14 (SmilingWolf EVA02-Large variant)
|
||||
|
||||
- Model: `SmilingWolf/wd-eva02-large-tagger-v3` (or current best available)
|
||||
- Purpose: Produces confidence-scored tags across categories (general, character, copyright/series, rating, meta)
|
||||
- Runs on GPU via ONNX Runtime with CUDA execution provider
|
||||
- Output: tag name + category + confidence score (0.0-1.0)
|
||||
|
||||
### Embedding Model: SigLIP or OpenCLIP
|
||||
|
||||
- Recommended: `google/siglip-so400m-patch14-384` (strong performance, reasonable size)
|
||||
- Alternative: `laion/CLIP-ViT-H-14-laion2B-s32B-b79K` if SigLIP integration is problematic
|
||||
- Output: Fixed-dimension embedding vector per image (SigLIP SO400M: 1152-dim)
|
||||
- Purpose: Enables similarity search for character matching against reference set
|
||||
|
||||
### Model Versioning
|
||||
|
||||
Store the model identifier (name + version) alongside every output. Schema must support multiple model versions coexisting to allow migration without requiring immediate reprocessing of all 62K images.
|
||||
|
||||
## Database Schema Changes
|
||||
|
||||
Assumes Postgres. If the existing app uses a different database, adapt accordingly - `pgvector` is strongly preferred for embedding storage at this scale.
|
||||
|
||||
### New extension
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
```
|
||||
|
||||
### New tables
|
||||
|
||||
```sql
|
||||
-- Raw WD14 tagger output, one row per (image, tag, model_version)
|
||||
CREATE TABLE image_tag_predictions (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
image_id BIGINT NOT NULL REFERENCES images(id) ON DELETE CASCADE,
|
||||
tag_name TEXT NOT NULL,
|
||||
tag_category TEXT NOT NULL, -- 'general', 'character', 'copyright', 'rating', 'meta'
|
||||
confidence REAL NOT NULL,
|
||||
model_version TEXT NOT NULL, -- e.g. 'wd-eva02-large-tagger-v3'
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_tag_predictions_image ON image_tag_predictions(image_id);
|
||||
CREATE INDEX idx_tag_predictions_tag ON image_tag_predictions(tag_name);
|
||||
CREATE INDEX idx_tag_predictions_model ON image_tag_predictions(model_version);
|
||||
|
||||
-- Image embeddings for similarity search
|
||||
CREATE TABLE image_embeddings (
|
||||
image_id BIGINT NOT NULL REFERENCES images(id) ON DELETE CASCADE,
|
||||
model_version TEXT NOT NULL,
|
||||
embedding vector(1152) NOT NULL, -- adjust dimension to match chosen model
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (image_id, model_version)
|
||||
);
|
||||
|
||||
-- HNSW index for fast nearest-neighbor search
|
||||
CREATE INDEX idx_embeddings_hnsw ON image_embeddings
|
||||
USING hnsw (embedding vector_cosine_ops);
|
||||
|
||||
-- Cached centroid embeddings per character, for fast suggestion queries
|
||||
CREATE TABLE character_reference_embeddings (
|
||||
character_tag TEXT NOT NULL,
|
||||
model_version TEXT NOT NULL,
|
||||
centroid vector(1152) NOT NULL,
|
||||
reference_count INTEGER NOT NULL, -- how many confirmed images contributed
|
||||
computed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (character_tag, model_version)
|
||||
);
|
||||
|
||||
-- Tracks user decisions on suggestions for future tuning/reranking
|
||||
CREATE TABLE suggestion_feedback (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
image_id BIGINT NOT NULL REFERENCES images(id) ON DELETE CASCADE,
|
||||
tag_name TEXT NOT NULL,
|
||||
suggestion_source TEXT NOT NULL, -- 'wd14' or 'embedding_similarity'
|
||||
confidence REAL NOT NULL,
|
||||
decision TEXT NOT NULL, -- 'accepted' or 'rejected'
|
||||
decided_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_feedback_image ON suggestion_feedback(image_id);
|
||||
CREATE INDEX idx_feedback_tag ON suggestion_feedback(tag_name);
|
||||
```
|
||||
|
||||
### Configuration table (or use existing config system)
|
||||
|
||||
Store these as tunable values, not hardcoded:
|
||||
|
||||
```sql
|
||||
CREATE TABLE tag_suggestion_config (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL,
|
||||
description TEXT
|
||||
);
|
||||
|
||||
-- Initial values (thresholds will need tuning)
|
||||
INSERT INTO tag_suggestion_config VALUES
|
||||
('threshold_general', '0.35', 'Min confidence for general tag suggestions'),
|
||||
('threshold_character_wd14', '0.75', 'Min confidence for WD14 character tags'),
|
||||
('threshold_copyright', '0.5', 'Min confidence for copyright/series tags'),
|
||||
('threshold_rating', '0.5', 'Min confidence for rating tags'),
|
||||
('threshold_meta', '0.5', 'Min confidence for meta tags'),
|
||||
('threshold_embedding_character', '0.85', 'Min cosine similarity for character via embeddings'),
|
||||
('min_reference_images_per_character', '5', 'Characters with fewer confirmed refs are excluded from embedding suggestions'),
|
||||
('centroid_recompute_delta', '3', 'Recompute character centroid after N new confirmed references');
|
||||
```
|
||||
|
||||
## Inference Pipeline
|
||||
|
||||
### On upload (per-image, near-realtime)
|
||||
|
||||
1. WD14 inference → insert rows into `image_tag_predictions`
|
||||
2. Embedding model inference → insert row into `image_embeddings`
|
||||
3. Suggestions are generated on-demand at view time (see below); no storage of suggestions themselves needed
|
||||
|
||||
### Initial backfill (one-time, for existing 62K images)
|
||||
|
||||
- Background job that iterates all images without predictions/embeddings
|
||||
- Batched GPU inference (WD14 supports batch sizes; 8-16 is a reasonable starting point on P4)
|
||||
- Expect several hours of runtime; run overnight
|
||||
- Must be resumable (track last-processed image_id, handle failures gracefully)
|
||||
- Progress reporting (log every N images, estimated time remaining)
|
||||
|
||||
### On confirmed character tag (user accepts a character tag on an image)
|
||||
|
||||
- Increment a counter for that character's reference set
|
||||
- If delta since last centroid computation exceeds `centroid_recompute_delta`, schedule centroid recomputation for that character
|
||||
- Recomputation: fetch all embeddings for images confirmed to have this character tag, compute mean (or medoid if desired), update `character_reference_embeddings`
|
||||
|
||||
### Suggestion generation (at view time or via background job)
|
||||
|
||||
When generating suggestions for an image:
|
||||
|
||||
**From WD14 predictions:**
|
||||
- Query `image_tag_predictions` for this image
|
||||
- Filter by per-category confidence thresholds
|
||||
- Exclude tags already confirmed on the image
|
||||
|
||||
**From embedding similarity (for characters only):**
|
||||
- Fetch this image's embedding
|
||||
- Query `character_reference_embeddings` for all characters with `reference_count >= min_reference_images_per_character`
|
||||
- Compute cosine similarity against each character's centroid
|
||||
- Filter by `threshold_embedding_character`
|
||||
- Exclude characters already confirmed on the image
|
||||
- Return top K (suggest K=5 initially)
|
||||
|
||||
**Merging:**
|
||||
- WD14 character suggestions and embedding-based character suggestions may overlap; dedupe by tag name, prefer whichever has higher normalized confidence
|
||||
- Return merged suggestion list grouped by category, sorted by confidence within each category
|
||||
|
||||
## Background Jobs
|
||||
|
||||
Three scheduled/triggered jobs:
|
||||
|
||||
### 1. Backfill job (one-time initially, resumable)
|
||||
|
||||
- Process any image missing WD14 predictions or embeddings for current model versions
|
||||
- Can be re-run after model version change to update stored data
|
||||
|
||||
### 2. Centroid recomputation (triggered or nightly)
|
||||
|
||||
- For each character whose `reference_count` has changed by more than `centroid_recompute_delta` since last computation, recompute centroid
|
||||
- Fast operation; can run frequently
|
||||
|
||||
### 3. Upload processing (per-image, on upload)
|
||||
|
||||
- Runs WD14 + embedding inference on newly uploaded images
|
||||
- Should not block the upload response; queue for async processing
|
||||
|
||||
## API / Integration Points
|
||||
|
||||
The spec doesn't prescribe exact endpoints since it depends on the existing app's conventions, but the following operations must be exposed:
|
||||
|
||||
- `get_suggestions(image_id) -> list of suggested tags grouped by category` - called when viewing an image's tag panel
|
||||
- `accept_suggestion(image_id, tag_name, source)` - records feedback, applies tag to image, triggers centroid recomputation if character
|
||||
- `reject_suggestion(image_id, tag_name, source)` - records feedback only
|
||||
- `queue_backfill()` - admin operation to process any unprocessed images
|
||||
- `get_character_reference_stats() -> list of (character, reference_count)` - admin visibility into which characters have enough references to participate in embedding suggestions
|
||||
|
||||
## Implementation Notes & Decisions
|
||||
|
||||
### Why not store suggestions in the database?
|
||||
|
||||
Suggestions are a function of (predictions + embeddings + current confirmed tags + current thresholds + current centroids). Any of these can change. Storing suggestions creates a cache invalidation problem. Instead, compute them on read - the operations are cheap once predictions and embeddings exist.
|
||||
|
||||
### Why store raw WD14 predictions at all?
|
||||
|
||||
Two reasons:
|
||||
1. Threshold tuning without re-running inference. Change thresholds, suggestions update immediately.
|
||||
2. Model upgrade path. When moving from model v3 to v4, old predictions remain queryable until new ones are generated, avoiding a blackout period.
|
||||
|
||||
### Why centroids instead of querying all reference images?
|
||||
|
||||
At 300 characters with potentially thousands of references each, querying per-image embeddings for every character at suggestion time gets expensive. Centroids reduce it to one vector per character. Trade-off: loses some nuance (a character with multiple distinct outfits is represented as one "average" point). If this becomes a problem, options include:
|
||||
- Multiple centroids per character via clustering (k-medoids with small k)
|
||||
- Fallback to top-K raw embedding search for characters with high intra-class variance
|
||||
|
||||
Not worth building upfront. Revisit if suggestion quality suffers.
|
||||
|
||||
### Why track rejection feedback?
|
||||
|
||||
Stored feedback enables:
|
||||
- Per-tag or per-category threshold tuning based on acceptance rates
|
||||
- Future reranker training if suggestion quality plateaus
|
||||
- Diagnosing which characters consistently produce bad suggestions
|
||||
|
||||
Low cost to collect; high value if needed later.
|
||||
|
||||
### Embedding dimension
|
||||
|
||||
Adjust `vector(1152)` in schema to match whatever model is actually selected. SigLIP SO400M is 1152-dim; ViT-H/14 CLIP is 1024-dim; some other variants differ. Mismatch will cause pgvector errors.
|
||||
|
||||
### GPU considerations
|
||||
|
||||
- P4 is compute-only (no NVDEC needed for this workload - we're doing ML inference, not video decode)
|
||||
- WD14 ONNX model fits comfortably in 8GB VRAM
|
||||
- SigLIP SO400M inference fits in 8GB with room to spare
|
||||
- If co-located with other GPU workloads (e.g., Ollama), use `CUDA_VISIBLE_DEVICES` to pin to a specific GPU and avoid contention
|
||||
|
||||
## Tuning Phase (post-implementation)
|
||||
|
||||
After initial deployment, expect to spend time tuning:
|
||||
|
||||
1. Per-category confidence thresholds (collect feedback data first, then analyze acceptance rate vs. confidence curves)
|
||||
2. Per-character embedding thresholds (some characters are distinctive, some generic-looking)
|
||||
3. Minimum reference count before a character enters the suggestion pool
|
||||
4. Centroid recomputation cadence
|
||||
|
||||
Build thresholds as config values (see `tag_suggestion_config` table), not constants. Expose them via whatever admin interface the existing app has.
|
||||
|
||||
## Open Questions for Implementation
|
||||
|
||||
- Which ONNX Runtime integration approach fits the existing app's stack (Python FastAPI, Rust, Node, etc.)?
|
||||
- Is there an existing background job system in the app, or does one need to be added?
|
||||
- What's the existing app's approach to config management - env vars, DB table, config file?
|
||||
|
||||
These should be answered by inspecting the existing codebase before implementation begins.
|
||||
Reference in New Issue
Block a user