Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.
Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries
Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints
Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
a flagged-corrupt record routes through _supersede_existing,
preserving tags/series/embeddings
Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
get_suggestions all filter integrity_status IN ('ok', 'unknown') so
flagged rows don't poison the gallery, ML, or suggestion math.
'unknown' is treated as healthy so post-migration data stays visible
until the sweep runs.
UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a per-image rejection filter to get_suggestions so once the user
clicks ✕ on an auto-accepted chip (or rejects a regular suggestion
via the existing ✕ button), that tag name stops reappearing in
suggestions — including auto_accept candidates.
Without this filter, the undo flow was cosmetic: the user's ✕
removed the tag from this image and logged a SuggestionFeedback
decision='rejected' row, but the next modal load re-ran auto-accept
against the same unchanged WD14 predictions and re-attached the same
tag. Rinse, repeat.
Implementation: _rejected_names_for_image(image_id) reads distinct
tag_names from suggestion_feedback where decision='rejected' for
this image. get_suggestions filters merged results against that set
alongside the existing blocklist filter. Both auto-accept candidates
and the three core categories respect the filter.
Tested on image 222 at threshold=0.9: first load auto-applied 23
general tags; rejected '1girl'; second load auto_accepted.count=0,
general suggestions had 10 new items, '1girl' absent from both.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two capabilities that work together to turn the suggestion system from
a 'review every noisy chip' UX into a 'curated tags only, configurable
auto-apply' UX for a solo-user library.
Auto-accept threshold
Service (tag_suggestions.py):
- New default auto_accept_general_threshold = 0.95 in _DEFAULTS
- get_suggestions splits general-category hits at/above the threshold
into a separate auto_accept_candidates list; the core 'character'/
'copyright'/'general' keys stay the same shape for existing callers
- get_bulk_suggestions ignores the new key (no side effects in bulk)
Route (main.py GET /image/<id>/suggestions):
- For each candidate: find-or-create Tag(kind='user', name=name), attach
to image, log SuggestionFeedback(source='wd14'|etc, decision='accepted')
- Response adds 'auto_accepted: [{id, name, display_name, kind,
confidence, source}, ...]' so the modal can review + undo
Config endpoints (main.py):
GET /api/suggestions/config/auto-accept-threshold
POST /api/suggestions/config/auto-accept-threshold {threshold}
Values > 1.0 effectively disable the feature
UI (view-modal.js):
- New renderAutoAccepted block at top of suggestions section with
green-tinted chips carrying ✕ (undo for this image, logs rejection)
and ⊘ (blocklist + remove from all images)
- loadSuggestions refreshes the tag pill list when auto_accepted has
items so the user sees them in the Tags section too
Settings:
- Maintenance tab gains a number input + save button backed by
/api/suggestions/config/auto-accept-threshold
Retroactive blocklist sweep
New Celery task app.tasks.maintenance.sweep_blocklisted_tag_from_images
on the maintenance queue. Enqueued automatically when a name is added
via POST /api/suggestions/blocklist or appears new in the bulk replace.
Task body: finds kind='user' Tag matching the name, deletes its
image_tags rows, deletes the Tag itself. Scope limited to kind='user'
so deliberate character/fandom/artist tags sharing a blocklisted name
aren't silently destroyed.
Verification (local dev):
- Threshold GET/POST round-trip correct (default 0.95, persisted in
tag_suggestion_config)
- Image 633 at threshold=0.9: 4 general tags auto-applied, each with
a SuggestionFeedback row, returned in auto_accepted
- Blocklist add of 'sweeptestonly' with an attached test tag: Redis
maintenance queue depth = 1, task body removes 1 image_tags row +
the Tag row
- get_bulk_suggestions still works (auto_accept_candidates key skipped)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a tag_suggestion_blocklist table + service-layer filter so the
user can permanently suppress specific canonical tag names from the
suggestion stream (WD14 anatomy/composition tags like '1girl',
'breasts', 'nipples' that aren't useful for the discoverability use
case this app targets).
Data model
- migration k26042201: tag_suggestion_blocklist(name TEXT PK, created_at)
- model TagSuggestionBlocklistEntry
Service
- tag_suggestions._blocklisted_names() snapshots the current set
- get_suggestions filters merged results before grouping, so both
WD14- and embedding-sourced suggestions respect the blocklist
- get_bulk_suggestions inherits the filter via its per-image call
to get_suggestions
API
- GET /api/suggestions/blocklist -> {ok, names}
- POST /api/suggestions/blocklist -> add one
- POST /api/suggestions/blocklist/delete -> remove one
- POST /api/suggestions/blocklist/bulk -> replace the whole list
(backs the settings textarea save button)
UI
- modal suggestion chip gets a third action button (⊘) alongside
accept (✓) / reject (✕). Clicking it adds the name to the
blocklist, logs a rejection for ML feedback on this image, and
sweeps every chip on the page carrying that same name.
- Settings -> Maintenance -> Suggestion blocklist section with a
monospaced textarea (one name per line) + Save. Loads current
entries on mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_FANDOM_SUFFIX_RE, _fandom_suffix_prefixes, and the prefix-suppression
branch in _wd14_suggestions existed to work around the string-embedded
(Fandom) kludge. The bare-name refactor makes them all dead code.
_canonicalize_wd14_name simplifies to a single normalize-or-underscore
swap since no prefix preservation is needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
WD14 emits bare names (e.g. "Ruby Rose") while curated character/fandom
tags carry a fandom suffix ("Ruby Rose (RWBY)"). Two gaps caused a
duplicate, fandom-less tag to be created when users accepted a WD14
suggestion for an image that already had the curated version:
- Suggestion compute: bare character/copyright suggestions are now
suppressed when the image already carries a `Name (Fandom)` variant,
so the duplicate chip no longer surfaces.
- Accept endpoint: if the exact-name lookup misses for character/fandom
kinds, we try a `Name (%)`-scoped lookup and attach the existing tag
only when exactly one candidate matches. Zero or multiple matches
fall through to prior behavior to avoid silently guessing across
ambiguous fandoms.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bulk editor now loads consensus ML suggestions across the current
selection via a new POST /api/suggestions/bulk endpoint, powered by
get_bulk_suggestions() in tag_suggestions.py. A tag is surfaced only if
it was suggested for or already applied to >= 80% of the selection;
coverage counts include images that already have the tag so a near-
universal tag isn't penalized for dropping out of suggestion lists.
Accept-only chips (no reject) match the rest of the suggestions UX.
Frontend polish in the same commit:
- Showcase session dedup (exclude seen ids, reset on second lap) and
aspect-ratio-aware placeholder heights for better column balancing
- Modal touch-swipe navigation on the image wrapper, auto-focus of the
tag input on desktop (>=768px), and autocomplete flip-up when the
dropdown would cover the Suggestions section
- Settings template inline styles replaced with utility classes
- Inline series-editor script extracted to gallery-series-editor.js
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consolidated merge of feat/tag-suggestions branch. Original 64-commit history
was lost to git-object corruption in a Nextcloud-synced checkout; this single
commit captures the equivalent diff.
Includes:
- pgvector-backed tag suggestion infra (WD14 + SigLIP centroids, ml-worker
container, Celery tasks, suggestion service, accept/reject endpoints + modal
UI with green/red chip buttons)
- Character/fandom integrity: title-case normalization on every write path,
fandom-id backfill, maintenance task + settings button, migrations g26041901
+ h26041901 to canonicalize legacy rows with case-only duplicate merging
- Tag-underscores + modal polish: WD14 name canonicalization at emit + accept
+ add/bulk-add paths, migration i26041901 for legacy-row rename-or-merge
across character/fandom/NULL kinds, suggestion-accept refresh parity via
awaited loadTags, persistent chip tint
bvandeusen