Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.
Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries
Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints
Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
a flagged-corrupt record routes through _supersede_existing,
preserving tags/series/embeddings
Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
get_suggestions all filter integrity_status IN ('ok', 'unknown') so
flagged rows don't poison the gallery, ML, or suggestion math.
'unknown' is treated as healthy so post-migration data stays visible
until the sweep runs.
UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The add-tag input lets the user type 'character:mocha' as a kind shortcut,
but the autocomplete was passing the whole query through to ilike, so
'%character:mocha%' never matched any display_name. Use parse_kind_prefix
to split the prefix into a kind filter, leaving the rest as the search
term. An explicit ?kind= (e.g. fandom picker) still wins. Empty rest
('character:') falls into the no-query branch and shows top characters.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
parse_kind_prefix now lowercases the prefix before checking KNOWN_KINDS,
matching the modal JS which already lowercases before its visibility
check. Without this, typing 'Character:Jinx ...' showed the fandom
picker but the server fell back to kind=user.
Both /tags/add and /api/images/bulk-add-tag now also split a trailing
'(Fandom)' suffix from a typed character name when no explicit fandom
is staged, so a manual entry like 'Character:Jinx (League Of Legends)'
produces a clean (name='Jinx', fandom_id=<league>) row instead of a
malformed character with the suffix baked into the name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two-part fix for the reported bug where accepting a character
suggestion named 'Jinx (League Of Legends)' created a single
malformed character tag with fandom_id=NULL and the whole suffix
embedded in the name, and never attached the fandom tag to the
image.
Root cause: WD14 predictions store names like
'jinx_(league_of_legends)'. app.services.tag_suggestions
._canonicalize_wd14_name transforms that to 'Jinx (League Of
Legends)' for character/copyright categories. The suggestion chip
renders that canonicalized string. On ✓, accept_image_suggestion
looked for an existing character with name = 'Jinx (League Of
Legends)' — post-refactor no such row exists (names are bare) —
fell into the 0-candidate branch, and created a fresh malformed
character.
Going forward (main.py):
accept_image_suggestion's character branch now splits a
'(Fandom)' suffix off the incoming name before the 0/1/N lookup.
If it splits, find-or-create the fandom tag, then find-or-create
a character with (kind='character', name=bare, fandom_id=<f>).
If no suffix, unchanged 0/1/N behavior.
Backfill for already-corrupted rows (tasks/maintenance.py):
New _heal_malformed_character_names pass runs before the
existing sync-fandoms-to-images logic. Finds
(kind='character', fandom_id IS NULL, name LIKE '% (%)%'),
parses the suffix via the same regex, and either promotes the
row to (name=bare, fandom_id=<f>) or merges into an existing
canonical character. tag_reference_embedding rows (string PK)
are cascade-renamed alongside. Same auto-merge semantics as
migration j26042101 Phase 2.
Implementation is all raw SQL to sidestep an ORM autoflush
ordering quirk: ORM-level fandom INSERT + subsequent tag UPDATE
referencing its fresh id hit a transient FK violation in worker
task context. Raw SQL with an explicit commit after fandom
insert avoids the ambiguity.
Verified locally:
- Fresh accept of 'CanonicalChar (Canonical Fandom)' against a
pre-existing canonical character: resolved to the canonical
row, fandom attached to image.
- Seeded malformed 'LegacyJinx (League Of Legends)' character
with fandom_id=NULL attached to an image: sync task healed it
to bare 'LegacyJinx' + fandom_id, attached the fandom row to
image_tags, final counts {healed:1, skipped:0,
links_added:1}.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New Celery task sync_character_fandoms_to_images on the maintenance
queue. For every character tag with a non-null fandom_id, runs the
same INSERT ... SELECT ... ON CONFLICT DO NOTHING that set_tag_fandom
already runs inline, attaching the fandom tag to every image that
has the character attached.
Why this exists: migration j26042101 extracted the '(Fandom)' suffix
from pre-refactor character names into tag.fandom_id, but it did NOT
walk image_tags to attach the fandom rows to every image that had
the character. Post-refactor auto-apply (in add_tag /
accept_image_suggestion / set_tag_fandom) only fires on NEW
add/accept events — pre-existing character attachments from before
the migration still show the character pill in the modal but no
fandom pill, because the fandom row was never added to image_tags.
Surfaced by Settings → Maintenance → 'Sync fandoms to images'.
Safe, additive, idempotent. Run once after the migration to close
the gap; subsequent set-fandom operations continue to maintain the
invariant inline.
Verified locally: seeded a character+fandom pair with only the
character attached to image 1; task added the fandom to image_tags;
final state has both rows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous fix only covered the suggestion-chip ✕ flow (which calls
/suggestions/reject). Removing a tag via the pill × in the Tags section
goes through /tags/remove and wrote no feedback row, so the per-image
rejection filter in get_suggestions had nothing to match on — auto-accept
would re-apply the tag on the next modal load.
/tags/remove now writes SuggestionFeedback(decision='rejected',
suggestion_source='manual_removal', confidence=0) alongside the image_tags
delete. The suggestion_source value distinguishes these rows from
chip-reject rejections in the feedback log without complicating the
get_suggestions filter (it matches on decision alone).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two capabilities that work together to turn the suggestion system from
a 'review every noisy chip' UX into a 'curated tags only, configurable
auto-apply' UX for a solo-user library.
Auto-accept threshold
Service (tag_suggestions.py):
- New default auto_accept_general_threshold = 0.95 in _DEFAULTS
- get_suggestions splits general-category hits at/above the threshold
into a separate auto_accept_candidates list; the core 'character'/
'copyright'/'general' keys stay the same shape for existing callers
- get_bulk_suggestions ignores the new key (no side effects in bulk)
Route (main.py GET /image/<id>/suggestions):
- For each candidate: find-or-create Tag(kind='user', name=name), attach
to image, log SuggestionFeedback(source='wd14'|etc, decision='accepted')
- Response adds 'auto_accepted: [{id, name, display_name, kind,
confidence, source}, ...]' so the modal can review + undo
Config endpoints (main.py):
GET /api/suggestions/config/auto-accept-threshold
POST /api/suggestions/config/auto-accept-threshold {threshold}
Values > 1.0 effectively disable the feature
UI (view-modal.js):
- New renderAutoAccepted block at top of suggestions section with
green-tinted chips carrying ✕ (undo for this image, logs rejection)
and ⊘ (blocklist + remove from all images)
- loadSuggestions refreshes the tag pill list when auto_accepted has
items so the user sees them in the Tags section too
Settings:
- Maintenance tab gains a number input + save button backed by
/api/suggestions/config/auto-accept-threshold
Retroactive blocklist sweep
New Celery task app.tasks.maintenance.sweep_blocklisted_tag_from_images
on the maintenance queue. Enqueued automatically when a name is added
via POST /api/suggestions/blocklist or appears new in the bulk replace.
Task body: finds kind='user' Tag matching the name, deletes its
image_tags rows, deletes the Tag itself. Scope limited to kind='user'
so deliberate character/fandom/artist tags sharing a blocklisted name
aren't silently destroyed.
Verification (local dev):
- Threshold GET/POST round-trip correct (default 0.95, persisted in
tag_suggestion_config)
- Image 633 at threshold=0.9: 4 general tags auto-applied, each with
a SuggestionFeedback row, returned in auto_accepted
- Blocklist add of 'sweeptestonly' with an attached test tag: Redis
maintenance queue depth = 1, task body removes 1 image_tags row +
the Tag row
- get_bulk_suggestions still works (auto_accept_candidates key skipped)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a tag_suggestion_blocklist table + service-layer filter so the
user can permanently suppress specific canonical tag names from the
suggestion stream (WD14 anatomy/composition tags like '1girl',
'breasts', 'nipples' that aren't useful for the discoverability use
case this app targets).
Data model
- migration k26042201: tag_suggestion_blocklist(name TEXT PK, created_at)
- model TagSuggestionBlocklistEntry
Service
- tag_suggestions._blocklisted_names() snapshots the current set
- get_suggestions filters merged results before grouping, so both
WD14- and embedding-sourced suggestions respect the blocklist
- get_bulk_suggestions inherits the filter via its per-image call
to get_suggestions
API
- GET /api/suggestions/blocklist -> {ok, names}
- POST /api/suggestions/blocklist -> add one
- POST /api/suggestions/blocklist/delete -> remove one
- POST /api/suggestions/blocklist/bulk -> replace the whole list
(backs the settings textarea save button)
UI
- modal suggestion chip gets a third action button (⊘) alongside
accept (✓) / reject (✕). Clicking it adds the name to the
blocklist, logs a rejection for ML feedback on this image, and
sweeps every chip on the page carrying that same name.
- Settings -> Maintenance -> Suggestion blocklist section with a
monospaced textarea (one name per line) + Save. Loads current
entries on mount.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1. Hide 'archive' kind from /tags default view. Archive tags only
appear when the user explicitly requests them via ?kind=archive.
_get_tags_with_previews gains a default_excluded_kinds tuple that
applies when no explicit kind filter is set.
2. Hide the 'Add' submit button on desktop (>=768px). Enter in the
tag input (and in the fandom picker when no row is highlighted)
already submits; the button is only needed for mobile virtual
keyboards where Enter-to-submit is unreliable.
3. Fandom picker keyboard parity: ArrowUp/Down navigate rows, Enter
picks a highlighted row or (if no row is highlighted) submits the
whole add-tag form. Escape clears the highlight. Mirrors the main
tagInput's keyboard block.
4. Tag editor merge bug: the edit modal populated nameInput.value
with data.tag.display_name — for a character with a fandom, that
value is 'Name (Fandom)'. Saving without stripping the suffix
routed to a rename that never collided with the canonical bare
name, so the merge offer never fired. Now uses data.tag.name (the
bare form). The (Fandom) display is still shown in the fandom
input which is rendered separately.
5. Fandom picker styling: match the main tag-form input — same
padding, border, background, focus state. Drop the separate
'Fandom' label; the placeholder 'Fandom (optional)' now carries
the label role.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Task 12's rewrite of list_tags dropped the ok flag from the response.
loadTags in view-modal.js gates its render on 'if (j.ok)', so every
POST add-tag returned 200, the feedback said 'Tag added', but the
pill list never refreshed because loadTags silently dropped the
response as not-ok.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clicking an autocomplete row (or pressing Enter on a highlighted row)
now sends tag_id in the add-tag POST. Server attaches that exact tag,
skipping parse_kind_prefix. Same-name ambiguity — e.g. a character
tag 'Yidhari' and a user tag 'Yidhari' both matching the autocomplete
query 'yid' — now resolves by whichever row the user picked, instead
of silently routing to the user-kind fallback.
Bare free-text input (no click) still goes through parse_kind_prefix
with its kind: shortcut and kind='user' default. The fandom picker
path is untouched.
The click handler also stashes tag_id on the dataset so the keyboard
Enter path on a highlighted row picks up the same resolution.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
artist_gallery redirect and reapply_artist_tags_from_paths both still
built f"artist:{name}" strings. Last stragglers cleaned up; the
dead-code sweep now returns zero prefix-construction hits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a header counter ('⚠ N characters need a fandom') that filters to
?null_fandom=1, plus an inline chip on each qualifying character card.
Backend accepts null_fandom_only in the list query.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
tag.fandom_id is now authoritative. set_tag_fandom auto-applies the
fandom tag inline; add_tag already did. There's no drift to sync on
a schedule anymore.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every standalone prefix-strip expression in app/main.py replaced with
tag.display_name (user-facing) or tag.name (identifier contexts). The
only remaining colon-splits are inside parse_kind_prefix and tag
suggestions canonicalization.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes _parse_character_fandom (no longer needed — names are bare).
update_tag respects the new (name, kind[, fandom_id]) uniqueness shape.
get_tag and list_tags return display_name so clients render without
re-parsing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
No rename, no retroactive-backfill-in-second-txn. display_name @property
reflects the change instantly. The auto-apply side effect (attach fandom
tag to images with the character) is preserved but simplified.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drops the ?tag=<name> lookup. URLs become unambiguous (distinct
same-name characters each have their own id-keyed URL). Eager-loads
Tag.fandom for the active-tag header rendering.
Applied to all four gallery routes: /gallery, /api/gallery/scroll,
/api/gallery/timeline, /api/gallery/jump — they all shared the same
name-keyed filter pattern. (Plan named only the first two; extended
to all four for behavioral consistency so partial JS migration to
tag_id doesn't silently drop filter on the sub-APIs.)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Post-refactor tag.name is bare. Autocomplete needs to match the
display form (name + ' (' + fandom.name + ')') so users can still
find 'Ruby Rose (RWBY)' by typing 'rwby'. Switched to
contains_eager(Tag.fandom, alias=f) to eliminate the duplicate
join that caused a Postgres GROUP BY error in the no-query path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the ilike-suffix fallback with explicit bare-name character
matching: 0 candidates -> create null-fandom, 1 -> attach, N -> 409
with candidate list for frontend disambiguation. Accepts explicit
tag_id to complete the disambiguation round-trip.
Preserves existing behavior on the accept side:
- defensive _canonicalize_wd14_name on the boundary
- SuggestionFeedback row logged on every accepted decision
(including the explicit_tag_id path). Ambiguous 409 does not
log — no decision has been made yet.
- conditional centroid recompute enqueue for eligible kinds
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors add_tag's new shape. Removes inline prefix-parsing, returns
display_name in the response.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Accepts 'character:Saber' user shortcut, parses via parse_kind_prefix,
stores tag.name bare. Character path accepts fandom_id (preferred) or
fandom_name for fandom association. Returns display_name in the
response so the client can render pills without a second round-trip.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Post-migration tag.name no longer embeds 'fandom:' prefix. Helper now
creates/finds by (kind='fandom', name=bare).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
WD14 emits bare names (e.g. "Ruby Rose") while curated character/fandom
tags carry a fandom suffix ("Ruby Rose (RWBY)"). Two gaps caused a
duplicate, fandom-less tag to be created when users accepted a WD14
suggestion for an image that already had the curated version:
- Suggestion compute: bare character/copyright suggestions are now
suppressed when the image already carries a `Name (Fandom)` variant,
so the duplicate chip no longer surfaces.
- Accept endpoint: if the exact-name lookup misses for character/fandom
kinds, we try a `Name (%)`-scoped lookup and attach the existing tag
only when exactly one candidate matches. Zero or multiple matches
fall through to prior behavior to avoid silently guessing across
ambiguous fandoms.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bulk editor now loads consensus ML suggestions across the current
selection via a new POST /api/suggestions/bulk endpoint, powered by
get_bulk_suggestions() in tag_suggestions.py. A tag is surfaced only if
it was suggested for or already applied to >= 80% of the selection;
coverage counts include images that already have the tag so a near-
universal tag isn't penalized for dropping out of suggestion lists.
Accept-only chips (no reject) match the rest of the suggestions UX.
Frontend polish in the same commit:
- Showcase session dedup (exclude seen ids, reset on second lap) and
aspect-ratio-aware placeholder heights for better column balancing
- Modal touch-swipe navigation on the image wrapper, auto-focus of the
tag input on desktop (>=768px), and autocomplete flip-up when the
dropdown would cover the Suggestions section
- Settings template inline styles replaced with utility classes
- Inline series-editor script extracted to gallery-series-editor.js
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consolidated merge of feat/tag-suggestions branch. Original 64-commit history
was lost to git-object corruption in a Nextcloud-synced checkout; this single
commit captures the equivalent diff.
Includes:
- pgvector-backed tag suggestion infra (WD14 + SigLIP centroids, ml-worker
container, Celery tasks, suggestion service, accept/reject endpoints + modal
UI with green/red chip buttons)
- Character/fandom integrity: title-case normalization on every write path,
fandom-id backfill, maintenance task + settings button, migrations g26041901
+ h26041901 to canonicalize legacy rows with case-only duplicate merging
- Tag-underscores + modal polish: WD14 name canonicalization at emit + accept
+ add/bulk-add paths, migration i26041901 for legacy-row rename-or-merge
across character/fandom/NULL kinds, suggestion-accept refresh parity via
awaited loadTags, persistent chip tint
Each settings tab is now a separate page load via ?tab= query param.
Flask renders only the active panel - no CSS or JS visibility logic needed.
Tab bar uses <a> links instead of buttons. Eliminates the display:none
cascade issue that was preventing Import and Maintenance tabs from showing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Link character tags to fandom tags via fandom_id FK with auto-apply
behavior. Adding character:Saber (Fate) auto-creates fandom:Fate and
applies it to the image. Renaming a fandom cascades to all linked
characters. Includes set-fandom endpoint for retroactive association
and fandom selector dropdown in the tag editor UI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quick scan (default):
- Pre-load all ImportTask records for O(1) duplicate checks
- Batch task creation commits (50 at a time)
- Cache import settings (5-min TTL)
- Skip pHash comparison for speed
Deep scan (on-demand):
- Full reprocessing of all files
- pHash similarity detection
- Optional thumbnail regeneration
- Optional sidecar re-application
Archive tag merging:
- Add archive tags to existing images when duplicates found
- Works for both hash and pHash duplicate detection
Also fixes:
- Add missing source/post tag icons to gallery templates
bvandeusen
Bryan Van Deusen