Tag-maintenance sweep + bug-fix batch: #699 #700 #701 #709 #711 #712 #713 #714 #73

Merged

bvandeusen merged 11 commits from dev into main 2026-06-06 16:49:01 -04:00

Conversation 0 Commits 11 Files Changed 31

+1162 -82

bvandeusen commented 2026-06-06 16:48:52 -04:00

Owner

Copy Link

Copy Source

Clears the open FabledCurator todo list. All CI-green on dev (run 615).

Tags / normalization

• #701 — Title-Case normalization for operator-entered tags at the create endpoint only; shared find_or_create/rename keep original casing (ML/booru vocabulary) but gain case-insensitive lookup + clash detection so case variants dedup instead of forking.
• #714 — retro-normalize the back-catalog: a maintenance action that Title-Cases every existing tag and merges case/whitespace-variant duplicates via the tested TagService._do_merge (FK repoints + protective ML aliases). dry-run projection + long-running maintenance task; "Standardize tag casing" section in the Cleanup tab behind a back-up-first warning.

Downloads / ingester

• #709 — live per-file progress (downloaded/skipped/errors/posts) on running native-ingester events, time-throttled (5s) and surfaced beside the elapsed timer in the active-downloads panel.

Maintenance / archives (#713)

• Magic-byte archive detection so mis-named / extension-less Patreon attachments extract correctly.
• Re-extract maintenance action that stages under the artist dir so extracted members link to the right artist + post.

UI / modal bug-fixes

• #699 — showcase over-samples + random-orders to break near-duplicate clustering.
• #700 — ESC reliably closes the modal after accepting a suggestion (tooltip-overlay guard).
• #711 — modal tag-chip kebab un-nested from the v-chip so Rename/Set-fandom menus open.
• #712 — fandom picker lists all fandoms (paginate the directory instead of abusing autocomplete).
• Artists view: card-preview dead-space + first-load empty-state flicker.

🤖 Generated with Claude Code

Clears the open FabledCurator todo list. All CI-green on dev (run 615). ## Tags / normalization - **#701** — Title-Case normalization for operator-entered tags at the create endpoint only; shared `find_or_create`/`rename` keep original casing (ML/booru vocabulary) but gain case-insensitive lookup + clash detection so case variants dedup instead of forking. - **#714** — retro-normalize the back-catalog: a maintenance action that Title-Cases every existing tag and merges case/whitespace-variant duplicates via the tested `TagService._do_merge` (FK repoints + protective ML aliases). dry-run projection + long-running maintenance task; "Standardize tag casing" section in the Cleanup tab behind a back-up-first warning. ## Downloads / ingester - **#709** — live per-file progress (downloaded/skipped/errors/posts) on running native-ingester events, time-throttled (5s) and surfaced beside the elapsed timer in the active-downloads panel. ## Maintenance / archives (#713) - Magic-byte archive detection so mis-named / extension-less Patreon attachments extract correctly. - Re-extract maintenance action that stages under the artist dir so extracted members link to the right artist + post. ## UI / modal bug-fixes - **#699** — showcase over-samples + random-orders to break near-duplicate clustering. - **#700** — ESC reliably closes the modal after accepting a suggestion (tooltip-overlay guard). - **#711** — modal tag-chip kebab un-nested from the v-chip so Rename/Set-fandom menus open. - **#712** — fandom picker lists all fandoms (paginate the directory instead of abusing autocomplete). - Artists view: card-preview dead-space + first-load empty-state flicker. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 11 commits 2026-06-06 16:48:52 -04:00

fix(artists): card preview dead-space + empty-state flicker on first load ... 911d535f56

Two operator-reported Artists-view bugs.

Layout: ArtistCard previews used aspect-ratio: 3/1 + max-height: 220px — when a
lone grid column got wide (>~660px), the max-height made the aspect-ratio box
shrink its own WIDTH to keep the ratio, leaving dead space beside the 3
thumbnails. Dropped max-height (kept the min-height floor) so the strip fills the
card width; lowered the grid min 440→360px so a lone column stays <732px (strip
≲244px) and desktop gets 2+ columns.

Flicker: isEmpty checked !loading, but on the first render loading is still false
(the initial loadMore fires in onMounted, after paint) so "No artists match"
flashed for a frame before the spinner/data. Added a reactive `loaded` flag (true
after the first load attempt, reset on reset()); isEmpty now also requires it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1 ... 5bb25245a5

Patreon attachment downloads land with sanitized URL-blob filenames
(01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never
.zip — so the extension-only is_archive() filed them as opaque PostAttachments
and never extracted them ("No images attached to this post").

Add archive_extractor.detect_archive_format() — extension first, then magic-byte
sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(),
and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a
mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested,
AND extracted regardless of filename. Stops new ones; part 2 re-extracts the
already-imported backlog.

Tests: mis-named zip detected + extracted; non-archive dotted name not
misdetected; _inspect_archive on a mis-named zip; signature updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(maintenance): re-extract archive attachments + link to post — #713 part 2 ... a497104661

Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.

Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.

Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(maintenance): stage re-extract under the artist dir so members link — #713 part 2 fix ... cb9b286c53

The integration test caught it: _import_media re-derives the artist by path-walk
from the attribution path (ignoring the explicit artist) AND _copy_to_library
lands members relative to that path. Staging the archive in /tmp meant the
artist didn't resolve (provenance skipped) and members would land in the temp
dir. Stage under images_root/<slug>/<platform>/<post>/ instead so the artist
resolves and members land in the real library; remove only the staged archive +
sidecar afterward (members stay). Require a real artist+slug (skip + count
otherwise).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(modal/tags): fandom list, modal kebab, ESC-after-accept — #712 #711 #700 ... 4fe53cdf6b

#712 — fandom picker showed no existing fandoms: loadFandoms enumerated via
/tags/autocomplete with q=' ', which the backend strips to empty → returns [].
Switch to the cursor-paged /tags/directory?kind=fandom (loop all pages); drop the
load-once guard so the dialog reflects fandoms created elsewhere.

#711 — modal tag-chip kebab never opened: the kebab + menu were nested INSIDE the
v-chip, which swallowed the click / mis-anchored the teleported menu. Un-nest it
as a sibling v-btn using the standard v-menu activator slot (Vuetify wires the
click and stacks the overlay above the modal natively). Removes the openTagId
workaround.

#700 — ESC didn't close the modal after accepting a suggested tag: the guard
suppressed close while ANY .v-overlay--active existed, which includes tooltips —
a lingering tooltip blocked the close. Exclude .v-tooltip from the guard so only
real interactive overlays (menus/dialogs) keep ESC from closing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(tags): Title-Case normalization on create/rename — #701 (core) ... 2b69540ecc

Tags now normalize to Title Case + collapsed whitespace at the central
TagService.find_or_create (and rename) — so manual entry, the create API, and
anything routed through find_or_create produce the canonical form. The lookup is
case-insensitive, so a differently-cased entry finds the existing tag instead of
forking a case-variant duplicate ('hatsune miku' / 'HATSUNE MIKU' → one tag).

normalize_tag_name uses per-word capitalize (not str.title(), which mangles
apostrophes) and folds ALL-CAPS input. Existing tags keep their current casing
until touched — a retro-normalize maintenance pass (Title-Case + merge
case-collisions) is the follow-up to convert the back-catalog.

Test: create title-cases + collapses whitespace; case/whitespace variants dedupe
to one tag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(showcase): over-sample + random-order to break near-dup clustering — #699 ... 89dfa42e18

TABLESAMPLE SYSTEM_ROWS reads CONTIGUOUS rows from each sampled page, so
sequentially-imported near-duplicates (multi-image posts, variant sets) came
back adjacent and clustered in the showcase ("three near-identical in a row").
Sample limit*5 rows (spanning more pages) then ORDER BY random() before taking
limit — breaks the physical adjacency for much better spread, still cheap
(random() over a few hundred rows, not the whole table).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(downloads): live per-file progress on running events — #709 ... 62cca64dce

Now that we own the walk, surface live counts on the in-flight download in the
Downloads view. ingest_core.run takes an event_id and does a TIME-THROTTLED
write (~5s, decoupled from page boundaries so it ticks steadily regardless of
how big/slow a page is) of {downloaded, skipped, errors, quarantined, posts} to
the running download_event's metadata.live (jsonb_set; short session; status
guard so a finalized event isn't clobbered). download_backends threads
event_id from ctx; the /api/downloads list surfaces `live`; ActiveDownloadsPanel
renders it beside the elapsed timer. Native (Patreon) only — gallery-dl is an
opaque subprocess; the row only shows when `live` is present. Phase 3 overwrites
metadata with run_stats on finish, dropping `live`.

Test: _write_live_progress updates a running event's metadata.live and leaves a
finalized (status != running) event alone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(tags): Title-Case operator-entered tags at create endpoint only (plan #701) ... 23f452021f

normalize_tag_name (per-word capitalize + whitespace collapse) is applied in
the POST /api/tags handler so operator-entered tags get clean display casing.
It is NOT applied in the shared find_or_create / rename paths — those are used
by the ML tagger and allowlist matching, which must preserve the booru
vocabulary's original casing (Title-Casing it broke apply_allowlist matching).

find_or_create / rename keep case-insensitive lookup + clash detection so a
differently-cased entry dedups onto the existing tag instead of forking.

Tests updated to expect Title-Cased create output (sunset→Sunset,
character:Saber→Character:saber, http://example.com→Http://example.com) and a
dedicated normalize_tag_name unit test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714) ... 3c89223dcb

Follow-up to #701: new tags are saved canonical, but the back-catalog keeps
whatever casing it was created with. This adds a maintenance action that
Title-Cases every existing tag (collapsing whitespace) and merges
case/whitespace-variant duplicates into one.

Backend:
- tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by
  (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor
  (prefer an already-canonical member → no rename/self-alias; else the
  best-connected tag → fewest FK repoints; else lowest id), merges the variants
  INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/
  aliases/series_page repoints + protective ML aliases), then renames the
  survivor to canonical. Losers are deleted before the rename so there's no
  transient unique-index clash; commits per group and isolates failures per
  group. Idempotent — an already-canonical lone tag is a no-op.
- normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async
  engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i.
- POST /api/admin/tags/normalize: dry_run=true returns a projection inline
  (group/collision/rename counts + sample); dry_run=false enqueues the task.

Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup
tab) — preview → apply (polls the activity dashboard to terminal status),
behind a back-up-first warning. admin store gains normalizeTags().

Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag
dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept
separate, ML-known loser keeps a protective alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

style(tags): C416 — use dict(members) instead of identity comprehension (#714) ... 6332ae13fd

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit ec3d27b219 into main 2026-06-06 16:49:01 -04:00

bvandeusen referenced this issue from a commit 2026-06-17 21:57:23 -04:00

feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)

bvandeusen referenced this pull request 2026-06-17 22:01:02 -04:00

feat(maintenance): reconcile duplicate posts (gallery-dl→native unify) #124

bvandeusen referenced this issue from a commit 2026-06-21 22:46:45 -04:00

fix(reconcile): preserve from_attachment_id when merging duplicate posts (#73/#87)

bvandeusen referenced this pull request 2026-06-22 20:22:48 -04:00

Provenance archive linkage, post reconciliation close-out, and the cleanup/admin DRY pass #127

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#73