Tag-hover → crop-region grounding overlay (#1206, milestone #133) #195

Merged

bvandeusen merged 5 commits from dev into main 2026-07-06 13:29:31 -04:00

Conversation 0 Commits 5 Files Changed 14

+494 -41

bvandeusen commented 2026-07-06 13:29:25 -04:00

Owner

Copy Link

Copy Source

Hovering a tag highlights the exact crop region on the image that explains it — visual proof the crop→SigLIP bag is firing. Hover a concept → its crop lights up (a proposer region won); the whole-image frame → the global vector won (not localized).

Steps

1. score_image (heads.py) keeps the ARGMAX beside the max-over-bag → each surfaced tag carries grounding {bbox, kind, detector} (or null when the whole-image vector won). Threaded through SuggestionService + /suggestions payload.
2. CCIP character/identity matches ground to the matched figure region's bbox.
3. Frontend: ImageCanvas overlay tracks the rendered image rect (correct under object-fit + pan/zoom) and draws the region; suggestion rows hover→overlay via a fcSuggestionHover provide/inject, in both the modal and Explore. Null grounding → a subtle whole-image frame.
4. Applied-tag chips too: on-demand GET /api/images/<id>/tags/<id>/grounding runs the applied tag's head over the same bag (argmax) → {grounding, has_head}; chips reuse the Step-3 overlay. has_head=false (nothing to localize with) → no overlay.

No migration — this is a read-path + UI change. CI green on dev head 2638cf1 (run 1962): ruff, frontend build (npm test:unit + build), backend unit, and integration all pass.

🤖 Generated with Claude Code

Hovering a tag highlights the exact crop region on the image that explains it — visual proof the crop→SigLIP bag is firing. Hover a concept → its crop lights up (a proposer region won); the whole-image frame → the global vector won (not localized). **Steps** 1. `score_image` (heads.py) keeps the ARGMAX beside the max-over-bag → each surfaced tag carries `grounding {bbox, kind, detector}` (or null when the whole-image vector won). Threaded through SuggestionService + `/suggestions` payload. 2. CCIP character/identity matches ground to the matched figure region's bbox. 3. Frontend: `ImageCanvas` overlay tracks the rendered image rect (correct under object-fit + pan/zoom) and draws the region; suggestion rows hover→overlay via a `fcSuggestionHover` provide/inject, in both the modal and Explore. Null grounding → a subtle whole-image frame. 4. Applied-tag chips too: on-demand `GET /api/images/<id>/tags/<id>/grounding` runs the applied tag's head over the same bag (argmax) → `{grounding, has_head}`; chips reuse the Step-3 overlay. `has_head=false` (nothing to localize with) → no overlay. No migration — this is a read-path + UI change. CI green on dev head `2638cf1` (run 1962): ruff, frontend build (npm test:unit + build), backend unit, and integration all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 5 commits 2026-07-06 13:29:26 -04:00

feat(ml): argmax grounding in score_image → suggestions carry the winning crop (#133 step 1) ... 409724b981

score_image now keeps the ARGMAX beside the max-over-bag: which bag row won each
head. The region query also selects bbox/kind/detector_version, a parallel
bag_meta maps each row → its region (None for the whole-image vector), and every
hit gains grounding {bbox,kind,detector} (null when the global vector won). Threaded
through SuggestionService (new Suggestion.grounding field) → /api/.../suggestions
payload. This is the data the #1206 hover-overlay draws. CCIP-only hits ground null
for now (figure grounding = step 2). Tests: winning crop grounds the tag with its
bbox+kind; whole-image win → grounding None.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ml): CCIP character matches ground to the matched figure region (#133 step 2) ... dfe2fda564

match_image now tracks WHICH query figure produced the winning cosine per
character (argmax over the per-figure best-reference sim) and attaches its bbox as
grounding {bbox,kind:'figure',detector}. SuggestionService carries it: a CCIP-only
character hit grounds to its figure; a 'both' hit keeps the head's localized crop
if it had one, else falls back to the CCIP figure — so corroborated characters
stay grounded. Test: a character match carries the matched figure's bbox+kind.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ui): hover a suggestion → highlight the crop region it came from (#133 step 3) ... 524a26c618

The payoff: hover a suggestion in the rail and the exact crop that produced it
lights up on the image (a booru:head, a panel, a figure); a null-grounding tag
shows a subtle dashed whole-image frame ('global vector won, not a crop').
ImageCanvas gains a grounding overlay that tracks the <img>'s live bounding rect
(correct under object-fit letterboxing + pan/zoom) and draws the normalized bbox
+ a detector/kind label. SuggestionItem sets the hovered grounding via
provide/inject (no 4-level event relay through TagPanel/SuggestionsPanel/group);
ImageViewer AND ExploreView provide it + pass it to their canvas. Overlay is
pointer-events:none so it never blocks pan/zoom/click. Videos out of v1 scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

feat(ui): hover an applied tag chip → highlight its grounding crop (#133 step 4) ... 9bb4211722

Applied tags aren't scored live, so compute the grounding on demand: run the
tag's head over the image's max-over-bag (whole-image + concept crops), argmax
→ the region that best explains the tag on this image, mirroring what
score_image records for live suggestions.

- heads.py: extract _image_bag (now shared by score_image) + ground_applied_tag.
  Returns (grounding, has_head): has_head False = no head to localize with →
  no overlay; grounding None = the whole-image vector won → whole-image frame.
- tags.py: GET /api/images/<id>/tags/<id>/grounding → {grounding, has_head}.
- TagChip/TagPanel: applied chips inject fcSuggestionHover and fetch grounding
  on hover (cached per image+tag, race-guarded), reusing Step 3's overlay in
  both the modal and Explore. No new frontend overlay code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

style(tests): hoist test_api_tags grounding imports to module level (ruff I001) ... 2638cf1a35

CI ruff flagged the in-function import block; move them to the top mirroring
test_ml_suggestions.py's import line, which ruff already accepts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

bvandeusen merged commit a66a695977 into main 2026-07-06 13:29:31 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#195