Tag-hover → crop-region grounding overlay (#1206, milestone #133) #195

Merged
bvandeusen merged 5 commits from dev into main 2026-07-06 13:29:31 -04:00

5 Commits

Author SHA1 Message Date
bvandeusen 2638cf1a35 style(tests): hoist test_api_tags grounding imports to module level (ruff I001)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m43s
CI ruff flagged the in-function import block; move them to the top mirroring
test_ml_suggestions.py's import line, which ruff already accepts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-06 13:24:55 -04:00
bvandeusen 9bb4211722 feat(ui): hover an applied tag chip → highlight its grounding crop (#133 step 4)
CI / lint (push) Failing after 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m41s
Applied tags aren't scored live, so compute the grounding on demand: run the
tag's head over the image's max-over-bag (whole-image + concept crops), argmax
→ the region that best explains the tag on this image, mirroring what
score_image records for live suggestions.

- heads.py: extract _image_bag (now shared by score_image) + ground_applied_tag.
  Returns (grounding, has_head): has_head False = no head to localize with →
  no overlay; grounding None = the whole-image vector won → whole-image frame.
- tags.py: GET /api/images/<id>/tags/<id>/grounding → {grounding, has_head}.
- TagChip/TagPanel: applied chips inject fcSuggestionHover and fetch grounding
  on hover (cached per image+tag, race-guarded), reusing Step 3's overlay in
  both the modal and Explore. No new frontend overlay code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-06 13:19:41 -04:00
bvandeusen 524a26c618 feat(ui): hover a suggestion → highlight the crop region it came from (#133 step 3)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 3m36s
The payoff: hover a suggestion in the rail and the exact crop that produced it
lights up on the image (a booru:head, a panel, a figure); a null-grounding tag
shows a subtle dashed whole-image frame ('global vector won, not a crop').
ImageCanvas gains a grounding overlay that tracks the <img>'s live bounding rect
(correct under object-fit letterboxing + pan/zoom) and draws the normalized bbox
+ a detector/kind label. SuggestionItem sets the hovered grounding via
provide/inject (no 4-level event relay through TagPanel/SuggestionsPanel/group);
ImageViewer AND ExploreView provide it + pass it to their canvas. Overlay is
pointer-events:none so it never blocks pan/zoom/click. Videos out of v1 scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-05 23:24:18 -04:00
bvandeusen dfe2fda564 feat(ml): CCIP character matches ground to the matched figure region (#133 step 2)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m42s
match_image now tracks WHICH query figure produced the winning cosine per
character (argmax over the per-figure best-reference sim) and attaches its bbox as
grounding {bbox,kind:'figure',detector}. SuggestionService carries it: a CCIP-only
character hit grounds to its figure; a 'both' hit keeps the head's localized crop
if it had one, else falls back to the CCIP figure — so corroborated characters
stay grounded. Test: a character match carries the matched figure's bbox+kind.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-05 23:18:41 -04:00
bvandeusen 409724b981 feat(ml): argmax grounding in score_image → suggestions carry the winning crop (#133 step 1)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m41s
score_image now keeps the ARGMAX beside the max-over-bag: which bag row won each
head. The region query also selects bbox/kind/detector_version, a parallel
bag_meta maps each row → its region (None for the whole-image vector), and every
hit gains grounding {bbox,kind,detector} (null when the global vector won). Threaded
through SuggestionService (new Suggestion.grounding field) → /api/.../suggestions
payload. This is the data the #1206 hover-overlay draws. CCIP-only hits ground null
for now (figure grounding = step 2). Tests: winning crop grounds the tag with its
bbox+kind; whole-image win → grounding None.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-05 23:13:29 -04:00