CCIP characters + crop/region pipeline + desktop GPU agent (#114) #144
Assignees
bvandeusen
Clear assignees
bvandeusen
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: bvandeusen/FabledCurator#144
Reference in New Issue
Block a user
Delete Branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The full cross-artist character (CCIP) + crop pipeline, server side complete, plus the desktop GPU agent. Builds on the heads/auto-apply milestone.
Server side (all CI-green)
image_regionstorage (migration 0061, video-ready viaframe_time) — shared by CCIP figure-crops and SigLIP concept-crops.gpu_jobqueue (migration 0062) + HTTP job API (/api/gpu/jobs/lease|submit|heartbeat|fail, bearer-token) so the desktop agent stays HTTP-only — Redis/Postgres never exposed.both/ccip/head— complementary, not exclusive).GET /api/ccip/overview+/api/ccip/images/<id>(coverage, per-character references, per-image regions + matches).Desktop agent (
agent/, outside CI — verified by running)A Dockerised GPU worker: lease → fetch pixels → detect figures + CCIP-embed (ffmpeg-sampled frames for video) → submit, with a localhost control UI (start/pause/stop + progress). See
agent/README.md.Also in this batch
Same-name-character-in-different-fandom fix, Explore video rendering fix.
Deploy
Migrations 0061 + 0062 run automatically. After deploy: GPU agent card → generate token → build + run the agent (
--gpus all) → verify the imgutils API seams infc_agent/models.py→ Queue character embedding.🤖 Generated with Claude Code
The Explore center pane hardcoded ImageCanvas, so a video anchor (e.g. a 169 MB MP4) tried to load the MP4 into an <img> and showed only the alt text — the thumbnail worked but the "main image" never rendered. Branch on mime.startsWith('video/') to VideoCanvas (with mime), exactly like the image modal. The anchor payload already carries mime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqaThe storage backbone both crop jobs write to and read from. image_region = normalized bbox (rx/ry/rw/rh) + kind ('face'/'figure' → CCIP character id; 'concept' → SigLIP head bag) + the crop's embedding (nullable Vector(768) CCIP / Vector(1152) SigLIP, one per kind) + version stamps for compute-once gating. The bbox doubles as grounded-tag provenance. Migration 0061. RegionService.replace_regions (scoped BY KIND so the figure + concept pipelines don't clobber each other) + get_regions — the GPU agent's results endpoint will call the writer; the character matcher + bag scorer read. Server-side, no GPU. Tests: replace/get round-trip, kind-scoped replacement, CCIP vector round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqaThe server-side brain that turns stored CCIP vectors into character suggestions — no GPU. character_references() gathers each character tag's prototype vectors (figure/face-region CCIP embeddings on images carrying that tag); match_image() cosine-matches an image's figure vectors against every character (multi- prototype: best over a character's examples), surfacing those above a tunable threshold as {tag_id, name, category:'character', score, source:'ccip'}, excluding already-applied characters. v1 = cosine on raw CCIP vectors; the exact CCIP metric/threshold gets validated against the model in the hands-on eval. Tests (synthetic vectors): same-character match across images, no-match for an orthogonal figure, already-applied exclusion, no-figure-vectors empty. NEXT: merge CCIP character suggestions into the rail; the agent container that actually produces the vectors (hands-on, GPU — not CI-verifiable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa