perf(agent): batch SigLIP crop embeds per image + load truncated images
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m26s

Two issues surfaced by the live logs (GPU pegged at ~0% util, 0.5 jobs/s,
truncated-image failures):

- BATCH the SigLIP embeds: collect all of an image's crops (figure + booru_yolo
  components + panels) and embed them in ONE forward pass instead of one
  forward+lock per crop. The per-crop path serialised every crop through the
  inference lock and starved the GPU (≈0% util, autoscaler stuck oscillating);
  batching gives a real GPU-bound workload + far higher throughput. CCIP still
  runs per figure inline.
- LOAD_TRUNCATED_IMAGES in the agent (matches the server embedder): slightly-
  truncated scraped images now load instead of failing the job 3× then erroring
  ("image file is truncated (N bytes not processed)").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-30 18:47:33 -04:00
parent 9eaefac385
commit 2713c3f773
3 changed files with 44 additions and 29 deletions
+6 -1
View File
@@ -6,7 +6,12 @@ import os
import subprocess
import tempfile
from PIL import Image
from PIL import Image, ImageFile
# Load slightly-truncated images (a few missing trailing bytes) instead of
# raising — matches the server embedder. These are common in scraped libraries
# and would otherwise fail the job 3× then error (operator-flagged 2026-06-30).
ImageFile.LOAD_TRUNCATED_IMAGES = True
def is_video(mime: str) -> bool: