feat(agent): temporal video dedup — drop near-duplicate frames before the GPU
Near-static videos are the dominant GPU load: sampled into up to 64 frames, each re-runs the whole detect→CCIP→SigLIP chain on ~identical content. Add a CPU perceptual-hash frame dedup upstream of the GPU so the redundant frames are never processed at all (not just their embeds). - media.dedupe_frames() + _dhash(): 8×8 difference-hash (64-bit) per frame; greedy keep — a frame survives only if its hash differs from every kept frame by >= min_distance bits (Hamming). A static run collapses to one frame; genuinely distinct scenes all survive. Order + frame_time preserved. - Called in worker._download_decode right after sample_frames, so it runs in the decode stage on the downloader thread (CPU) — the GPU consumers only ever see deduped frames, and buffered video items shrink (less RAM too). - Env-tunable FRAME_DEDUPE_DISTANCE (default 8; higher keeps more frames for brief localized changes an 8×8 hash can miss; 0 disables). Logs `video frames N→M` when it drops any, so video load reduction is visible. Complements the spatial per-frame crop dedup (2026-07-01.2); this is the temporal axis. Build marker 2026-07-01.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -25,6 +25,47 @@ def is_video(mime: str) -> bool:
|
||||
return bool(mime) and (mime.startswith("video/") or mime in {"image/gif"})
|
||||
|
||||
|
||||
def _dhash(img: Image.Image, size: int = 8) -> int:
|
||||
"""Difference hash: compare adjacent pixels of a (size+1 × size) grayscale
|
||||
thumbnail → a `size*size`-bit fingerprint. Cheap (64 comparisons on a 72-px
|
||||
thumbnail) and robust to scaling/compression noise — near-identical frames
|
||||
hash within a few bits, a real scene change moves many."""
|
||||
small = img.convert("L").resize((size + 1, size))
|
||||
px = list(small.getdata())
|
||||
bits = 0
|
||||
for row in range(size):
|
||||
base = row * (size + 1)
|
||||
for col in range(size):
|
||||
bits = (bits << 1) | int(px[base + col] > px[base + col + 1])
|
||||
return bits
|
||||
|
||||
|
||||
def dedupe_frames(
|
||||
frames: list[tuple[float, Image.Image]], min_distance: int
|
||||
) -> list[tuple[float, Image.Image]]:
|
||||
"""Drop visually near-duplicate frames. A near-static video sampled into many
|
||||
frames re-runs the WHOLE detect→CCIP→SigLIP chain on ~identical frames — the
|
||||
dominant video load. Greedy perceptual-hash dedup: keep a frame only if its
|
||||
dHash differs from every already-kept frame by >= min_distance bits (Hamming),
|
||||
so a static run collapses to one frame while genuinely distinct scenes all
|
||||
survive. Order + timestamps preserved. CPU-only (64-bit int XORs), so it runs
|
||||
in the decode stage and spares the GPU the skipped frames entirely.
|
||||
|
||||
min_distance is the coarseness dial: higher keeps more frames (safer for brief
|
||||
localized changes an 8×8 hash can miss), 0 disables. The first frame is always
|
||||
kept (nothing to compare against)."""
|
||||
if min_distance <= 0 or len(frames) <= 1:
|
||||
return frames
|
||||
kept: list[tuple[float, Image.Image]] = []
|
||||
hashes: list[int] = []
|
||||
for t, frame in frames:
|
||||
h = _dhash(frame)
|
||||
if all(bin(h ^ k).count("1") >= min_distance for k in hashes):
|
||||
hashes.append(h)
|
||||
kept.append((t, frame))
|
||||
return kept
|
||||
|
||||
|
||||
def to_rgb(img: Image.Image) -> Image.Image:
|
||||
"""RGB, flattening any transparency onto white first. A naive convert('RGB')
|
||||
on a palette-with-transparency image (common for character PNGs on a clear
|
||||
|
||||
Reference in New Issue
Block a user