feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash
The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms), so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit) left the fast GPU idle during every download. Rearchitect worker.py into a producer/consumer pipeline: downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU consumers (detect+embed→submit) - Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a 2nd consumer if it has util/VRAM headroom and lifts throughput, else trim). - Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping RAM + lease look-ahead. VRAM pressure sheds a consumer immediately. - Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU; curator's 180s TTL would otherwise reclaim them mid-buffer). - Preserves all resilience: lease exp-backoff, submit-path retry (#169), release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools and releases every held lease at once (single held-set as source of truth). - Consumers SHARE one embedder + proposers instance (a 2nd consumer adds concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots). - UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU · processed · errors, a buffer-occupancy meter, and a consumers/waited-out line; the dial now tunes downloaders. Build marker 2026-07-01.1. Also fix the operator-flagged detector warning: yolo11n + the comic-panel model threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load- time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3 crop proposers and spamming the log per image. Disable that fusion (unfused inference is correct, marginally slower) and permanently self-disable a proposer on the first inference failure instead of re-throwing forever. Refs milestone 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -17,6 +17,7 @@ cached under HF_HOME so the download happens once.
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger("fc_agent.detectors")
|
||||
@@ -93,6 +94,18 @@ class YoloProposer:
|
||||
self._ok = False
|
||||
return
|
||||
self._model = YOLO(path)
|
||||
# Disable ultralytics' load-time Conv+BN fusion. AutoBackend fuses
|
||||
# the graph on the first predict; some checkpoints (yolo11n, the
|
||||
# comic-panel model) crash that step with "'Conv' object has no
|
||||
# attribute 'bn'" (a partially-fused / version-mismatched graph),
|
||||
# which silently disabled those proposers (operator-flagged
|
||||
# 2026-07-01). Unfused inference is correct — only marginally
|
||||
# slower — and this is robust across ultralytics versions; if a
|
||||
# future version ignores the override, the detect() guard below
|
||||
# still self-disables the proposer instead of spamming per image.
|
||||
inner = getattr(self._model, "model", None)
|
||||
if inner is not None:
|
||||
inner.fuse = types.MethodType(lambda self, *a, **k: self, inner)
|
||||
log.info("detector %s loaded (%s)", self.name, path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("detector %s disabled (load failed): %s", self.name, exc)
|
||||
@@ -105,7 +118,12 @@ class YoloProposer:
|
||||
try:
|
||||
res = self._model.predict(image, conf=self._conf, verbose=False)[0]
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("detector %s inference failed: %s", self.name, exc)
|
||||
# Permanently self-disable on the FIRST inference failure rather than
|
||||
# re-throwing (and re-logging) on every image forever — an unfixable
|
||||
# model fault degrades to "this proposer is off", logged once.
|
||||
log.warning("detector %s disabled (inference failed): %s", self.name, exc)
|
||||
self._ok = False
|
||||
self._model = None
|
||||
return []
|
||||
iw, ih = image.size
|
||||
names = getattr(res, "names", None) or {}
|
||||
|
||||
Reference in New Issue
Block a user