feat(agent): download/GPU producer-consumer pipeline + fix detector fuse crash
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m25s

The agent workload is download-bound (download 400–5462ms vs GPU ~300–600ms),
so the old N-slot serial chain (each slot: lease→download→decode→GPU→submit)
left the fast GPU idle during every download. Rearchitect worker.py into a
producer/consumer pipeline:

  downloader pool (autoscaled by BUFFER OCCUPANCY) → bounded queue → 1–2 GPU
  consumers (detect+embed→submit)

- Downloaders are I/O-bound → many overlap; the autoscaler now tunes DOWNLOADER
  count by buffer fill (empty = GPU starving → add; full = outpacing GPU → add a
  2nd consumer if it has util/VRAM headroom and lifts throughput, else trim).
- Bounded buffer (12) = backpressure: a full buffer blocks downloaders, capping
  RAM + lease look-ahead. VRAM pressure sheds a consumer immediately.
- Heartbeat thread keeps every held lease alive (buffered jobs wait on the GPU;
  curator's 180s TTL would otherwise reclaim them mid-buffer).
- Preserves all resilience: lease exp-backoff, submit-path retry (#169),
  release-on-stop, region caps + video early-exit (#171). Stop drains BOTH pools
  and releases every held lease at once (single held-set as source of truth).
- Consumers SHARE one embedder + proposers instance (a 2nd consumer adds
  concurrent inference, not N× VRAM — bounds the VRAM creep seen with N slots).
- UI reworked for the pipeline: tiles show downloaders · buffer · on-GPU ·
  processed · errors, a buffer-occupancy meter, and a consumers/waited-out line;
  the dial now tunes downloaders. Build marker 2026-07-01.1.

Also fix the operator-flagged detector warning: yolo11n + the comic-panel model
threw "'Conv' object has no attribute 'bn'" on every image (ultralytics' load-
time Conv+BN fusion on a version-mismatched graph), silently disabling 2 of 3
crop proposers and spamming the log per image. Disable that fusion (unfused
inference is correct, marginally slower) and permanently self-disable a proposer
on the first inference failure instead of re-throwing forever.

Refs milestone 122.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-30 23:34:12 -04:00
parent 83f1070a11
commit afef95a87d
3 changed files with 478 additions and 273 deletions
+19 -1
View File
@@ -17,6 +17,7 @@ cached under HF_HOME so the download happens once.
import logging
import os
import threading
import types
from pathlib import Path
log = logging.getLogger("fc_agent.detectors")
@@ -93,6 +94,18 @@ class YoloProposer:
self._ok = False
return
self._model = YOLO(path)
# Disable ultralytics' load-time Conv+BN fusion. AutoBackend fuses
# the graph on the first predict; some checkpoints (yolo11n, the
# comic-panel model) crash that step with "'Conv' object has no
# attribute 'bn'" (a partially-fused / version-mismatched graph),
# which silently disabled those proposers (operator-flagged
# 2026-07-01). Unfused inference is correct — only marginally
# slower — and this is robust across ultralytics versions; if a
# future version ignores the override, the detect() guard below
# still self-disables the proposer instead of spamming per image.
inner = getattr(self._model, "model", None)
if inner is not None:
inner.fuse = types.MethodType(lambda self, *a, **k: self, inner)
log.info("detector %s loaded (%s)", self.name, path)
except Exception as exc: # noqa: BLE001
log.warning("detector %s disabled (load failed): %s", self.name, exc)
@@ -105,7 +118,12 @@ class YoloProposer:
try:
res = self._model.predict(image, conf=self._conf, verbose=False)[0]
except Exception as exc: # noqa: BLE001
log.warning("detector %s inference failed: %s", self.name, exc)
# Permanently self-disable on the FIRST inference failure rather than
# re-throwing (and re-logging) on every image forever — an unfixable
# model fault degrades to "this proposer is off", logged once.
log.warning("detector %s disabled (inference failed): %s", self.name, exc)
self._ok = False
self._model = None
return []
iw, ih = image.size
names = getattr(res, "names", None) or {}