refactor(ml): drop GPU code, cap inference threads by default (#747/#872)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m18s

GPU enablement (#872) cancelled — not worth the Pascal-specific build for a
modest CPU→GPU win on an old P4. Remove the dead GPU code (device.py, the CUDA
provider branch in tagger, the .to('cuda') path in embedder) so nothing carries
it forward.

Instead, bound CPU inference threads by default so the ml-worker is a predictable
core consumer on a SHARED node — the intended scaling model is multiple worker
replicas (each --concurrency=1, each its own cgroup limit), not one big
container. ONNX Runtime and torch otherwise size their thread pools to ALL host
cores, so each replica would grab every core and oversubscribe / starve the
co-located DB+web. Cap both to _INTRA_OP_THREADS=4 (matches the prior per-worker
cpus:4 unit): run N replicas where N×4 stays within the cores allotted to ML.

- tagger: ort.SessionOptions().intra_op_num_threads = 4 (CPUExecutionProvider).
- embedder: torch.set_num_threads(4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 13:39:55 -04:00
parent db7e1f2b59
commit 60a9c9e6ef
4 changed files with 31 additions and 108 deletions
-31
View File
@@ -1,31 +0,0 @@
"""ML device-selection env parsing (#872). Pure logic — no models/GPU/DB."""
from backend.app.services.ml import device
def test_gpu_requested_default_is_auto(monkeypatch):
monkeypatch.delenv("FC_ML_DEVICE", raising=False)
assert device.gpu_requested() is True
def test_gpu_requested_modes(monkeypatch):
for v in ("auto", "cuda", "gpu", "CUDA", " Auto "):
monkeypatch.setenv("FC_ML_DEVICE", v)
assert device.gpu_requested() is True
for v in ("cpu", "CPU", "none", "0"):
monkeypatch.setenv("FC_ML_DEVICE", v)
assert device.gpu_requested() is False
def test_onnx_gpu_mem_bytes(monkeypatch):
monkeypatch.delenv("FC_ML_ONNX_GPU_MEM_GB", raising=False)
assert device.onnx_gpu_mem_bytes() == 3 * 1024 ** 3
monkeypatch.setenv("FC_ML_ONNX_GPU_MEM_GB", "2")
assert device.onnx_gpu_mem_bytes() == 2 * 1024 ** 3
def test_torch_mem_fraction(monkeypatch):
monkeypatch.delenv("FC_ML_TORCH_MEM_FRACTION", raising=False)
assert device.torch_mem_fraction() == 0.6
monkeypatch.setenv("FC_ML_TORCH_MEM_FRACTION", "0.5")
assert device.torch_mem_fraction() == 0.5