fix: the ML dial offered slots the machine had no cores to feed (4295)

Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in
flight by Little's law, and the daily CCIP sweep dying on its 1800s soft
limit in a numpy matmul. The billiard/pool.py frame in that traceback is
the soft-timeout signal handler, not a pool fault.

Two causes, both mine.

1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile
   `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading
   "keep N_replicas x this within the cores allotted to ML" — a constraint
   stated where nothing could act on it. A large-memory host offered ~49
   slots, the operator took what the dial offered, and the lane asked the
   box for ~200 torch threads.

   The number moves onto the lane as `threads_per_slot`, the embedder
   reads it rather than restating it, and the ceiling is now the smaller
   of the two bounds. They fail differently on purpose: too little memory
   is honestly zero, because the first task would OOM the container; too
   few cores is merely slow, so it floors at one rather than making the
   lane unreachable on a small box.

2. `scheduled_ccip_auto_apply` scored one image per matmul, over every
   image in the library, on every daily run — ~119k products each too
   small to pay for its own BLAS setup. `char_maxima` does the same
   arithmetic in blocks bounded by elements, so its memory stays flat as
   either axis grows.

   Batching changes no arithmetic: a character's score for an image is a
   max over that image's figures AND that character's prototypes, and max
   does not care how it is grouped. Pinned against the old loop written
   out longhand, and against itself with the blocking forced to split
   every row.

The UI copy said the ML ceiling came from memory; it says cores or
memory, whichever runs out first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/services/worker_lanes.py

@@ -139,8 +139,22 @@ class Lane:
     # tells them to raise it rather than quietly consuming the machine.
     default_slots_cap: int
     # True when a slot costs a copy of the ML model rather than just a process.
-    # The only lane whose ceiling is decided by memory instead of by cores.
+    # Such a lane is bounded by memory AS WELL AS by cores, never instead of.
     memory_bound: bool = False
+    # CPU threads ONE slot uses. More than one for a lane whose work is an
+    # inference library with its own thread pool: `services/ml/embedder.py`
+    # calls `torch.set_num_threads` with this number, so a slot is four cores'
+    # worth of demand rather than one process's.
+    #
+    # It lives here because the CEILING has to know it. It was a private
+    # constant in the embedder with a comment saying "keep N_replicas x this
+    # within the cores allotted to ML" — a rule stated where nothing could
+    # enforce it. Nothing did: the ML ceiling was computed from memory alone,
+    # so a large-memory host offered ~49 slots, the operator took them, and
+    # 2026-09-23's log shows ~200 torch threads fighting over the box —
+    # embeds at 107-246s each, and the daily CCIP sweep sharing that pool
+    # timing out at 1800s.
+    threads_per_slot: int = 1
     # Models this lane downloads the first time it is enabled. Empty for every
     # lane that needs none, which is how the UI knows whether to warn at all.
     models: tuple[ModelRequirement, ...] = ()
@@ -209,6 +223,7 @@ LANES: tuple[Lane, ...] = (
         entrypoint_role="ml-worker",
         default_slots_cap=0,
         memory_bound=True,
+        threads_per_slot=4,
         models=(SIGLIP_MODEL,),
         optional=True,
     ),
@@ -370,6 +385,21 @@ def lane_for_node(hostname: str) -> Lane | None:
     return LANES_BY_NAME.get(hostname.split("@", 1)[0])
 
 
+def _cpu_bound_slots(lane: Lane) -> int:
+    """How many slots this container's cores can feed, at `threads_per_slot`.
+
+    Never zero: a machine with fewer cores than one slot wants still runs the
+    lane, just slowly. That is a real trade an operator may want, and refusing
+    to offer the lane at all on a small box would make ML unreachable there —
+    unlike the memory bound, where the honest answer IS zero, because the
+    first task would OOM the container rather than merely be slow.
+    """
+    cores = container_cpu_count()
+    if cores is None:
+        return UNKNOWN_CEILING
+    return max(MIN_CEILING, cores // lane.threads_per_slot)
+
+
 def derived_ceiling(lane: Lane) -> int:
     """The most slots `lane` may be given on this container.
 
@@ -377,26 +407,31 @@ def derived_ceiling(lane: Lane) -> int:
     is bounded by what it has NOW rather than by what it had when its row was
     written.
     """
-    if lane.memory_bound:
-        total = container_memory_bytes()
-        if total is None:
-            log.warning(
-                "worker_lanes: cannot read a memory limit; capping %s at %d",
-                lane.name, UNKNOWN_CEILING,
-            )
-            return UNKNOWN_CEILING
-        usable = total - RESERVED_BYTES
-        if usable < ML_BYTES_PER_SLOT:
-            # Honestly zero. A box that cannot hold one model alongside the web
-            # process must be told it cannot run tagging, not sold a slot that
-            # will OOM the container the first time it is used.
-            return 0
-        return int(usable // ML_BYTES_PER_SLOT)
+    by_cpu = _cpu_bound_slots(lane)
+    if not lane.memory_bound:
+        return by_cpu
 
-    cores = container_cpu_count()
-    if cores is None:
+    total = container_memory_bytes()
+    if total is None:
+        log.warning(
+            "worker_lanes: cannot read a memory limit; capping %s at %d",
+            lane.name, UNKNOWN_CEILING,
+        )
         return UNKNOWN_CEILING
-    return max(MIN_CEILING, cores)
+    usable = total - RESERVED_BYTES
+    if usable < ML_BYTES_PER_SLOT:
+        # Honestly zero. A box that cannot hold one model alongside the web
+        # process must be told it cannot run tagging, not sold a slot that
+        # will OOM the container the first time it is used.
+        return 0
+
+    # BOTH bounds, whichever binds first. Memory alone was the whole answer
+    # until 2026-09-23, and on a large-memory host that is the wrong one: RAM
+    # said ~49 slots, and each of those slots wants `threads_per_slot` cores.
+    # The operator raised the cap to what the dial offered and the lane
+    # starved itself — a control is not allowed to offer a number the machine
+    # cannot feed.
+    return min(int(usable // ML_BYTES_PER_SLOT), by_cpu)
 
 
 def ceilings() -> dict[str, int]: