fix: the ML dial offered slots the machine had no cores to feed (4295)
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 1m7s
CI and images / promote (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m21s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m42s
CI and images / smoke-web (push) Successful in 1m7s
CI and images / promote (push) Skipped
Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in flight by Little's law, and the daily CCIP sweep dying on its 1800s soft limit in a numpy matmul. The billiard/pool.py frame in that traceback is the soft-timeout signal handler, not a pool fault. Two causes, both mine. 1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading "keep N_replicas x this within the cores allotted to ML" — a constraint stated where nothing could act on it. A large-memory host offered ~49 slots, the operator took what the dial offered, and the lane asked the box for ~200 torch threads. The number moves onto the lane as `threads_per_slot`, the embedder reads it rather than restating it, and the ceiling is now the smaller of the two bounds. They fail differently on purpose: too little memory is honestly zero, because the first task would OOM the container; too few cores is merely slow, so it floors at one rather than making the lane unreachable on a small box. 2. `scheduled_ccip_auto_apply` scored one image per matmul, over every image in the library, on every daily run — ~119k products each too small to pay for its own BLAS setup. `char_maxima` does the same arithmetic in blocks bounded by elements, so its memory stays flat as either axis grows. Batching changes no arithmetic: a character's score for an image is a max over that image's figures AND that character's prototypes, and max does not care how it is grouped. Pinned against the old loop written out longhand, and against itself with the blocking forced to split every row. The UI copy said the ML ceiling came from memory; it says cores or memory, whichever runs out first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -170,9 +170,9 @@
|
||||
FabledCurator to work at all; ML tagging is the one that is genuinely
|
||||
optional, and it ships at zero because switching it on downloads a
|
||||
model. <em>Up to N</em> beneath each dial is what this machine can
|
||||
hold — memory for ML tagging, processor cores for the rest — and it is
|
||||
recalculated from the container's real limits every time this page
|
||||
loads.
|
||||
hold — processor cores for every lane, and for ML tagging whichever
|
||||
runs out first, its cores or its memory. It is recalculated from the
|
||||
container's real limits every time this page loads.
|
||||
</p>
|
||||
<p class="mb-2">
|
||||
The shipped caps are one of each, which is right for a first boot and
|
||||
@@ -218,8 +218,9 @@
|
||||
</li>
|
||||
</ul>
|
||||
<p class="mb-0 text-caption">
|
||||
Each worker loads its own copy, which is why this machine allows it
|
||||
at most {{ lane.ceiling }}.
|
||||
Each worker loads its own copy and asks for several processor cores
|
||||
while it runs, which is why this machine allows it at most
|
||||
{{ lane.ceiling }}.
|
||||
<template v-if="lane.ceiling === 0">
|
||||
It has too little memory to run this at all.
|
||||
</template>
|
||||
|
||||
Reference in New Issue
Block a user