feat: one image for every lane, with the model fetch gated on enabling (4296)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 23s
CI / backend-lint-and-test (push) Failing after 31s
CI / integration (push) Successful in 2m16s
Build images / build-ml (push) Successful in 3m8s
Build images / build-web (push) Successful in 3m16s
Build images / smoke-web (push) Skipped
Build images / promote (push) Skipped
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 23s
CI / backend-lint-and-test (push) Failing after 31s
CI / integration (push) Successful in 2m16s
Build images / build-ml (push) Successful in 3m8s
Build images / build-web (push) Successful in 3m16s
Build images / smoke-web (push) Skipped
Build images / promote (push) Skipped
Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch,
torchvision, transformers, onnxruntime and opencv, and serves every lane.
WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second
image would mean the `ml` lane could never be enabled from the UI — there
would be no worker in that container to enable. The switch needs something to
switch.
THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran
download_models before celery started, so every boot of that role reached
HuggingFace for ~3.5GB — a startup dependency on a third party for a feature
the operator may never use. Rule 164 permits a runtime fetch only for
something "optional and clearly off", so the fetch is now a TASK, enqueued
the moment the lane is ENABLED.
Being a task is what makes it visible: it gets a TaskRun row, so the download
shows in Activity with a duration and a status, and a failure is something an
operator can see and retry rather than a container that quietly never became
useful. Idempotent, so re-enabling a provisioned lane costs one no-op.
Enqueued only when the lane actually came ON (`enabled is True`, not the
resolved value) so re-saving slots does not re-fetch, and only when the
consumer change landed — a task queued onto a queue nothing consumes would
sit pending with no explanation.
`fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The
operator's Swarm stack references that name and lives outside this repo;
dropping it would not break their deploy, it would freeze it silently at the
last publish — the exact failure class this milestone keeps finding. Retiring
the NAME is its own task, gated on that stack moving. Same two-phase shape
#406 used for pixiv.
THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather
than assuming the build was the only consumer:
- `docker-compose.override.yml` built the ml service from it (contributor
path would have failed at `docker compose build`).
- `tests/test_artifact_paths.py` pins the ml path set.
- `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file
silently stops contributing to the derived revision — which the reuse check
and the version string both read. That is #3202's recorded shape.
The `--with-ml` flag is gone from the generator and the healthcheck rather
than left defaulting to true. One image carries every lane now, so a flag
that can only be passed one way is a branch pretending to be a choice.
The advisory shipped in ecbd325 is what makes this honest to an adopter: the
lane says it is optional, names the model, and gives its download and
per-slot RAM before the switch is thrown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -408,6 +408,20 @@ async def set_lane(
|
||||
if applied and slots is not None:
|
||||
applied, error = await asyncio.to_thread(set_lane_slots_sync, lane, new_slots)
|
||||
|
||||
# Enabling a lane that needs models is what triggers the fetch (milestone
|
||||
# 422 step 6). Never at boot: that made every start of the ML role reach
|
||||
# HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a
|
||||
# feature that is optional and clearly OFF.
|
||||
#
|
||||
# Only when the lane actually came on — `enabled is True` rather than
|
||||
# `new_enabled`, so re-saving slots on an already-enabled lane does not
|
||||
# re-enqueue. And only when the consumer change landed: enqueueing a task
|
||||
# onto a queue nothing is consuming would leave it pending with no
|
||||
# explanation until the lane returns.
|
||||
fetching = False
|
||||
if enabled is True and lane.models and applied:
|
||||
fetching = _enqueue_model_fetch()
|
||||
|
||||
return {
|
||||
"name": lane.name,
|
||||
"slots": row.slots,
|
||||
@@ -416,9 +430,34 @@ async def set_lane(
|
||||
"enabled": row.enabled,
|
||||
"applied": applied,
|
||||
"apply_error": error,
|
||||
# Tells the card to say a download has started rather than leaving the
|
||||
# operator to wonder why a freshly enabled lane is busy.
|
||||
"fetching_models": fetching,
|
||||
}
|
||||
|
||||
|
||||
def _enqueue_model_fetch() -> bool:
|
||||
"""Queue the model download. Returns whether it was accepted.
|
||||
|
||||
Import inside the function: `backend.app.tasks.ml` pulls in torch, and web
|
||||
must not pay that import cost on a module that every settings request
|
||||
touches.
|
||||
|
||||
Never raises. A broker that will not take the task is worth reporting, but
|
||||
the SETTING has already been stored and the lane is already enabled — so
|
||||
failing the whole request here would roll back nothing and tell the
|
||||
operator their change did not happen when it did.
|
||||
"""
|
||||
try:
|
||||
from ..tasks.ml import ensure_models
|
||||
|
||||
ensure_models.delay()
|
||||
return True
|
||||
except Exception: # noqa: BLE001 — reported, never raised at a caller
|
||||
log.warning("worker_control: could not enqueue the model fetch", exc_info=True)
|
||||
return False
|
||||
|
||||
|
||||
def reconcile_lanes_sync(desired: dict[str, tuple[int, bool]]) -> dict:
|
||||
"""Drive every RUNNING lane to its stored slots and enabled flag.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user