feat: one image for every lane, with the model fetch gated on enabling (4296)

Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch,
torchvision, transformers, onnxruntime and opencv, and serves every lane.

WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second
image would mean the `ml` lane could never be enabled from the UI — there
would be no worker in that container to enable. The switch needs something to
switch.

THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran
download_models before celery started, so every boot of that role reached
HuggingFace for ~3.5GB — a startup dependency on a third party for a feature
the operator may never use. Rule 164 permits a runtime fetch only for
something "optional and clearly off", so the fetch is now a TASK, enqueued
the moment the lane is ENABLED.

Being a task is what makes it visible: it gets a TaskRun row, so the download
shows in Activity with a duration and a status, and a failure is something an
operator can see and retry rather than a container that quietly never became
useful. Idempotent, so re-enabling a provisioned lane costs one no-op.

Enqueued only when the lane actually came ON (`enabled is True`, not the
resolved value) so re-saving slots does not re-fetch, and only when the
consumer change landed — a task queued onto a queue nothing consumes would
sit pending with no explanation.

`fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The
operator's Swarm stack references that name and lives outside this repo;
dropping it would not break their deploy, it would freeze it silently at the
last publish — the exact failure class this milestone keeps finding. Retiring
the NAME is its own task, gated on that stack moving. Same two-phase shape
#406 used for pixiv.

THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather
than assuming the build was the only consumer:

- `docker-compose.override.yml` built the ml service from it (contributor
  path would have failed at `docker compose build`).
- `tests/test_artifact_paths.py` pins the ml path set.
- `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file
  silently stops contributing to the derived revision — which the reuse check
  and the version string both read. That is #3202's recorded shape.

The `--with-ml` flag is gone from the generator and the healthcheck rather
than left defaulting to true. One image carries every lane now, so a flag
that can only be passed one way is a branch pretending to be a choice.

The advisory shipped in ecbd325 is what makes this honest to an adopter: the
lane says it is optional, names the model, and gives its download and
per-slot RAM before the switch is thrown.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/scripts/gen_supervisord.py

@@ -36,12 +36,16 @@ puts tini in front of supervisord. Neither choice reaches the application —
 nothing in FC talks to the supervisor — so this is reversible without touching
 a line of product code.
 
-## What this does NOT start
+## Every lane, including ml
 
-The `ml` lane, unless `--with-ml` is passed. Until step 6 merges the images,
-torch and the ML requirements live only in `Dockerfile.ml`, so an `ml` program
-in the web image would fail to import on every restart forever. Step 6 is
-where that flag turns on.
+Step 6 merged the images, so this one carries torch and the ML requirements
+and the `ml` lane gets a program like any other. It starts at one slot with
+its consumers CANCELLED — `enabled=false` in the seeded settings — so it
+holds a process and no model. That matters: `add_consumer` needs a running
+worker to reach, and without one the UI switch would have nothing to switch.
+
+Nothing is downloaded by starting it. The model fetch is enqueued when the
+lane is enabled, which is what lets rule 164 permit a runtime fetch at all.
 """
 
 from __future__ import annotations
@@ -69,10 +73,6 @@ STOP_WAIT_SECONDS: dict[str, int] = {
 }
 DEFAULT_STOP_WAIT = 60
 
-# Lanes whose code is not in this image yet. See the module docstring.
-_NEEDS_ML_DEPS = frozenset({"ml"})
-
-
 def _program(lane: Lane, *, slots: int) -> str:
     """One [program:x] block.
 
@@ -141,7 +141,7 @@ def _web_program() -> str:
     ])
 
 
-def render(*, with_ml: bool = False) -> str:
+def render() -> str:
     parts = [
         "\n".join([
             "[supervisord]",
@@ -158,8 +158,6 @@ def render(*, with_ml: bool = False) -> str:
     ]
     # Lanes after web, in LANES order, so the log reads in a stable sequence.
     for lane in LANES:
-        if lane.name in _NEEDS_ML_DEPS and not with_ml:
-            continue
         # A lane configured at zero slots still gets a PROCESS, at one slot
         # with its consumers cancelled by the reconcile. Without a running
         # worker there is nothing for `add_consumer` to reach, so enabling the
@@ -171,12 +169,8 @@ def render(*, with_ml: bool = False) -> str:
 
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(description=__doc__)
-    ap.add_argument(
-        "--with-ml", action="store_true",
-        help="include the ml lane (only valid once the ML deps are in this image)",
-    )
-    args = ap.parse_args(argv)
-    sys.stdout.write(render(with_ml=args.with_ml))
+    ap.parse_args(argv)
+    sys.stdout.write(render())
     return 0
 
 

backend/app/scripts/healthcheck_all.py

@@ -47,14 +47,15 @@ def _web_ok() -> tuple[bool, str]:
         return False, f"web unreachable: {exc}"
 
 
-def _lanes_ok(*, with_ml: bool) -> tuple[bool, str]:
+def _lanes_ok() -> tuple[bool, str]:
     from ..services.worker_control import inspect_lanes_sync
     from ..services.worker_lanes import LANES
 
-    expected = {
-        lane.name for lane in LANES
-        if with_ml or lane.name != "ml"
-    }
+    # Every lane, ml included: one image carries them all since step 6, and a
+    # disabled lane still runs a process (consumers cancelled), so it answers
+    # inspect and is healthy. Health is "is the process alive"; whether it
+    # should be consuming is the reconcile's business.
+    expected = {lane.name for lane in LANES}
     live = inspect_lanes_sync()
     missing = sorted(n for n in expected if not live[n].present)
     if missing:
@@ -63,15 +64,12 @@ def _lanes_ok(*, with_ml: bool) -> tuple[bool, str]:
 
 
 def main(argv: list[str] | None = None) -> int:
-    argv = sys.argv[1:] if argv is None else argv
-    with_ml = "--with-ml" in argv
-
     ok, detail = _web_ok()
     if not ok:
         print(detail, file=sys.stderr)
         return 1
 
-    ok, detail = _lanes_ok(with_ml=with_ml)
+    ok, detail = _lanes_ok()
     if not ok:
         print(detail, file=sys.stderr)
         return 1