6 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusenandClaude Opus 5	172e33de9a	feat: run web and every worker lane in one container (4295) ... Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d` gives three containers — FabledCurator, Postgres, Redis — where the stack previously needed seven. THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app services separately and remains the right shape for a Swarm deployment spread across hosts, where per-service rolling rollback and placement constraints matter. This adds a compose file; it deletes none. `entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES and execs it as PID 1. Generated rather than checked in because a static .conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of the queue names — after celery_app.task_routes and the three collapsed in steps 1, 2 and 4. Every one of those had already drifted when found. Generating gives a stronger guarantee than "they match today": a lane added to LANES gets a process, and a queue cannot end up with no consumer because someone missed a file. supervisord over s6-overlay: one pip dependency on an image already Python, with per-program stop timeouts and stopasgroup. The process-group part is not a detail — celery's prefork pool forks children, and a TERM reaching only the parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and zombie handling) comes from `init: true` instead. Nothing in FC talks to the supervisor, so the choice is reversible without touching product code. FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements live only in Dockerfile.ml until step 6 merges the images, so an `ml` program here would fail to import on every restart forever. `--with-ml` is the flag step 6 turns on. THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught: 1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord parses that key as a COMMA-separated list, so it reads as CELERY_QUEUES=default plus three malformed entries and the worker lane would have consumed only `default`. Silent: the worker starts, reports healthy, never picks up an import. Now quoted, and the test asserts the quoted form rather than the bare substring, which passed either way. 2. The generator emitted `entrypoint.sh <lane.name>`, but `maintenance_long` is not a role — compose runs it as the plain `worker` role with different queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh to assert every role a lane names actually exists. 3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY. Harmless while only compose started it and set none; with a generated value being passed, the lane would have sat at 1 until the reconcile noticed, with nothing saying why. The healthcheck asserts BOTH halves — hypercorn answers and every configured lane is answering the broker. That is the failure mode consolidation creates: docker can no longer see the lanes as separate services, so a web-only check would report a healthy container with every lane inside it dead. It deliberately ignores the `enabled` flag: a disabled lane still has a running process with its consumers cancelled, and marking the container unhealthy for turning tagging off would be wrong. stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s) rather than the average, with a test asserting no program's stopwaitsecs can exceed what compose allows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:32:27 -04:00
bvandeusenandClaude Opus 4.8	0fe1674753	perf(web): stream files in 4 MiB chunks + 4 hypercorn workers (fix 40s downloads) ... The image library is on a CIFS/SMB share (mounted rsize=4 MiB, actimeo=1), and Quart's FileBody streams in 8 KiB chunks — so serving one large original was ~19k network round-trips to the storage server, i.e. 30–58s per download (operator-flagged). That's what starved the GPU agent (constant "curator unreachable" backoff) AND slowed the browser: every byte is read off CIFS and streamed through the Python app (no reverse-proxy sendfile), and only 2 hypercorn workers meant the agent + the browser's thumbnail grid queued behind each other. In-container fix, no new service: - Raise FileBody.buffer_size 8 KiB → 4 MiB in create_app, matching the mount's read size: one round-trip per read, ~500× fewer. buffer_size is the MAX read so small thumbnails still read in one gulp, and Range/mime/ETag/conditional handling lives on Response — all preserved. Guarded so a Quart-internal change can't break boot. - HYPERCORN_WORKERS default 2 → 4 so concurrent /images requests stop queuing. Expected: large-file transfers drop from ~40s toward link speed (a few seconds) for the agent and the browser. See issue #1223. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 11:51:09 -04:00
bvandeusen	4a29a6d197	fix(web): pass create_app as a hypercorn factory (backend.app:create_app())	2026-05-15 22:53:49 -04:00
bvandeusenandClaude Opus 4.7	b68a382b60	feat(fc2b): importer enqueues tag_and_embed + ml-worker model self-heal ... import_media_file now enqueues tag_and_embed alongside generate_thumbnail after a successful import. scripts/download_models.py snapshots Camie + SigLIP into /models, idempotent (skips when present). The ml-worker entrypoint runs it before starting the Celery worker so a fresh /models volume self-heals on first boot. Downloader tests are pure-logic (no network in CI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 07:45:09 -04:00
bvandeusenandClaude Opus 4.7	1db0167bfc	feat: add Dockerfile for web/worker/scheduler roles and entrypoint script ... Multi-stage build: node:20 builds the SPA, python:3.12-slim runs the app. Same image handles web, worker, scheduler roles via entrypoint.sh's first arg. ffmpeg + unar are baked in for FC-2's transcode + archive paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:42:04 -04:00