12 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	48108a3569	fix: a lane that is OFF was not attributable to itself (4295) ... Operator, 2026-09-23: *"clean up the stale service_seen rows"*. **They were not stale.** They were phantoms, written on purpose, and they will come back on every install that turns a lane off — so the rows are the smaller half of this. A celery worker was attributed to its lane by the queues it was CONSUMING. A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` with an empty list, matches no lane, and is dropped. Three consequences, all on the operator's screen at once: 1. The lanes table reported the lane **not answering** — the signal for a crashed worker, not for one the operator turned off. 2. The roster grew a phantom row named **`Worker ()`** — the empty queue set rendered as a display name — shown "running" beside the real lane's row going stale, because nothing updated it any more. 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every lane present. ML ships at cap 0, so a fresh install was permanently unhealthy and Swarm restarts an unhealthy task forever. That third one is the severe one, and its docstring asserted the opposite of what the code did — *"a disabled lane still runs its process with its consumers cancelled, so it answers inspect and is healthy"*. It answers. It was not attributed. A comment can be right about the intent and wrong about the program, and this one had been wrong since the consolidated container shipped. `worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels with the process rather than with what it happens to be doing. `gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the information was there and nothing read it. Falls back to the queue set for a deployment that names no node, and `docker-compose.yml` now sets one per service so the multi-service stack gets it too. The roster keys on the LANE's queue set when the node resolves, which is the same string the row already had while it was consuming — so an existing row keeps updating rather than a second one appearing. Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately NOT a retention sweep: the roster never forgets on purpose, so a quiet row is what it is FOR, and only a row that cannot correspond to anything real is safe to remove. An `agent:agent` row, if one exists, is left alone — nothing here can tell an abandoned agent id from a second agent that is genuinely down, and hiding a dead GPU agent is the one thing the roster must not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:21:21 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	274f7ffe21	ci: the tests gate the publish — ci.yml folds into build.yml ... Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*, then *"I don't want failing builds to publish anywhere going forward."* Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc` and `build-web` pushed `:dev` in the same minute, because the lanes and the build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and nothing about that was visible from either run. Two workflows cannot express the gate. A `needs:` edge only exists inside one graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted; `sign-extension`, `build-web` and `build-agent` now need all five. Nothing here is a new mechanism — it is the same edge that has gated `promote` since milestone 362 step 4, and it keeps that step's hardest-won property: **not running is not the same as passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane that silently skips itself blocks the publish exactly as a failing one does. Run 5290 is why that is worth stating. Scope, said plainly rather than implied: - Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension` publishes too, so it is gated with the rest. - Not gated, deliberately: `extension.yml` publishes nothing, and `release.yml` runs on a `v*` tag, generates notes rather than an artifact, and its commit already went through main's gated build. - `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its runs are the lanes and nothing else, via an `if:` on each publishing job rather than an inference from the `needs` chain. The cost, accepted knowingly: this workflow queues per branch and never cancels, so on two pushes in quick succession the second's lint feedback waits out the first's build. A slower red beats a fast red that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:15:42 -04:00
bvandeusenandClaude Opus 5	b2da3acce9	feat: wait for Postgres and Redis before starting work (4295) ... Operator, 2026-09-23: *"it's a single container that need to connect successfully to redis and postgres before starting work shouldn't that simply be a check (with retries) at the start of the container."* Yes, and the consolidated layout makes it necessary rather than tidy. Swarm has no ordering primitive — it ignores `depends_on` outright — so every service in a stack starts at once and this container has always raced its own database on a cold deploy. The multi-service stack hid how sharp that is: a `web` task that failed `alembic upgrade head` against a still-initialising Postgres simply died, and Swarm restarted it until it worked. Nobody ever saw a problem worth naming. Consolidation removes that safety net. Each supervisord program gets `startretries=3`, so three quick failures put the program in FATAL and leave it there — supervisord keeps running, the container keeps running, and the application never starts. It would present as a permanently unhealthy container whose image was fine and whose database merely took twenty seconds to initialise, which is a miserable thing to debug on a first deploy. A TCP connect, not a query: the same probe ci.yml's integration lane and the build smoke already use. It answers the question actually being asked — is something listening — and cannot fail for a reason that retrying will never fix. A real query would be a stronger readiness signal and a worse gate, since a wrong password or a missing database is not transient, and a loop waiting for one to heal turns a five-second misconfiguration into a two-minute timeout with a misleading message. Those belong to alembic, which runs seconds later and says exactly what is wrong. Targets are derived from the same env the application reads, so the wait cannot drift from what the app will actually connect to — a gate checking a different host than the app uses is worse than no gate. Bounded at 120s (rule 156), reporting every few attempts so `docker logs` on a waiting container says what it is waiting for. Skipped for `shell`, which exists precisely for when something else is broken and you want a prompt rather than a gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:17:58 -04:00
bvandeusenandClaude Opus 5	b09ee87255	fix: supervisorctl could not reach supervisord inside the container (4295) ... The smoke's own diagnostic line printed this for a whole run and passed, because it was behind `|| true`: Error: .ini file does not include supervisorctl section supervisord was fine. `supervisorctl` simply could not talk to it — the generated config had no `[unix_http_server]`, `[supervisorctl]` or `[rpcinterface:supervisor]`. That is the first tool anyone reaches for when a lane misbehaves in the consolidated container. `docker exec <c> supervisorctl status` to see which processes are actually up; `restart ml` to bounce one without taking the whole application down with it. Consolidation took `docker ps` away as the way to see the lanes, and this is what replaces it — so shipping without it would have left an operator with one container, five processes inside it, and no way to ask about any of them. They are about to run this in production. The test asserts the three sections AGREE on one socket path rather than merely existing: a serverurl pointing where nothing listens fails in exactly the same way and reads as configured. The smoke's line loses its `|| true`. A diagnostic allowed to fail silently is one that stops being true without telling anyone — which is precisely what happened here. It still printed the evidence that something was wrong while nothing depended on it, which is the argument for printing it at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:09:11 -04:00
bvandeusenandClaude Opus 5	efde3b188f	refactor: the image carries its own healthcheck and picks it by role (4295) ... Operator, 2026-09-23: *"why isn't the healthcheck built into the image or base on what command runs if one is passed in. why is it manually declared in the stack here."* No good reason. The container is the only thing that knows what it was asked to run, and every compose file, stack file and README had to restate it: web -> urllib /api/health worker -> celery inspect ping -d celery@$HOSTNAME all -> both, for every lane Three checks written by hand, once per service, in every file anyone ever wrote — none of them wrong until a role changed, and all of them silently wrong after. The same duplication the lane table exists to remove one level down, and I built it without noticing. `entrypoint.sh` now records the role it started. The Dockerfile declares ONE `HEALTHCHECK` that reads it and asks the right question: HTTP for web, a self-addressed celery ping for a worker lane, both-for-every-lane for `all`, and nothing for shell/alembic, which are one-shot and have no liveness to probe. `docker-compose.single.yml` and the consolidated stack declare none. A service that wants something else can still declare its own; docker prefers it, so the escape hatch is the default docker behaviour rather than a flag. Two details that are load-bearing: * The role is written ONCE, by the outermost invocation. `all` starts the other roles through this same script under supervisord, and a child overwriting the container's role would turn the composite check into a web-only one — silently, and only on the consolidated path. FC_ROLE is exported so a child sees it set and skips. * The celery ping is addressed to THIS node, not a bare ping. A bare one is answered by any worker on the broker, so in a stack with replicas a dead container would report healthy for as long as a sibling lived — the check would be measuring the cluster rather than the container it is inside. `healthcheck_all.py` is deleted; its two probes moved into the dispatcher rather than being a second copy beside it. An unrecorded role PASSES. The entrypoint always writes the file, so the only way to miss it is bypassing the entrypoint — a debugging shape, where a check that cannot tell what it is looking at must not assert the thing is broken (snippet #3969). Said on stdout rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:07:39 -04:00
bvandeusenandClaude Opus 5	828c6a5ae3	fix: each lane needs its own celery node name, or three of four vanish (4295) ... Found by the all-role smoke on its very first execution (run 7319), which is the whole argument for having added it one commit ago. Celery's default node name is `celery@<hostname>`. In the single-container layout all four lanes share one hostname, so all four registered as the SAME node. Celery says so itself: DuplicateNodenameWarning: Received multiple replies from node name: celery@72adc5b706a7 `inspect` collapses four replies into one dict key and the last one wins, so three lanes read as absent — and WHICH three varies between calls: lanes not answering: maintenance_long, ml, worker lanes not answering: maintenance_long, scheduler, worker Fatal twice over: * The composite healthcheck can never pass. In Swarm that is a container that never goes healthy — restart loop, then an automatic rollback of a deploy whose image was fine. * `pool_grow`/`pool_shrink` take a `destination` of node names. The UI dial and the autoscaler would have resized whichever lane happened to answer rather than the one asked for — silently, and differently each time. Every celery role now starts with `-n "${CELERY_NODENAME:-celery}@%h"`, and the generated supervisord config sets that per lane. The lanes become worker@<cid>, scheduler@<cid>, maintenance_long@<cid>, ml@<cid> — distinct, so inspect keeps four entries and `destination` addresses what it names. `inspect_lanes_sync` maps hostname to lane by QUEUES, so nothing there changes; it just stops having three of its four entries overwritten. Unset, it falls back to `celery` — exactly celery's own default — so every service in the multi-service stack is byte-identical to before, including the `celery@$HOSTNAME` healthcheck in docker-compose.yml and in the operator's Swarm stack. The test asserts DISTINCTNESS across the whole lane table rather than a fixed string per lane. The property that broke is that no two collide, and stating it that way keeps holding when a lane is added. This is the bug I said a live deploy was needed to find, found in CI instead for the price of one `docker run` — and it would have met the operator as a rollback loop on their first consolidated deploy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:52:56 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusenandClaude Opus 5	172e33de9a	feat: run web and every worker lane in one container (4295) ... Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d` gives three containers — FabledCurator, Postgres, Redis — where the stack previously needed seven. THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app services separately and remains the right shape for a Swarm deployment spread across hosts, where per-service rolling rollback and placement constraints matter. This adds a compose file; it deletes none. `entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES and execs it as PID 1. Generated rather than checked in because a static .conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of the queue names — after celery_app.task_routes and the three collapsed in steps 1, 2 and 4. Every one of those had already drifted when found. Generating gives a stronger guarantee than "they match today": a lane added to LANES gets a process, and a queue cannot end up with no consumer because someone missed a file. supervisord over s6-overlay: one pip dependency on an image already Python, with per-program stop timeouts and stopasgroup. The process-group part is not a detail — celery's prefork pool forks children, and a TERM reaching only the parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and zombie handling) comes from `init: true` instead. Nothing in FC talks to the supervisor, so the choice is reversible without touching product code. FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements live only in Dockerfile.ml until step 6 merges the images, so an `ml` program here would fail to import on every restart forever. `--with-ml` is the flag step 6 turns on. THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught: 1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord parses that key as a COMMA-separated list, so it reads as CELERY_QUEUES=default plus three malformed entries and the worker lane would have consumed only `default`. Silent: the worker starts, reports healthy, never picks up an import. Now quoted, and the test asserts the quoted form rather than the bare substring, which passed either way. 2. The generator emitted `entrypoint.sh <lane.name>`, but `maintenance_long` is not a role — compose runs it as the plain `worker` role with different queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh to assert every role a lane names actually exists. 3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY. Harmless while only compose started it and set none; with a generated value being passed, the lane would have sat at 1 until the reconcile noticed, with nothing saying why. The healthcheck asserts BOTH halves — hypercorn answers and every configured lane is answering the broker. That is the failure mode consolidation creates: docker can no longer see the lanes as separate services, so a web-only check would report a healthy container with every lane inside it dead. It deliberately ignores the `enabled` flag: a disabled lane still has a running process with its consumers cancelled, and marking the container unhealthy for turning tagging off would be wrong. stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s) rather than the average, with a test asserting no program's stopwaitsecs can exceed what compose allows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:32:27 -04:00
bvandeusenandClaude Opus 4.8	bc6d43d3f2	refactor(ml): drop dead tagger/suggestion settings + columns (#1199) ... Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:41:25 -04:00
bvandeusenandClaude Opus 4.7	3b3e7565fb	fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output) ... The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB) + camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx + selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping) not CSV. Per the published onnx_inference.py reference: input is NCHW not NHWC, normalize with ImageNet mean/std, pad-square color (124,116, 104), sigmoid the second output (refined predictions) not the first. Operator hit this during the IR migration ML backfill — download_models silently fetched only 3 json files (allow_patterns matched nothing useful), tagger.load() then raised RuntimeError. Fetched the actual v2 layout via WebFetch, rewrote tagger to match published reference. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 02:25:30 -04:00
bvandeusenandClaude Opus 4.7	b68a382b60	feat(fc2b): importer enqueues tag_and_embed + ml-worker model self-heal ... import_media_file now enqueues tag_and_embed alongside generate_thumbnail after a successful import. scripts/download_models.py snapshots Camie + SigLIP into /models, idempotent (skips when present). The ml-worker entrypoint runs it before starting the Celery worker so a fresh /models volume self-heals on first boot. Downloader tests are pure-logic (no network in CI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 07:45:09 -04:00