14 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	b74a4c964b	fix: the image brings its own PID 1 instead of asking for init: true (4295) ... Operator, 2026-09-23: *"it is out of the norm to require this init call we need to fix this."* Correct, and it is the same mistake as declaring the healthcheck per service — the image needing a deployment to remember a flag before it behaves correctly. PID 1 carries a duty no other process has: every orphaned process in the container reparents to it and must be reaped or it stays a zombie holding a PID slot. This app makes orphans in normal operation — six service modules shell out (gallery_dl, thumbnailer, backup_service, external_fetch, download_service, download_backends) and celery's prefork pool forks children that spawn them. Whatever the role, something that is not an init ends up as PID 1: supervisord for `all`, hypercorn for `web`, celery for a worker. `init: true` covered that, and cost correctness the moment it was forgotten or silently dropped — an older Swarm, a plain `docker run`, a compose file someone copied. No signal either way. So tini goes in the image and is the ENTRYPOINT. `docker run <image>` is correct on its own now, `init: true` comes out of docker-compose.single.yml, and nothing downstream has to know. The smoke asserts /proc/1/comm is tini, read from /proc because the runtime stage installs no `ps`. ## A correction to what I told the operator I justified `init: true` by saying supervisord "has no idea about orphans it never started". That is very likely wrong: supervisord's reaper calls waitpid(-1) and logs "reaped unknown pid" for children it did not spawn, so it does reap orphans. I asserted the mechanism without checking it, and could not check it here — `supervisor` is not installed in this environment. It does not change this commit. tini is correct whichever way that lands, and it covers the single-role containers too, where celery or hypercorn is PID 1 and the subprocess-spawning is heaviest. But the reason I gave was not a verified one and should not have been stated as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:18:47 -04:00
bvandeusenandClaude Opus 5	efde3b188f	refactor: the image carries its own healthcheck and picks it by role (4295) ... Operator, 2026-09-23: *"why isn't the healthcheck built into the image or base on what command runs if one is passed in. why is it manually declared in the stack here."* No good reason. The container is the only thing that knows what it was asked to run, and every compose file, stack file and README had to restate it: web -> urllib /api/health worker -> celery inspect ping -d celery@$HOSTNAME all -> both, for every lane Three checks written by hand, once per service, in every file anyone ever wrote — none of them wrong until a role changed, and all of them silently wrong after. The same duplication the lane table exists to remove one level down, and I built it without noticing. `entrypoint.sh` now records the role it started. The Dockerfile declares ONE `HEALTHCHECK` that reads it and asks the right question: HTTP for web, a self-addressed celery ping for a worker lane, both-for-every-lane for `all`, and nothing for shell/alembic, which are one-shot and have no liveness to probe. `docker-compose.single.yml` and the consolidated stack declare none. A service that wants something else can still declare its own; docker prefers it, so the escape hatch is the default docker behaviour rather than a flag. Two details that are load-bearing: * The role is written ONCE, by the outermost invocation. `all` starts the other roles through this same script under supervisord, and a child overwriting the container's role would turn the composite check into a web-only one — silently, and only on the consolidated path. FC_ROLE is exported so a child sees it set and skips. * The celery ping is addressed to THIS node, not a bare ping. A bare one is answered by any worker on the broker, so in a stack with replicas a dead container would report healthy for as long as a sibling lived — the check would be measuring the cluster rather than the container it is inside. `healthcheck_all.py` is deleted; its two probes moved into the dispatcher rather than being a second copy beside it. An unrecorded role PASSES. The entrypoint always writes the file, so the only way to miss it is bypassing the entrypoint — a debugging shape, where a check that cannot tell what it is looking at must not assert the thing is broken (snippet #3969). Said on stdout rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:07:39 -04:00
bvandeusenandClaude Opus 5	43ac737516	feat: the whole application is what the image runs by default (4296) ... Operator, 2026-09-23: *"I also want to see that we remove the need for the command line of the configuration in the consolidated version."* `CMD` was `web`, so the single-container layout only worked if you knew to ask for it by name. A compose file that forgot `command: ["all"]` got a web server with nothing processing its queues — a gallery that loads, accepts an import, and never finishes one. Nothing errors; it just never progresses. Now `docker run fabledcurator` with no command starts hypercorn plus every lane under supervisord. `entrypoint.sh`'s own default moves with it, since the two are doors to the same decision and a disagreement would only show up as `--entrypoint` behaving differently from a plain run. `docker-compose.single.yml` drops its `command:` line; `["all"]` still works and still means the same thing. The multi-service stack is untouched — every service there names its role explicitly, which is what makes it the multi-service stack. ## And CI now actually boots it This is the gap I should have named when I reported milestone 422 at 7/7 and did not. Measured, not inferred: the smoke booted role `web` only (build.yml:1454), nothing in CI ran `all`, `docker-compose.single.yml` was read as TEXT by one test checking stop_grace_period and never run, and test_gen_supervisord asserts the generated config against the lane table without ever handing it to supervisord. So the shape this milestone is NAMED for had started nowhere. Steps 5-7 were marked done on evidence that did not cover it, and the operator is about to collapse their production stack onto exactly that. The smoke now boots the image with NO command — checking the Dockerfile CMD, the entrypoint default and the role together, the way an adopter gets it — and asserts `healthcheck_all`, which was itself never executed. That check passes only when hypercorn answers AND every lane in the table answers the broker; a web-only check goes green with every worker dead, which is the failure mode consolidation creates. It then prints `supervisorctl status`, so a lane that is merely restart-looping is visible rather than inferred. Cheap because ml ships at 0 slots and disabled: nothing loads a model, and the lane answers `inspect` with its consumers cancelled, which is what healthy means for a disabled lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:48:11 -04:00
bvandeusenandClaude Opus 5	0f98e46200	docs: the merged image's cost, measured — and it corrects my own estimate (4296) ... I wrote in ffcd130's Dockerfile comment that merging ML in means "everyone pulls it, including the many who will never turn tagging on", framed as a real cost the milestone accepted. My working estimate behind that was ~4GB. Measured from run 7273's build-web log: torch 2.12.1+cpu wheel 192.3 MB torchvision 0.27.1+cpu 1.8 MB transformers / onnxruntime / opencv / sklearn and friends 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB largest newly-pushed layer 222.07 MB The ML code adds a few HUNDRED MB, not gigabytes. The `--index-url` CPU resolution is what makes that true — the default PyPI torch wheel carries the CUDA runtime and is ~2GB by itself, and the log confirms 2.12.1+cpu resolved, so it is working as intended rather than as intended-but-unverified. Why this matters beyond a comment being wrong: it settles the trade this step was explicitly asked to weigh and could not, and it reverses how close the call looked. Baking the weights in adds ~3.5GB to every pull for a feature many adopters never enable; shipping the code and fetching on demand adds ~350MB. An order of magnitude, where the estimate had them within 15% of each other. Off-by-default is not a judgement call here, it is arithmetic. The gigabytes were always in the MODEL, and the model is not in the image. Two things NOT measured, still: the total image size (the push only transfers layers the registry lacks, so a push log cannot give it) and the per-slot resident RAM, which stays flagged `measured=False` in the lane table and renders as "about" in the UI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:02:47 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusen	bce894ba24	feat(settings): the instance reports which build it is (318 step 6) ... A dim line at the foot of Settings: `FabledCurator 2026.08.28.1249 · dev`. This is no longer a convenience. Milestone 318 stopped publishing version image tags, so an instance's own report is the ONLY answer to "which build is this?" — there is no registry name left to check it against. Note #3127 §5 says it directly: a wrong answer here has no second source to contradict it. Three states, kept distinct because collapsing any two of them lies: not asked yet render nothing asked, no version render "unknown" asked, has a version render it A blank footer reads as "no version", which is a different claim from "I cannot say". And a failed health call deliberately does NOT mark the build loaded — a network blip says nothing about the image, and presenting it as "unknown" would look like a defective build. Carried on /api/health rather than a new route: it answers at the same cost (two module constants, no I/O) and TopNav already fetches it app-wide, so a separate endpoint would mean a second request for two strings. Both fields are OMITTED when unset rather than sent empty. Absence already means "cannot say" — an image predating the field says exactly that by not having the key — so a second spelling would make every reader special-case it. The pre-existing test asserting the body is EXACTLY {"status": "ok"} is what keeps a well-meaning `or ""` default from creeping in. FC_CHANNEL now has one definition. It was read from the environment in extension.py and would have been read again here; the new build_info module holds both, and extension.py binds it as a module-level name so existing tests monkeypatch it exactly as before. Separate from config.py on purpose: those are operator settings meant to be changed, these describe the artifact. Channel sits beside the version, never inside it (rule 149), asserted from both ends. A `-dev` suffix would read as a 0 segment to the extension's parseInt comparator and make every dev build compare equal — #2993 exactly. Not hidden, per the operator and §7: the JS bundle and asset hashes fingerprint the build anyway, and "I'm on 2026.08.28.1249" is the single most useful line in a bug report.	2026-08-28 18:08:31 -04:00
bvandeusen	a7e626a67a	feat(extension): report the channel beside the version (step 7) ... Closes the half of the ask the signing work didn't: a way to tell a dev build from a main one. FC_CHANNEL is baked into the web image at build time and /api/extension/manifest reports it as its own key, next to version — the popup banner, the toolbar tooltip and the Settings card all name it. Beside the version, never inside it. A `1.0.3499884-dev` suffix is the obvious shortcut and it is the exact failure this design comes from: versionIsNewer parses each dotted segment with parseInt, so a suffixed segment reads as 0, every dev build compares equal to every other, and "no update available" stops being distinguishable from "I cannot read this version". The comparator already degrades rather than discarding (rule 150), which is a reason not to NEED the suffix, not a licence to add one. Two tests hold the line — one backend, asserting version and channel are separate keys; one frontend, asserting the rendered version text stays the bare derived number. Optional on the read side, and absent rather than defaulted. An image built before this field says nothing by not having the key; an image built without a channel now says nothing the same way, so there is one absence to handle instead of a second spelling of "unknown". Every reader drops the label entirely when it is missing and reads exactly as it did before. Reported verbatim rather than validated against {dev, main}: if an image declares something else, showing what it claims helps whoever is debugging more than dropping it would. FC_CHANNEL is declared LAST in the Dockerfile. An ARG invalidates every layer below it, and this is the one value that differs between the dev and main builds of identical source — earlier, and the two channels could never share a cached pip install. A tag push counts as main: a vYY.MM.DD tag is cut from main, so that image is a main-channel artifact wearing an immutable name. No channel switcher, deliberately. background.js:34 already records that Firefox's static update_url cannot apply, because every FC instance is a different host — so the extension asks its configured backend, and the channel IS the instance it points at. Switching is repointing apiUrl and reinstalling from that host. A separate setting would contradict each server build shipping its own extension. This commit touches packaged extension files, so it moves the derived version and will sign a new one via AMO — the first push to exercise the extension-changed path from dev end to end.	2026-08-27 11:47:30 -04:00
Renovate Bot	6ab495292a	chore(deps): update node.js to v24	2026-07-12 01:33:07 +00:00
Renovate Bot	e00deee451	chore(deps): update docker/dockerfile docker tag to v1.25	2026-07-11 23:58:54 +00:00
bvandeusenandClaude Opus 4.8	bd2807cdd1	feat(external): mega.nz via megatools in the runtime image (Phase 4c) ... Use `megatools dl` (Debian-native apt package) for mega.nz public links rather than MEGAcmd — no external MEGA apt repo/key to add, one apt line. Adds `megatools` to the runtime Dockerfile; the fetcher's mega backend now shells `megatools dl --path <dir> <url>` (key in the #fragment is preserved by the extractor). gdown (gdrive) is already a pip dep in the runtime image. NOTE: build.yml builds the image on main/tags only (not dev), so this Dockerfile change is verified on the next dev→main merge, not by this dev push. The fetcher code path is unit-tested via the mocked _run_mega_get seam. With this, all 5 hosts download end-to-end once a celery download-worker runs. Refs FC #830 (Phase 4c). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 15:41:18 -04:00
bvandeusenandClaude Opus 4.7	7f26217166	fc5: install postgresql-client + zstd in runtime image so backup/rollback work ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-22 22:17:18 -04:00
bvandeusenandClaude Opus 4.7	aa9705882f	fix: use npm install in Dockerfile frontend-builder stage (no lockfile) ... Same fix as the CI workflow — `npm ci` requires package-lock.json and we don't track one. Missed this in the earlier CI fix. Local `docker compose up` builds now succeed past the frontend stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 21:01:29 -04:00
bvandeusenandClaude Opus 4.7	37e97d52ee	chore: bump runtime targets to Python 3.14 and Node 22 ... Forward-looking pin per operator direction ("build for the future and not rebuild the past"). Touches everywhere a version is named so all downstream artifacts (Dockerfiles, ruff config, package.json engines) agree. Python 3.14 (released Oct 2025) is the current stable. Node 22 is the most recent LTS line still receiving updates (Maintenance LTS since Oct 2025); Node 24 (released Apr 2026) goes Active LTS in Oct 2026 and will be the natural next bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:52:15 -04:00
bvandeusenandClaude Opus 4.7	1db0167bfc	feat: add Dockerfile for web/worker/scheduler roles and entrypoint script ... Multi-stage build: node:20 builds the SPA, python:3.12-slim runs the app. Same image handles web, worker, scheduler roles via entrypoint.sh's first arg. ffmpeg + unar are baked in for FC-2's transcode + archive paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:42:04 -04:00