bvandeusenandClaude Opus 5 aa2bb665b9
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 6s
Build images / smoke-web (push) Skipped
Build images / promote (push) Skipped
CI / frontend-build (push) Successful in 26s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 2m20s
fix: serialise builds per branch, and pin :c-<sha> to the digest this run built (4290)
Two independent defects from one missing mechanism. `grep -rn concurrency
.forgejo/workflows/` returned nothing, so two pushes to a branch ran
build.yml in full parallel with no ordering.

Both jobs read `fc.revision` off the channel tag before either has pushed, so
both miss the reuse check and both build. Whichever finishes LAST owns the
tag.

Half 1 — the rolling tag. A slower older build leaves `:dev` carrying content
older than the commit that moved it. Family rule 146 says a rolling channel
refreshes itself; this is the case where it quietly does not. Self-healing on
the next push.

Half 2, and the reason this is not filed low — `:c-<sha>`. The repoint step
wrote every non-channel tag by copying the channel tag BY NAME, so the
immutable rollback tag named whatever `:latest` pointed at when that step
happened to run. Lose the race and `:c-<shaA>` names the OTHER run's bytes.
Rule 145 makes that tag the rollback unit and immutable, so this does not
break immutability — it makes the tag wrong from birth, and immutability then
guarantees nobody ever corrects it. Nothing goes red; it surfaces the day
someone rolls back and gets a commit they did not choose. Not self-healing.

Two fixes, deliberately both:

A. A workflow-level `concurrency` group keyed on `github.ref`, so dev and
main never block each other. `cancel-in-progress: false` — queue, never
cancel: cancelling could kill sign-extension mid-AMO-upload, leaving the
version registered at AMO with no cached asset, which is the unrecoverable
stuck state that job's rollback trap exists to prevent, reached by another
door. Not keyed on BUILD_REF because the group is evaluated before any job
starts and cannot read the `env` context.

B. Each build-push-action step gains `id: build`, and the repoint step copies
from `$IMAGE@${{ steps.build.outputs.digest }}` — the manifest THIS run
pushed — rather than from the channel tag by name. On a reuse hit there is no
digest and the channel tag remains correct by definition: "hit" means that
tag already carries this commit's fc.revision.

B is not redundant with A. A depends on this Gitea honouring a key whose
failure mode is silent, and this file has been burned by exactly that before
(the `format()` note records `true == 'true'` evaluating FALSE on run 5270,
every lane green, the feature simply not happening). B holds the :c-<sha>
correctness property whether or not A is honoured.

The loop's exclusion is now keyed on CHANNEL_REF rather than on SOURCE, and
that is load-bearing: SOURCE may now be a digest ref, which never equals a
tag string, so testing against it would stop excluding the channel tag.
imagetools would index-wrap it and `.Image.Config.Labels` would stop
resolving, killing the reuse label permanently — #3183 arriving again.

Neither half is verifiable on a dev push: dev's tag list is the channel tag
alone, so the repoint correctly does nothing there. B is observable on the
next merge to main (:c-<sha>'s digest must equal the build step's), and A by
pushing twice in quick succession and reading the run list for a queued
second run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-21 22:07:53 -04:00

FabledCurator

Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven downloading in one application. Part of the FabledSword family.

What it does

You point it at creators you follow. It downloads what they post, files it, tags it, and gives you something better than a folder full of images to look through afterwards.

  • Gallery and browsing. Images, videos and multi-page works, organised by artist, tag, post and series. A newest-first feed of what just arrived as the front page, a random Showcase, a filterable gallery, a similarity-driven Explore view, and a page-turning reader for series.
  • Subscriptions. Follows creators on Patreon, SubscribeStar, Discord and HentaiFoundry, on a schedule. Handles paywalled posts using your own logged-in session.
  • ML tagging. Runs image models in-container to suggest tags, group characters, find near-duplicates and power similarity search. Suggestions are reviewable — it proposes, you confirm, and it learns which proposals you keep rejecting.
  • Deduplication and provenance. Everything that arrives is hashed and deduplicated by content, metadata sidecars are read wherever the source writes them, and every file keeps a record of where it came from.
  • Maintenance. Backups, library audits, thumbnail and embedding backfills, orphan cleanup — all from the UI, all as background jobs you can watch.

Everything is configured from the Settings UI and stored in the database. There is no config file to edit beyond a handful of bootstrap environment variables.

Before you expose it

FabledCurator has no login. There are no user accounts, no passwords and no permission model. Anything that can reach the port is an administrator.

That matters more here than it would in most self-hosted apps, because of what this one stores: live platform session cookies for Patreon and SubscribeStar — accounts that usually have a payment method attached. Whoever reaches the port can read them, alongside your entire library.

So:

  • Bind it to a LAN, a VPN, or a tunnel you control.
  • Do not port-forward it. Do not put it on a public hostname.
  • A reverse proxy that adds TLS but no authentication does not help. If you want it reachable from outside, put an authenticating proxy in front of it — a forward-auth provider, HTTP basic auth, an identity-aware tunnel — and treat that layer as the only thing standing between the internet and your accounts.

This is a deliberate design decision for a single-operator tool on a trusted network, not a bug and not an oversight. It is stated here because it decides how you are allowed to deploy it. SECURITY.md covers the rest of the threat model.

Requirements

  • Docker with Compose v2.
  • ~4 GB RAM for the app, plus whatever Postgres needs for your library size.
  • Disk for your media, plus several GB for ML model weights.
  • No GPU required. The ML worker runs on CPU — tagging and embedding are slower, and that is the whole difference. A GPU is only involved if you separately run the optional agent (below), which is a different machine's job.

Install

git clone https://git.fabledsword.com/bvandeusen/FabledCurator.git
cd FabledCurator

cp .env.example .env
$EDITOR .env          # set DB_PASSWORD and SECRET_KEY

docker compose -f docker-compose.yml up -d

Then open http://localhost:8080.

The -f docker-compose.yml is required, not decoration. Compose auto-merges docker-compose.override.yml when you leave it off, and that override builds the images locally from source — the contributor path, not yours. Naming the file explicitly skips the override and pulls the published :latest images, which is the stable channel built from main.

If you forget it, the symptom is a long build instead of a quick pull.

First run

The database schema is created automatically on first start — the web container runs its migrations before serving. Nothing to initialise by hand.

One thing does need a deliberate act, and the app will not start without it. FabledCurator encrypts your stored platform credentials with a key it keeps at ./images/secrets/credential_key.b64. On a brand-new install that file does not exist, and rather than quietly creating one the app stops:

MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64

Set CURATOR_BOOTSTRAP_NEW_KEY=1 in your .env for the first up, then delete the line once the container is running. .env.example ships it with that instruction attached.

The refusal is deliberate, and worth understanding rather than working around: auto-creating a key is indistinguishable from the disaster case — a restore that brought the database back but lost ./images/secrets — where it would mint a key that cannot decrypt anything, leaving an instance that looks healthy while every paywalled download fails. Making you say so once, on an empty install, is the price of that not happening silently later.

Which means: back up ./images/secrets/ alongside your database. It is the only thing that can read your stored credentials. A database restored without it needs every credential entered again by hand.

A few other things are worth knowing about the first few minutes:

  • The ML worker downloads its model weights on first boot, several GB from HuggingFace into ./models. Until that finishes, tagging is queued rather than broken. It is idempotent — a restart resumes rather than refetches.

  • The gallery starts empty, and that is the expected state. Add a creator under Subscriptions and it fills as posts come down.

  • If you already have a library on disk, there is no screen that imports it, and there is not going to be one. Folder ingestion had a UI until July 2026; it was retired once posts began arriving entirely through subscriptions and the browser extension, and the decision to leave it retired is deliberate — the folder path carries complexity the product does not need in order to do its job. The supported way to fill a new install is to add the creators you follow under Subscriptions and let it pull.

    The /api/import/trigger endpoint is still wired up for anyone who wants to script a one-off against a folder mounted at ./import, and its progress shows under Settings → Activity. Treat it as an unsupported escape hatch rather than a feature: nothing in the UI drives it and nothing else in this README depends on it.

  • To download from a paywalled account, FabledCurator needs that account's session — see the browser extension below. Without one it can still fetch public posts.

  • Check Settings → Overview to confirm the workers are alive. Every long operation in FabledCurator is a background job, so if the queues are not running, the UI will look like it is ignoring you rather than like it is broken.

The browser extension

A Firefox extension does two jobs: it hands your logged-in platform sessions to FabledCurator so it can download on your behalf, and it adds a creator as a subscription in one click from their page.

It ships inside the web image — there is no add-on store listing to find. Go to Subscriptions → Settings, find the Browser extension card, and click Install Firefox extension. The XPI is Mozilla-signed, so Firefox installs it like any other add-on; the button serves it directly rather than making you download and side-load a file.

It pairs with your instance using an API key generated automatically on first use. The bar directly under that card shows the key and can rotate it.

See extension/README.md for what it does in detail.

The GPU agent

Optional, and separate. If you have a desktop with a graphics card, you can run an agent on it that leases ML jobs from FabledCurator over HTTP, does them on the GPU, and hands the results back. It never touches the database or Redis, so it is safe to run somewhere the rest of the stack is not.

Run it for a burst of tagging, stop it to get your card back. It deploys from agent/docker-compose.yml, not the main stack — see agent/README.md.

Upgrading

docker compose -f docker-compose.yml pull
docker compose -f docker-compose.yml up -d

Migrations run automatically on start. Take a database backup first — Settings → Maintenance has one — because the schema moves forward and does not move back.

Deployment posture

FabledCurator is built to run inside a homelab over plain HTTP. It does not generate certificates, redirect to HTTPS, or set HSTS. If you want TLS, terminate it at your reverse proxy. See Before you expose it for why TLS alone is not enough.

Troubleshooting

The UI loads but nothing ever finishes. The web container is up and the workers are not. docker compose -f docker-compose.yml ps — check worker, scheduler and ml-worker are healthy, not restarting.

docker compose up started building instead of pulling. You left off -f docker-compose.yml, so the dev override took over. See Install.

Downloads fail with an auth error. The stored session for that platform has expired. Re-capture it with the extension; sessions do not last forever.

Which build am I running? The foot of Settings shows a version and a channel, and /api/health returns the same two fields. There are no version tags on the images, so this is the authoritative answer.


Developing FabledCurator

Everything below is about working on FabledCurator rather than running it. If you are installing it, you are done — see CONTRIBUTING.md if you want to send a patch.

Status and channels

In production. main is continuously deployed — every merge builds and publishes :latest, so whatever is on main is what is running. Day-to-day work happens on dev, which publishes :dev.

For local development, the dev override handles everything:

docker compose up -d          # note: no -f, so the override applies

That builds the images from source, turns on DEBUG logging, and exposes Postgres and Redis on the host. No .env required.

Versions and tags

Three image tags exist, and no others:

Tag Branch Meaning
:latest main Production. Moves on every merge.
:c-<sha> main Immutable — the rollback unit, all three images together.
:dev dev The rolling test channel. Moves on every push.

There are deliberately no version tags. Nothing pins one, and a per-build name nobody reads is upkeep for a model FC does not run (family rule 145; the reasoning is note #3127 §5). Rolling back is docker pull …:c-<sha>.

Each artifact still has a version, derived rather than chosen: the commit time of the newest change to that artifact's own shipped files, as YYYY.MM.DD.HHMM UTC (rule 148). Four artifacts, four independent versions — a push touching only agent/ re-versions the agent and leaves web and ml alone, and CI skips the builds whose content did not move.

Because no registry name carries it, the running instance's own report is the only answer to "which build is this?". The foot of Settings shows FabledCurator 2026.08.29.0201 · dev, and /api/health returns the same two fields.

Release tags are optional bookmarks — FC went twelve weeks without one and nothing was wrong. Pushing v<version> publishes a Forgejo release listing the commits since the previous tag; it builds no image.

What's in here

Five deployable pieces, built by .forgejo/workflows/build.yml:

Piece Built from Image Role
Web / workers Dockerfile fabledcurator Quart API + the built Vue SPA in one image. entrypoint.sh picks the role: web, worker, scheduler. The maintenance-long service is a second worker pinned to the long-running maintenance queue.
ML worker Dockerfile.ml fabledcurator-ml Same app, plus requirements-ml.txt — tagging and embedding models that run in-container.
GPU agent agent/Dockerfile fabledcurator-agent Optional desktop-GPU worker (agent/). Leases jobs over HTTP only — never touches the database or Redis. See agent/README.md.
Firefox extension extension/ signed XPI MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both dev and main (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See extension/README.md.
Data pgvector/pgvector:pg16, redis:7-alpine Postgres with pgvector for embeddings; Redis as the Celery broker.

CI / Forgejo setup

Four workflows: ci.yml (lint, extension-version check, backend unit tests, frontend build, integration), extension.yml (extension lint, vitest, XPI content verification), build.yml (sign + publish), and release.yml, which runs only on a v* tag and publishes a changelog without building anything.

The toolchain each job runs in is its container.image, not its runs-on label. runs-on: python-ci only schedules the job onto a runner; every job then names the image it actually wants. ci-requirements.md is the current, authoritative list of images and per-job installs — read that rather than a copy here, so the two can't drift.

The repo expects one secret:

  • RELEASE_TOKEN — a Forgejo PAT with:

    • write:package + read:package — for docker push to git.fabledsword.com
    • write:release — for the ext-<version> releases that cache the signed XPI
    • write:issue — for issue-management automation

    Generate at https://git.fabledsword.com/user/settings/applications. The injected GITHUB_TOKEN cannot be used because it lacks write:package.

AMO signing additionally needs MOZILLA_AMO_JWT_KEY / MOZILLA_AMO_JWT_SECRET. It runs on both channels and is cached per version: because the version is derived from commit time, dev and main derive the same number for the same source, so main finds dev's signature already cached and makes no second AMO call. That cache is why signing must be one-shot — AMO rejects a re-signed version.

History

FabledCurator combines what was ImageRepo (gallery, ML, importer) and GallerySubscriber (gallery-dl wrapper, subscriptions, credential capture) into a single product. Both are superseded; neither is maintained.

License

GNU Affero General Public License v3.0 — see LICENSE.

You may run, study, modify and redistribute this software. The condition is reciprocity: if you distribute a modified version, or run one as a network service that other people use, you must offer those users the corresponding source under the same licence. That second clause (AGPL §13) is the reason this licence rather than the GPL — for a self-hosted web application, "distribution" otherwise never happens, and the obligation would never bite.

Running an unmodified copy for yourself, your household or your organisation carries no obligation at all. Neither does modifying it privately. The licence asks something of you only when you hand your modified version to others.

Contributions ship under the same licence — see CONTRIBUTING. Security reports: SECURITY.md.

S
Description
Self-hosted media curation — gallery, ML tagging, and subscription-driven downloads. Part of the FabledSword family. (Merge of ImageRepo + GallerySubscriber.)
Readme AGPL-3.0
10 MiB
2026-09-13 11:51:25 -04:00
Languages
Python 73.7%
Vue 17.3%
JavaScript 8%
Shell 0.5%
CSS 0.3%