FabledCurator/README.md

# FabledCurator

<!-- overview:start -->
Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
downloading in one application. Part of the FabledSword family.

## What it does

You point it at creators you follow. It downloads what they post, files it,
tags it, and gives you something better than a folder full of images to look
through afterwards.

- **Gallery and browsing.** Images, videos and multi-page works, organised by
  artist, tag, post and series. A Showcase front page, a filterable gallery, a
  similarity-driven Explore view, and a page-turning reader for series.
- **Subscriptions.** Follows creators on Patreon, SubscribeStar, Pixiv and
  anything `gallery-dl` supports, on a schedule. Handles paywalled posts using
  your own logged-in session.
- **ML tagging.** Runs image models in-container to suggest tags, group
  characters, find near-duplicates and power similarity search. Suggestions are
  reviewable — it proposes, you confirm, and it learns which proposals you keep
  rejecting.
- **Deduplication and provenance.** Everything that arrives is hashed and
  deduplicated by content, metadata sidecars are read wherever the source
  writes them, and every file keeps a record of where it came from.
- **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
  orphan cleanup — all from the UI, all as background jobs you can watch.

Everything is configured from the Settings UI and stored in the database. There
is no config file to edit beyond a handful of bootstrap environment variables.
<!-- overview:end -->

## Before you expose it

**FabledCurator has no login.** There are no user accounts, no passwords and no
permission model. Anything that can reach the port is an administrator.

That matters more here than it would in most self-hosted apps, because of what
this one stores: **live platform session cookies for Patreon, SubscribeStar and
Pixiv** — accounts that usually have a payment method attached. Whoever reaches
the port can read them, alongside your entire library.

So:

- Bind it to a LAN, a VPN, or a tunnel you control.
- Do not port-forward it. Do not put it on a public hostname.
- A reverse proxy that adds TLS but no authentication **does not help**. If you
  want it reachable from outside, put an authenticating proxy in front of it —
  a forward-auth provider, HTTP basic auth, an identity-aware tunnel — and treat
  that layer as the only thing standing between the internet and your accounts.

This is a deliberate design decision for a single-operator tool on a trusted
network, not a bug and not an oversight. It is stated here because it decides
how you are allowed to deploy it. [SECURITY.md](SECURITY.md) covers the rest of
the threat model.

## Requirements

- **Docker** with Compose v2.
- **~4 GB RAM** for the app, plus whatever Postgres needs for your library size.
- **Disk** for your media, plus several GB for ML model weights.
- **No GPU required.** The ML worker runs on CPU — tagging and embedding are
  slower, and that is the whole difference. A GPU is only involved if you
  separately run the optional agent (below), which is a different machine's job.

## Install

```bash
git clone https://git.fabledsword.com/bvandeusen/FabledCurator.git
cd FabledCurator

cp .env.example .env
$EDITOR .env          # set DB_PASSWORD and SECRET_KEY

docker compose -f docker-compose.yml up -d
```

Then open <http://localhost:8080>.

**The `-f docker-compose.yml` is required, not decoration.** Compose
auto-merges `docker-compose.override.yml` when you leave it off, and that
override builds the images locally from source — the contributor path, not
yours. Naming the file explicitly skips the override and pulls the published
`:latest` images, which is the stable channel built from `main`.

If you forget it, the symptom is a long build instead of a quick pull.

## First run

The database schema is created automatically on first start — the web container
runs its migrations before serving. Nothing to initialise by hand.

**One thing does need a deliberate act, and the app will not start without it.**
FabledCurator encrypts your stored platform credentials with a key it keeps at
`./images/secrets/credential_key.b64`. On a brand-new install that file does not
exist, and rather than quietly creating one the app stops:

```
MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64
```

Set `CURATOR_BOOTSTRAP_NEW_KEY=1` in your `.env` for the first `up`, then delete
the line once the container is running. `.env.example` ships it with that
instruction attached.

The refusal is deliberate, and worth understanding rather than working around:
auto-creating a key is indistinguishable from the disaster case — a restore that
brought the database back but lost `./images/secrets` — where it would mint a key
that cannot decrypt anything, leaving an instance that looks healthy while every
paywalled download fails. Making you say so once, on an empty install, is the
price of that not happening silently later.

**Which means: back up `./images/secrets/` alongside your database.** It is the
only thing that can read your stored credentials. A database restored without it
needs every credential entered again by hand.

A few other things are worth knowing about the first few minutes:

- **The ML worker downloads its model weights on first boot**, several GB from
  HuggingFace into `./models`. Until that finishes, tagging is queued rather
  than broken. It is idempotent — a restart resumes rather than refetches.
- **The gallery starts empty**, and that is the expected state. Add a creator
  under **Subscriptions** and it fills as posts come down.
- **If you already have a library on disk**, there is no screen that imports
  it, and there is not going to be one. Folder ingestion had a UI until July
  2026; it was retired once posts began arriving entirely through
  subscriptions and the browser extension, and the decision to leave it
  retired is deliberate — the folder path carries complexity the product does
  not need in order to do its job. The supported way to fill a new install is
  to add the creators you follow under **Subscriptions** and let it pull.

  The `/api/import/trigger` endpoint is still wired up for anyone who wants to
  script a one-off against a folder mounted at `./import`, and its progress
  shows under **Settings → Activity**. Treat it as an unsupported escape
  hatch rather than a feature: nothing in the UI drives it and nothing else
  in this README depends on it.
- **To download from a paywalled account**, FabledCurator needs that account's
  session — see the browser extension below. Without one it can still fetch
  public posts.
- **Check Settings → Overview** to confirm the workers are alive. Every long
  operation in FabledCurator is a background job, so if the queues are not
  running, the UI will look like it is ignoring you rather than like it is
  broken.

## The browser extension

A Firefox extension does two jobs: it hands your logged-in platform sessions to
FabledCurator so it can download on your behalf, and it adds a creator as a
subscription in one click from their page.

It ships **inside the web image** — there is no add-on store listing to find.
Go to **Subscriptions → Settings**, find the *Browser extension* card, and click
**Install Firefox extension**. The XPI is Mozilla-signed, so Firefox installs it
like any other add-on; the button serves it directly rather than making you
download and side-load a file.

It pairs with your instance using an API key generated automatically on first
use. The bar directly under that card shows the key and can rotate it.

See [extension/README.md](extension/README.md) for what it does in detail.

## The GPU agent

Optional, and separate. If you have a desktop with a graphics card, you can run
an agent on it that leases ML jobs from FabledCurator over HTTP, does them on
the GPU, and hands the results back. It never touches the database or Redis, so
it is safe to run somewhere the rest of the stack is not.

Run it for a burst of tagging, stop it to get your card back. It deploys from
`agent/docker-compose.yml`, not the main stack — see
[agent/README.md](agent/README.md).

## Upgrading

```bash
docker compose -f docker-compose.yml pull
docker compose -f docker-compose.yml up -d
```

Migrations run automatically on start. Take a database backup first — Settings →
Maintenance has one — because the schema moves forward and does not move back.

## Deployment posture

FabledCurator is built to run inside a homelab over plain HTTP. It does not
generate certificates, redirect to HTTPS, or set HSTS. If you want TLS,
terminate it at your reverse proxy. See [Before you expose it](#before-you-expose-it)
for why TLS alone is not enough.

## Troubleshooting

**The UI loads but nothing ever finishes.** The web container is up and the
workers are not. `docker compose -f docker-compose.yml ps` — check `worker`,
`scheduler` and `ml-worker` are healthy, not restarting.

**`docker compose up` started building instead of pulling.** You left off
`-f docker-compose.yml`, so the dev override took over. See [Install](#install).

**Downloads fail with an auth error.** The stored session for that platform has
expired. Re-capture it with the extension; sessions do not last forever.

**Which build am I running?** The foot of Settings shows a version and a
channel, and `/api/health` returns the same two fields. There are no version
tags on the images, so this is the authoritative answer.

---

# Developing FabledCurator

Everything below is about working on FabledCurator rather than running it. If
you are installing it, you are done — see [CONTRIBUTING.md](CONTRIBUTING.md) if
you want to send a patch.

## Status and channels

In production. `main` is continuously deployed — every merge builds and
publishes `:latest`, so whatever is on `main` is what is running. Day-to-day
work happens on `dev`, which publishes `:dev`.

For local development, the dev override handles everything:

```bash
docker compose up -d          # note: no -f, so the override applies
```

That builds the images from source, turns on DEBUG logging, and exposes
Postgres and Redis on the host. No `.env` required.

## Versions and tags

Three image tags exist, and no others:

| Tag | Branch | Meaning |
| --- | --- | --- |
| `:latest` | `main` | Production. Moves on every merge. |
| `:c-<sha>` | `main` | Immutable — the rollback unit, all three images together. |
| `:dev` | `dev` | The rolling test channel. Moves on every push. |

There are deliberately **no version tags**. Nothing pins one, and a per-build
name nobody reads is upkeep for a model FC does not run (family rule 145; the
reasoning is note #3127 §5). Rolling back is `docker pull …:c-<sha>`.

Each artifact still has a version, derived rather than chosen: the commit time
of the newest change to that artifact's *own* shipped files, as
`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions —
a push touching only `agent/` re-versions the agent and leaves web and ml
alone, and CI skips the builds whose content did not move.

Because no registry name carries it, the running instance's own report is the
only answer to "which build is this?". The foot of Settings shows
`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two
fields.

Release tags are optional bookmarks — FC went twelve weeks without one and
nothing was wrong. Pushing `v<version>` publishes a Forgejo release listing the
commits since the previous tag; it builds no image.

## What's in here

Five deployable pieces, built by `.forgejo/workflows/build.yml`:

| Piece | Built from | Image | Role |
| --- | --- | --- | --- |
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. See `agent/README.md`. |
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |

## CI / Forgejo setup

Four workflows: `ci.yml` (lint, extension-version check, backend unit tests,
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
content verification), `build.yml` (sign + publish), and `release.yml`, which
runs only on a `v*` tag and publishes a changelog without building anything.

**The toolchain each job runs in is its `container.image`, not its `runs-on`
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
then names the image it actually wants. `ci-requirements.md` is the current,
authoritative list of images and per-job installs — read that rather than a
copy here, so the two can't drift.

The repo expects one secret:

- **`RELEASE_TOKEN`** — a Forgejo PAT with:
  - `write:package` + `read:package` — for `docker push` to `git.fabledsword.com`
  - `write:release` — for the `ext-<version>` releases that cache the signed XPI
  - `write:issue` — for issue-management automation

  Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.

AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`.
It runs on **both** channels and is cached per version: because the version is
derived from commit time, `dev` and `main` derive the same number for the same
source, so `main` finds `dev`'s signature already cached and makes no second AMO
call. That cache is why signing must be one-shot — AMO rejects a re-signed
version.

## History

FabledCurator combines what was
[ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo) (gallery, ML,
importer) and
[GallerySubscriber](https://git.fabledsword.com/bvandeusen/GallerySubscriber)
(gallery-dl wrapper, subscriptions, credential capture) into a single product.
Both are superseded; neither is maintained.

## License

**GNU Affero General Public License v3.0** — see [LICENSE](LICENSE).

You may run, study, modify and redistribute this software. The condition is
reciprocity: if you distribute a modified version, or **run one as a network
service that other people use**, you must offer those users the corresponding
source under the same licence. That second clause (AGPL §13) is the reason this
licence rather than the GPL — for a self-hosted web application, "distribution"
otherwise never happens, and the obligation would never bite.

Running an unmodified copy for yourself, your household or your organisation
carries no obligation at all. Neither does modifying it privately. The licence
asks something of you only when you hand your modified version to others.

Contributions ship under the same licence — see [CONTRIBUTING](CONTRIBUTING.md).
Security reports: [SECURITY.md](SECURITY.md).