diff --git a/.env.example b/.env.example index d78fd8f..cbb94d4 100644 --- a/.env.example +++ b/.env.example @@ -1,24 +1,75 @@ -# Database -DB_USER=fabledcurator -DB_PASSWORD=changeme_use_a_real_password -DB_HOST=postgres -DB_PORT=5432 -DB_NAME=fabledcurator +# FabledCurator configuration. +# +# Copy to `.env` and edit before your first production start: +# +# cp .env.example .env +# +# Only the two values under CHANGE THESE actually need your attention. The +# rest have working defaults baked into docker-compose.yml and are listed +# here so you know they exist, not because you have to set them. +# +# Almost nothing else lives here on purpose. FabledCurator is configured from +# its own Settings UI, backed by the database — no restart, no YAML. If you +# are looking for where to set an import path, a download schedule or an ML +# threshold, it is in the app, not in this file. -# Redis / Celery -CELERY_BROKER_URL=redis://redis:6379/0 -CELERY_RESULT_BACKEND=redis://redis:6379/0 -# App -# Generate with: openssl rand -hex 32 -SECRET_KEY=changeme_32_byte_hex_secret +# --------------------------------------------------------------------------- +# CHANGE THESE +# --------------------------------------------------------------------------- -# Extension API key — used in FC-3, lands later but reserved now -# Generate with: openssl rand -hex 32 -EXTENSION_API_KEY= +# The Postgres password. docker-compose.yml falls back to a published default +# (`fabledcurator_dev`) so that `docker compose up` works with no config at +# all — which is exactly why you must not leave it at that on a real install. +# It is the credential protecting your stored platform session cookies. +DB_PASSWORD= -# Logging +# Sets Quart's app.secret_key. Today it signs nothing: FabledCurator has no +# login and uses no session cookies, so no value here is protecting anything +# right now. Set it anyway. It is required at boot rather than defaulted so +# that the day something session-backed does land, no instance is already +# running on a value published in this file. +# +# openssl rand -hex 32 +SECRET_KEY= + + +# --------------------------------------------------------------------------- +# Optional — defaults are fine +# --------------------------------------------------------------------------- + +# Host port the UI is published on. The container always listens on 8080; +# this is only the left-hand side of the port mapping. +PORT=8080 + +# DEBUG | INFO | WARNING | ERROR LOG_LEVEL=INFO -# Deployment posture: plain HTTP (no TLS in the app; reverse proxy if needed) -# See docs/superpowers/specs/2026-05-13-fabledcurator-merge-design.md §2.1 +# Postgres identity. Change these only if you are pointing at a database you +# manage yourself — the bundled postgres service is created with whatever is +# set here, so changing them after the first start will not rename anything. +DB_USER=fabledcurator +DB_NAME=fabledcurator + +# Set by docker-compose.yml to reach the bundled services. Override only when +# running Postgres or Redis outside this stack. +# DB_HOST=postgres +# DB_PORT=5432 +# CELERY_BROKER_URL=redis://redis:6379/0 +# CELERY_RESULT_BACKEND=redis://redis:6379/0 + + +# --------------------------------------------------------------------------- +# There is no authentication variable here, and that is not an omission +# --------------------------------------------------------------------------- +# +# FabledCurator has no login, no accounts and no permission model. Anything +# that can reach PORT is an administrator and can read the platform session +# cookies the app stores for Patreon, SubscribeStar and Pixiv. +# +# Bind it to a trusted network. See "Before you expose it" in README.md and +# the deployment posture section of SECURITY.md. +# +# The Firefox extension's API key is NOT configured here — it is generated +# automatically on first use and shown under Settings → Maintenance, where you +# can also rotate it. diff --git a/README.md b/README.md index efeff23..1f2af0b 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,201 @@ # FabledCurator -Self-hosted media curation — gallery, ML tagging, and subscription-driven downloading in one app. Part of the FabledSword family. +Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven +downloading in one application. Part of the FabledSword family. -Combines what was [ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo) (gallery, ML, importer) and [GallerySubscriber](https://git.fabledsword.com/bvandeusen/GallerySubscriber) (gallery-dl wrapper, subscriptions, credential capture) into a single product. +## What it does -## Status +You point it at creators you follow. It downloads what they post, files it, +tags it, and gives you something better than a folder full of images to look +through afterwards. -In production. `main` is continuously deployed — every merge to `main` builds -and publishes `:latest` images, so whatever is on `main` is what is running. -Day-to-day work happens on `dev`, which publishes `:dev` images. +- **Gallery and browsing.** Images, videos and multi-page works, organised by + artist, tag, post and series. A Showcase front page, a filterable gallery, a + similarity-driven Explore view, and a page-turning reader for series. +- **Subscriptions.** Follows creators on Patreon, SubscribeStar, Pixiv and + anything `gallery-dl` supports, on a schedule. Handles paywalled posts using + your own logged-in session. +- **ML tagging.** Runs image models in-container to suggest tags, group + characters, find near-duplicates and power similarity search. Suggestions are + reviewable — it proposes, you confirm, and it learns which proposals you keep + rejecting. +- **Importing.** Ingests an existing library from disk, deduplicates it by + content hash, and reads metadata sidecars. +- **Maintenance.** Backups, library audits, thumbnail and embedding backfills, + orphan cleanup — all from the UI, all as background jobs you can watch. + +Everything is configured from the Settings UI and stored in the database. There +is no config file to edit beyond a handful of bootstrap environment variables. + +## Before you expose it + +**FabledCurator has no login.** There are no user accounts, no passwords and no +permission model. Anything that can reach the port is an administrator. + +That matters more here than it would in most self-hosted apps, because of what +this one stores: **live platform session cookies for Patreon, SubscribeStar and +Pixiv** — accounts that usually have a payment method attached. Whoever reaches +the port can read them, alongside your entire library. + +So: + +- Bind it to a LAN, a VPN, or a tunnel you control. +- Do not port-forward it. Do not put it on a public hostname. +- A reverse proxy that adds TLS but no authentication **does not help**. If you + want it reachable from outside, put an authenticating proxy in front of it — + a forward-auth provider, HTTP basic auth, an identity-aware tunnel — and treat + that layer as the only thing standing between the internet and your accounts. + +This is a deliberate design decision for a single-operator tool on a trusted +network, not a bug and not an oversight. It is stated here because it decides +how you are allowed to deploy it. [SECURITY.md](SECURITY.md) covers the rest of +the threat model. + +## Requirements + +- **Docker** with Compose v2. +- **~4 GB RAM** for the app, plus whatever Postgres needs for your library size. +- **Disk** for your media, plus several GB for ML model weights. +- **No GPU required.** The ML worker runs on CPU — tagging and embedding are + slower, and that is the whole difference. A GPU is only involved if you + separately run the optional agent (below), which is a different machine's job. + +## Install + +```bash +git clone https://git.fabledsword.com/bvandeusen/FabledCurator.git +cd FabledCurator + +cp .env.example .env +$EDITOR .env # set DB_PASSWORD and SECRET_KEY + +docker compose -f docker-compose.yml up -d +``` + +Then open . + +**The `-f docker-compose.yml` is required, not decoration.** Compose +auto-merges `docker-compose.override.yml` when you leave it off, and that +override builds the images locally from source — the contributor path, not +yours. Naming the file explicitly skips the override and pulls the published +`:latest` images, which is the stable channel built from `main`. + +If you forget it, the symptom is a long build instead of a quick pull. + +## First run + +The database schema is created automatically on first start — the web container +runs its migrations before serving. Nothing to initialise by hand. + +A few things are worth knowing about the first few minutes: + +- **The ML worker downloads its model weights on first boot**, several GB from + HuggingFace into `./models`. Until that finishes, tagging is queued rather + than broken. It is idempotent — a restart resumes rather than refetches. +- **The gallery starts empty**, and that is the expected state. Add a creator + under **Subscriptions** and it fills as posts come down. +- **If you already have a library on disk**, note that importing it is + currently an API call rather than a button — the manual-scan UI was retired + in July 2026, when imports started arriving entirely via downloads and the + extension. Mount the folder at `./import` and kick it off with: + + ```bash + curl -X POST http://localhost:8080/api/import/trigger + ``` + + Progress shows up under **Settings → Activity**. This is a known rough + edge, not the intended shape (#3367). +- **To download from a paywalled account**, FabledCurator needs that account's + session — see the browser extension below. Without one it can still fetch + public posts. +- **Check Settings → Overview** to confirm the workers are alive. Every long + operation in FabledCurator is a background job, so if the queues are not + running, the UI will look like it is ignoring you rather than like it is + broken. + +## The browser extension + +A Firefox extension does two jobs: it hands your logged-in platform sessions to +FabledCurator so it can download on your behalf, and it adds a creator as a +subscription in one click from their page. + +It ships **inside the web image** — there is no add-on store listing to find. +Go to **Subscriptions → Settings**, find the *Browser extension* card, and click +**Install Firefox extension**. The XPI is Mozilla-signed, so Firefox installs it +like any other add-on; the button serves it directly rather than making you +download and side-load a file. + +It pairs with your instance using an API key generated automatically on first +use. The bar directly under that card shows the key and can rotate it. + +See [extension/README.md](extension/README.md) for what it does in detail. + +## The GPU agent + +Optional, and separate. If you have a desktop with a graphics card, you can run +an agent on it that leases ML jobs from FabledCurator over HTTP, does them on +the GPU, and hands the results back. It never touches the database or Redis, so +it is safe to run somewhere the rest of the stack is not. + +Run it for a burst of tagging, stop it to get your card back. It deploys from +`agent/docker-compose.yml`, not the main stack — see +[agent/README.md](agent/README.md). + +## Upgrading + +```bash +docker compose -f docker-compose.yml pull +docker compose -f docker-compose.yml up -d +``` + +Migrations run automatically on start. Take a database backup first — Settings → +Maintenance has one — because the schema moves forward and does not move back. + +## Deployment posture + +FabledCurator is built to run inside a homelab over plain HTTP. It does not +generate certificates, redirect to HTTPS, or set HSTS. If you want TLS, +terminate it at your reverse proxy. See [Before you expose it](#before-you-expose-it) +for why TLS alone is not enough. + +## Troubleshooting + +**The UI loads but nothing ever finishes.** The web container is up and the +workers are not. `docker compose -f docker-compose.yml ps` — check `worker`, +`scheduler` and `ml-worker` are healthy, not restarting. + +**`docker compose up` started building instead of pulling.** You left off +`-f docker-compose.yml`, so the dev override took over. See [Install](#install). + +**Downloads fail with an auth error.** The stored session for that platform has +expired. Re-capture it with the extension; sessions do not last forever. + +**Which build am I running?** The foot of Settings shows a version and a +channel, and `/api/health` returns the same two fields. There are no version +tags on the images, so this is the authoritative answer. + +--- + +# Developing FabledCurator + +Everything below is about working on FabledCurator rather than running it. If +you are installing it, you are done — see [CONTRIBUTING.md](CONTRIBUTING.md) if +you want to send a patch. + +## Status and channels + +In production. `main` is continuously deployed — every merge builds and +publishes `:latest`, so whatever is on `main` is what is running. Day-to-day +work happens on `dev`, which publishes `:dev`. + +For local development, the dev override handles everything: + +```bash +docker compose up -d # note: no -f, so the override applies +``` + +That builds the images from source, turns on DEBUG logging, and exposes +Postgres and Redis on the host. No `.env` required. ## Versions and tags @@ -47,48 +234,10 @@ Five deployable pieces, built by `.forgejo/workflows/build.yml`: | --- | --- | --- | --- | | **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. | | **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. | -| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. | +| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. See `agent/README.md`. | | **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. | | **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. | -## Quick start - -For local development and testing, just: - -```bash -docker compose up -d -# UI: http://localhost:8080 -``` - -That uses sane dev defaults baked into `docker-compose.yml` and the dev -override (`docker-compose.override.yml`, auto-merged) — local builds, DEBUG -logging, exposed Postgres + Redis ports on the host. No `.env` required. - -For a production-like deployment, override the dev defaults via shell env -or a `.env` file (see `.env.example` for the variable names) and use: - -```bash -docker compose -f docker-compose.yml up -d -# (skips the dev override, so containers pull published :latest images) -``` - -`-f` is doing real work there: it tells Compose to use *only* that file, which -skips `docker-compose.override.yml` and its local builds. What you get is the -`:latest` images — the stable channel, built from `main`. This is the install -path, and it is the one to use if you are running FabledCurator rather than -working on it. - -`:dev` is the other channel: rebuilt from the `dev` branch several times a day, -bleeding edge, no stability promise. Nothing in this repo points an installer at -it, and nothing should. - -The GPU agent is deployed separately, on the machine with the card — -`agent/docker-compose.yml`, not this stack. - -## Deployment posture - -FabledCurator is designed to run inside a self-hosted homelab environment over plain HTTP. If you want TLS, terminate it at your reverse proxy. The app does not generate certificates, redirect to HTTPS, or set HSTS. - ## CI / Forgejo setup Four workflows: `ci.yml` (lint, extension-version check, backend unit tests, @@ -118,6 +267,15 @@ source, so `main` finds `dev`'s signature already cached and makes no second AMO call. That cache is why signing must be one-shot — AMO rejects a re-signed version. +## History + +FabledCurator combines what was +[ImageRepo](https://git.fabledsword.com/bvandeusen/ImageRepo) (gallery, ML, +importer) and +[GallerySubscriber](https://git.fabledsword.com/bvandeusen/GallerySubscriber) +(gallery-dl wrapper, subscriptions, credential capture) into a single product. +Both are superseded; neither is maintained. + ## License **GNU Affero General Public License v3.0** — see [LICENSE](LICENSE). diff --git a/SECURITY.md b/SECURITY.md index b9b29d1..f59adcf 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -32,29 +32,43 @@ they shape what counts as a serious bug here: them, or lets one user of a shared instance read another's is high severity. - **An extension API key.** The Firefox extension authenticates to the backend with a shared key. Anything that leaks it or lets it be bypassed is a way in. -- **A multi-user sharing ACL.** Instances can be shared. A bug that lets one - account see content another has not shared is an access-control failure, not - a cosmetic one. +- **No authentication of its own.** This is the most important thing on this + page. FabledCurator has no login, no user accounts and no permission model — + there is no `User` table and no session auth anywhere in the backend. Every + HTTP client that can reach the port is the administrator, with full read and + write access to everything above, including the stored platform credentials. + Access control is entirely the operator's job, done at the network layer. + Reports that an unauthenticated caller can reach an endpoint are therefore + describing the design; reports that something *crosses the network boundary + the operator drew* — an SSRF, a request forgery that rides a browser the + operator already has open, a path that leaks state to an origin the operator + did not authorise — are in scope and are serious. - **Arbitrary media from the internet.** Downloaded files are decoded, hashed, thumbnailed and fed to ML models. Anything that turns a hostile file into code execution is in scope. ## Deployment posture — read this before reporting -FabledCurator is designed to run **inside a private network, over plain HTTP**. -It does not terminate TLS, redirect to HTTPS, or set HSTS; if you want -transport security, terminate it at your reverse proxy. This is a documented -design decision, not an oversight. +FabledCurator is designed to run **inside a private network, over plain HTTP, +reachable only by its operator**. It does not terminate TLS, redirect to +HTTPS, or set HSTS; if you want transport security, terminate it at your +reverse proxy. It also does not authenticate anyone — see above. These are +documented design decisions, not oversights. -Reports that reduce to "the application is served over HTTP" or "there is no -HSTS header" describe that decision rather than a vulnerability. Reports that -an authenticated operator can cause the software to do something destructive -are usually also by design — the operator is the administrator of their own -instance. +Putting this on the public internet, with or without TLS, hands whoever finds +it your Patreon, SubscribeStar and Pixiv sessions. A reverse proxy that adds +TLS but not an authentication layer does not change that. + +Reports that reduce to "the application is served over HTTP", "there is no +HSTS header", or "the API needs no credentials" describe those decisions +rather than vulnerabilities. Reports that the operator can cause the software +to do something destructive are usually also by design — the operator is the +administrator of their own instance. What remains in scope is everything that crosses a boundary the software is -supposed to hold: between one user and another, between an unauthenticated -visitor and any of it, and between untrusted downloaded content and the host. +actually supposed to hold: between untrusted downloaded content and the host, +between a third-party origin and an operator's open browser session, and +between the credentials at rest and anything that is not the operator. ## Supported versions diff --git a/backend/app/config.py b/backend/app/config.py index 2338f25..410bcf9 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -16,8 +16,11 @@ class Config: celery_broker_url: str celery_result_backend: str + # Sets Quart's app.secret_key. Nothing signs a cookie today (FC has no + # login and no session use), so this currently protects nothing — it is + # required rather than defaulted so that the day something session-backed + # does land, no instance is already running on a value we published. secret_key: str - extension_api_key: str # used by the Firefox extension; lands in FC-3 but read here log_level: str @property @@ -47,6 +50,5 @@ def get_config() -> Config: celery_broker_url=os.environ.get("CELERY_BROKER_URL", "redis://redis:6379/0"), celery_result_backend=os.environ.get("CELERY_RESULT_BACKEND", "redis://redis:6379/0"), secret_key=os.environ["SECRET_KEY"], - extension_api_key=os.environ.get("EXTENSION_API_KEY", ""), log_level=os.environ.get("LOG_LEVEL", "INFO"), ) diff --git a/docker-compose.yml b/docker-compose.yml index 59b4503..cf13fa0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,12 +1,21 @@ -# Base compose stack. Uses ${VAR:-default} interpolation throughout so the -# stack boots with zero config — sane dev defaults baked in. For production -# deployments, override the defaults via shell env vars or a .env file: +# Base compose stack, and the install path. Uses ${VAR:-default} throughout so +# the stack boots with zero config — but those defaults are DEV defaults, and +# two of them (DB_PASSWORD, SECRET_KEY) are published in this file. Copy +# .env.example to .env and set them before running this anywhere real. # -# DB_PASSWORD=...real... SECRET_KEY=...real... docker compose up +# To run FabledCurator: # -# The dev override (docker-compose.override.yml) is auto-merged when you -# run `docker compose up` from this directory and switches images to -# local builds + DEBUG logging. +# docker compose -f docker-compose.yml up -d +# +# The -f is load-bearing. Without it Compose auto-merges +# docker-compose.override.yml, which replaces every image: with a local +# build: and turns on DEBUG logging — the contributor path. Naming this file +# explicitly skips the override and pulls the published :latest images. +# +# FabledCurator has no authentication. Whatever can reach ${PORT} is an +# administrator, including over the stored Patreon/SubscribeStar/Pixiv session +# cookies. Do not publish this port beyond a network you trust — see +# "Before you expose it" in README.md. # Rolling-deploy safety (Swarm / `docker stack deploy`): update one task at a # time, START the new task before stopping the old (zero-downtime via the ingress @@ -123,18 +132,24 @@ services: CELERY_BROKER_URL: redis://redis:6379/0 CELERY_RESULT_BACKEND: redis://redis:6379/0 SECRET_KEY: ${SECRET_KEY:-dev_secret_key_not_for_production_change_me} - EXTENSION_API_KEY: ${EXTENSION_API_KEY:-} LOG_LEVEL: ${LOG_LEVEL:-INFO} volumes: - ./images:/images - ./import:/import - # FC-5 legacy migration: bind-mount the host's ImageRepo images dir - # under /import (FC's existing filesystem scan picks them up). Read-only - # is sufficient — FC copies into /images during the scan. The worker + - # scheduler services see the same /import via their own mounts below - # because of /import volume reuse. Edit the host path to match your - # install before running Settings → Maintenance → Legacy migration. - # - /var/lib/imagerepo/images:/import/imagerepo:ro + # /import is the staging area for ingesting a library you already have + # on disk. Drop files in ./import, or bind-mount an existing directory + # under it as below, then trigger the scan: + # + # curl -X POST http://localhost:8080/api/import/trigger + # + # Read-only is sufficient — FC copies into /images during the scan. The + # worker + scheduler services mount the same /import so the scan can run + # on whichever lane picks it up. + # + # There is no UI button for this: the manual-scan surface was retired + # 2026-07-02 once imports arrived via downloads + the extension, which is + # true for an established install and not for a new one. Tracked in #3367. + # - /srv/media/my-library:/import/my-library:ro depends_on: postgres: { condition: service_healthy } redis: { condition: service_healthy }