docs: README matches current stack and covers ML suggestions
Quick Start had drifted: postgres:16-alpine instead of pgvector/pg16, a single celery-worker + celery-beat layout instead of the actual worker / scheduler / ml-worker split, and no mention of the ML tag-suggestion surface. Updates the compose example to mirror the real docker-compose.yml, adds ml-worker env vars and /models volume docs, and documents the ML maintenance tools in Settings. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,7 @@ A self-hosted image and video gallery application designed for organizing and vi
|
|||||||
- **Video Support** - Playback for video files with automatic transcoding to MP4
|
- **Video Support** - Playback for video files with automatic transcoding to MP4
|
||||||
- **Tagging System** - Organize images with tags (artist, character, series, rating, archive, user-defined)
|
- **Tagging System** - Organize images with tags (artist, character, series, rating, archive, user-defined)
|
||||||
- **Tag Autocomplete** - Quick tag entry with search suggestions
|
- **Tag Autocomplete** - Quick tag entry with search suggestions
|
||||||
|
- **ML Tag Suggestions** - WD14 tagger + SigLIP embedding centroids propose tags you can accept/reject per image
|
||||||
- **Automatic Importing** - Celery-based task queue scans `/import` directory on schedule
|
- **Automatic Importing** - Celery-based task queue scans `/import` directory on schedule
|
||||||
- **Archive Extraction** - Supports ZIP, RAR, 7z and other archive formats
|
- **Archive Extraction** - Supports ZIP, RAR, 7z and other archive formats
|
||||||
- **Duplicate Detection** - Perceptual hash (pHash) comparison to skip similar images
|
- **Duplicate Detection** - Perceptual hash (pHash) comparison to skip similar images
|
||||||
@@ -35,7 +36,7 @@ services:
|
|||||||
retries: 5
|
retries: 5
|
||||||
|
|
||||||
postgres:
|
postgres:
|
||||||
image: postgres:16-alpine
|
image: pgvector/pgvector:pg16
|
||||||
environment:
|
environment:
|
||||||
POSTGRES_USER: imagerepo
|
POSTGRES_USER: imagerepo
|
||||||
POSTGRES_PASSWORD: your_secure_password
|
POSTGRES_PASSWORD: your_secure_password
|
||||||
@@ -52,7 +53,7 @@ services:
|
|||||||
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
||||||
ports:
|
ports:
|
||||||
- "5000:5000"
|
- "5000:5000"
|
||||||
environment:
|
environment: &app_env
|
||||||
- DB_USER=imagerepo
|
- DB_USER=imagerepo
|
||||||
- DB_PASS=your_secure_password
|
- DB_PASS=your_secure_password
|
||||||
- DB_HOST=postgres
|
- DB_HOST=postgres
|
||||||
@@ -68,16 +69,11 @@ services:
|
|||||||
redis:
|
redis:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
||||||
celery-worker:
|
# Heavy processing: import, thumbnail, sidecar, default queues.
|
||||||
|
worker:
|
||||||
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
||||||
command: celery -A app.celery_app:celery worker --loglevel=info -Q scan,import,thumbnail,sidecar,default --concurrency=2
|
command: celery -A app.celery_app:celery worker --loglevel=info -Q import,thumbnail,sidecar,default --concurrency=2
|
||||||
environment:
|
environment: *app_env
|
||||||
- DB_USER=imagerepo
|
|
||||||
- DB_PASS=your_secure_password
|
|
||||||
- DB_HOST=postgres
|
|
||||||
- DB_NAME=imagerepo
|
|
||||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
|
||||||
- CELERY_RESULT_BACKEND=redis://redis:6379/0
|
|
||||||
volumes:
|
volumes:
|
||||||
- ./imagerepo/images:/images
|
- ./imagerepo/images:/images
|
||||||
- ./your-media:/import
|
- ./your-media:/import
|
||||||
@@ -87,20 +83,33 @@ services:
|
|||||||
redis:
|
redis:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
||||||
celery-beat:
|
# Beat scheduler + maintenance/scan worker, split off so long imports don't starve periodic tasks.
|
||||||
|
scheduler:
|
||||||
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
image: git.fabledsword.com/bvandeusen/imagerepo:latest
|
||||||
command: celery -A app.celery_app:celery beat --loglevel=info
|
command: celery -A app.celery_app:celery worker --beat --loglevel=info -Q maintenance,scan --concurrency=1
|
||||||
environment:
|
environment: *app_env
|
||||||
- DB_USER=imagerepo
|
volumes:
|
||||||
- DB_PASS=your_secure_password
|
- ./imagerepo/images:/images
|
||||||
- DB_HOST=postgres
|
- ./your-media:/import
|
||||||
- DB_NAME=imagerepo
|
|
||||||
- CELERY_BROKER_URL=redis://redis:6379/0
|
|
||||||
- CELERY_RESULT_BACKEND=redis://redis:6379/0
|
|
||||||
- IMPORT_EVERY_SECONDS=28800
|
|
||||||
depends_on:
|
depends_on:
|
||||||
- redis
|
postgres:
|
||||||
- celery-worker
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
|
# CPU-only ML inference: WD14 tags + SigLIP embeddings. Models self-heal into ${MODELS_DIR} on start.
|
||||||
|
ml-worker:
|
||||||
|
image: git.fabledsword.com/bvandeusen/imagerepo-ml:latest
|
||||||
|
command: celery -A app.celery_app:celery worker --loglevel=info -Q ml --concurrency=1
|
||||||
|
environment: *app_env
|
||||||
|
volumes:
|
||||||
|
- ./imagerepo/images:/images:ro
|
||||||
|
- ./imagerepo/models:/models
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
redis_data:
|
redis_data:
|
||||||
@@ -114,9 +123,10 @@ Then visit `http://localhost:5000` in your browser.
|
|||||||
ImageRepo uses a task queue architecture for background processing:
|
ImageRepo uses a task queue architecture for background processing:
|
||||||
|
|
||||||
- **Web** - Flask application serving the UI and API
|
- **Web** - Flask application serving the UI and API
|
||||||
- **Celery Worker** - Processes import, thumbnail, and metadata tasks
|
- **Worker** - Heavy processing: `import`, `thumbnail`, `sidecar`, `default` queues
|
||||||
- **Celery Beat** - Schedules periodic tasks (directory scans, recovery)
|
- **Scheduler** - Celery Beat + a `maintenance`/`scan` worker (kept separate so long imports don't starve periodic tasks)
|
||||||
- **PostgreSQL** - Primary database for all data
|
- **ML Worker** - CPU-only WD14 + SigLIP inference on the `ml` queue (separate image, models self-heal on start)
|
||||||
|
- **PostgreSQL** - Primary database, uses `pgvector` extension for SigLIP embedding similarity
|
||||||
- **Redis** - Message broker for Celery task queue
|
- **Redis** - Message broker for Celery task queue
|
||||||
|
|
||||||
## Volumes
|
## Volumes
|
||||||
@@ -125,6 +135,7 @@ ImageRepo uses a task queue architecture for background processing:
|
|||||||
|------|-------------|
|
|------|-------------|
|
||||||
| `/images` | Where imported images and thumbnails are stored |
|
| `/images` | Where imported images and thumbnails are stored |
|
||||||
| `/import` | Source directory the importer scans for new media |
|
| `/import` | Source directory the importer scans for new media |
|
||||||
|
| `/models` | ml-worker only — WD14 + SigLIP weights (~4 GB, fetched on first start) |
|
||||||
|
|
||||||
## Environment Variables
|
## Environment Variables
|
||||||
|
|
||||||
@@ -176,6 +187,16 @@ These can also be configured via the Settings page in the UI.
|
|||||||
| `ARCHIVE_MIN_FREE_GB` | `0` | Minimum free disk space (GB) required to start extraction (0 = disabled) |
|
| `ARCHIVE_MIN_FREE_GB` | `0` | Minimum free disk space (GB) required to start extraction (0 = disabled) |
|
||||||
| `ARCHIVE_NUM_WIDTH` | `4` | Zero-padding width for archive sequence numbers in tags |
|
| `ARCHIVE_NUM_WIDTH` | `4` | Zero-padding width for archive sequence numbers in tags |
|
||||||
|
|
||||||
|
### ML Tag Suggestions (ml-worker only)
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `ML_MODEL_DIR` | `/models` | Where WD14 + SigLIP weights are written/read |
|
||||||
|
| `WD14_REPO` | `SmilingWolf/wd-eva02-large-tagger-v3` | HuggingFace repo for the WD14 tagger |
|
||||||
|
| `SIGLIP_REPO` | `google/siglip-so400m-patch14-384` | HuggingFace repo for the SigLIP encoder |
|
||||||
|
| `WD14_REVISION` | (latest) | Pin WD14 to a specific commit SHA |
|
||||||
|
| `SIGLIP_REVISION` | (latest) | Pin SigLIP to a specific commit SHA |
|
||||||
|
|
||||||
### Other
|
### Other
|
||||||
|
|
||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
@@ -227,6 +248,11 @@ The Settings page (`/settings`) provides:
|
|||||||
- **Find Duplicates** - Scan for visually similar images using pHash
|
- **Find Duplicates** - Scan for visually similar images using pHash
|
||||||
- **Reset Image Database** - Clear all image records (files remain on disk)
|
- **Reset Image Database** - Clear all image records (files remain on disk)
|
||||||
|
|
||||||
|
### ML Maintenance Tools
|
||||||
|
- **Run ML backfill** - Enqueue `tag_and_embed` for every image missing predictions or embeddings for the current model versions. Safe to re-run; paginates forward and drops already-processed images. Expected runtime on a fresh DB: hours to days on a single ml-worker.
|
||||||
|
- **Recompute all centroids** - Rebuild per-tag SigLIP centroids from current image tags. Run after the initial backfill drains, or whenever bulk manual tagging has drifted suggestions.
|
||||||
|
- **Sync character fandoms** - Additively re-apply each character's fandom tag to every image already tagged with that character. Never removes existing fandom tags.
|
||||||
|
|
||||||
### Import Filters
|
### Import Filters
|
||||||
Configure filtering rules that apply during import:
|
Configure filtering rules that apply during import:
|
||||||
- Minimum dimensions
|
- Minimum dimensions
|
||||||
|
|||||||
Reference in New Issue
Block a user