# ImageRepo

A self-hosted image and video gallery application designed for organizing and viewing media collections. Originally built for managing Patreon content archives, it features automatic importing, tagging, duplicate detection, and a responsive dark-themed UI.

## Features

- **Gallery View** - Browse images in a paginated grid with thumbnail previews
- **Showcase View** - Randomized masonry layout for discovery
- **Image Modal** - Full-size viewing with zoom, pan, and keyboard navigation
- **Video Support** - Playback for video files with automatic transcoding to MP4
- **Tagging System** - Organize images with tags (artist, character, series, rating, archive, user-defined)
- **Tag Autocomplete** - Quick tag entry with search suggestions
- **ML Tag Suggestions** - WD14 tagger + SigLIP embedding centroids propose tags you can accept/reject per image. Videos are sampled at 10 frames (configurable via `VIDEO_ML_FRAMES`) and aggregated by max-confidence WD14 + mean-pool SigLIP.
- **Bulk Tag Suggestions** - Select multiple images in the gallery to see consensus ML suggestions (tags suggested for or already applied to >=80% of the selection)
- **Automatic Importing** - Celery-based task queue scans `/import` directory on schedule
- **Archive Extraction** - Supports ZIP, RAR, 7z and other archive formats
- **Duplicate Detection** - Perceptual hash (pHash) comparison to skip similar images
- **Import Filters** - Skip images by dimensions or transparency percentage
- **Bulk Deletion** - Delete images by tag or filter criteria
- **Artist Organization** - Auto-tags images based on folder structure
- **Dark Theme** - Easy on the eyes for extended browsing

## Quick Start

### Docker Compose

```yaml
services:
  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_USER: imagerepo
      POSTGRES_PASSWORD: your_secure_password
      POSTGRES_DB: imagerepo
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U imagerepo"]
      interval: 10s
      timeout: 5s
      retries: 5

  web:
    image: git.fabledsword.com/bvandeusen/imagerepo:latest
    ports:
      - "5000:5000"
    environment: &app_env
      - DB_USER=imagerepo
      - DB_PASS=your_secure_password
      - DB_HOST=postgres
      - DB_NAME=imagerepo
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    volumes:
      - ./imagerepo/images:/images
      - ./your-media:/import
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

  # Heavy processing: import, thumbnail, sidecar, default queues.
  worker:
    image: git.fabledsword.com/bvandeusen/imagerepo:latest
    command: celery -A app.celery_app:celery worker --loglevel=info -Q import,thumbnail,sidecar,default --concurrency=2
    environment: *app_env
    volumes:
      - ./imagerepo/images:/images
      - ./your-media:/import
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

  # Beat scheduler + maintenance/scan worker, split off so long imports don't starve periodic tasks.
  scheduler:
    image: git.fabledsword.com/bvandeusen/imagerepo:latest
    command: celery -A app.celery_app:celery worker --beat --loglevel=info -Q maintenance,scan --concurrency=1
    environment: *app_env
    volumes:
      - ./imagerepo/images:/images
      - ./your-media:/import
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

  # CPU-only ML inference: WD14 tags + SigLIP embeddings. Models self-heal into ${MODELS_DIR} on start.
  ml-worker:
    image: git.fabledsword.com/bvandeusen/imagerepo-ml:latest
    command: celery -A app.celery_app:celery worker --loglevel=info -Q ml --concurrency=1
    environment: *app_env
    volumes:
      - ./imagerepo/images:/images:ro
      - ./imagerepo/models:/models
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

volumes:
  redis_data:
  postgres_data:
```

Then visit `http://localhost:5000` in your browser.

## Architecture

ImageRepo uses a task queue architecture for background processing:

- **Web** - Flask application serving the UI and API
- **Worker** - Heavy processing: `import`, `thumbnail`, `sidecar`, `default` queues
- **Scheduler** - Celery Beat + a `maintenance`/`scan` worker (kept separate so long imports don't starve periodic tasks)
- **ML Worker** - CPU-only WD14 + SigLIP inference on the `ml` queue (separate image, models self-heal on start)
- **PostgreSQL** - Primary database, uses `pgvector` extension for SigLIP embedding similarity
- **Redis** - Message broker for Celery task queue

## Volumes

| Path | Description |
|------|-------------|
| `/images` | Where imported images and thumbnails are stored |
| `/import` | Source directory the importer scans for new media |
| `/models` | ml-worker only — WD14 + SigLIP weights (~4 GB, fetched on first start) |

## Environment Variables

### Database Configuration (PostgreSQL)

| Variable | Default | Description |
|----------|---------|-------------|
| `DB_NAME` | `imagerepo` | PostgreSQL database name |
| `DB_USER` | `imagerepo` | PostgreSQL username |
| `DB_PASS` | `postgres` | PostgreSQL password |
| `DB_HOST` | `postgres` | PostgreSQL host |
| `DB_PORT` | `5432` | PostgreSQL port |

### Celery Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `CELERY_BROKER_URL` | `redis://redis:6379/0` | Redis broker URL |
| `CELERY_RESULT_BACKEND` | `redis://redis:6379/0` | Redis result backend URL |
| `CELERY_WORKER_CONCURRENCY` | `2` | Number of worker processes |
| `IMPORT_EVERY_SECONDS` | `28800` | Directory scan interval (8 hours) |

### Import Settings

These can also be configured via the Settings page in the UI.

| Variable | Default | Description |
|----------|---------|-------------|
| `IMPORT_MIN_WIDTH` | `0` | Minimum image width in pixels (0 = no minimum) |
| `IMPORT_MIN_HEIGHT` | `0` | Minimum image height in pixels (0 = no minimum) |
| `IMPORT_SKIP_TRANSPARENT` | `false` | Skip mostly-transparent images (PNG/GIF/WebP) |
| `IMPORT_TRANSPARENCY_THRESHOLD` | `0.9` | Transparency % threshold (0.0-1.0) for skipping |
| `IMPORT_PHASH_THRESHOLD` | `10` | Duplicate detection sensitivity (0=disabled, lower=stricter) |

### Thumbnail & Video Processing

| Variable | Default | Description |
|----------|---------|-------------|
| `THUMBS_VERBOSE` | `0` | Enable verbose logging (`1`, `true`, or `yes`) |
| `THUMBS_LOG_EVERY` | `50` | Log progress every N thumbnails |
| `THUMBS_FFMPEG_TIMEOUT` | `60` | Timeout in seconds for video thumbnail extraction |
| `FFMPEG_TRANSCODE_TIMEOUT` | `600` | Timeout in seconds for video transcoding |

### Archive Extraction

| Variable | Default | Description |
|----------|---------|-------------|
| `ARCHIVE_TMPDIR` | System temp | Directory for extracting archives |
| `ARCHIVE_MIN_FREE_GB` | `0` | Minimum free disk space (GB) required to start extraction (0 = disabled) |
| `ARCHIVE_NUM_WIDTH` | `4` | Zero-padding width for archive sequence numbers in tags |

### ML Tag Suggestions (ml-worker only)

| Variable | Default | Description |
|----------|---------|-------------|
| `ML_MODEL_DIR` | `/models` | Where WD14 + SigLIP weights are written/read |
| `WD14_REPO` | `SmilingWolf/wd-eva02-large-tagger-v3` | HuggingFace repo for the WD14 tagger |
| `SIGLIP_REPO` | `google/siglip-so400m-patch14-384` | HuggingFace repo for the SigLIP encoder |
| `WD14_REVISION` | (latest) | Pin WD14 to a specific commit SHA |
| `SIGLIP_REVISION` | (latest) | Pin SigLIP to a specific commit SHA |
| `VIDEO_ML_FRAMES` | `10` | Frames sampled per video for WD14+SigLIP inference (between 10% and 90% of duration) |
| `WD14_STORE_FLOOR` | `0.05` | Minimum WD14 confidence stored in the DB |

### Other

| Variable | Default | Description |
|----------|---------|-------------|
| `PIL_MAX_IMAGE_PIXELS` | `178956970` | Maximum image size Pillow will process (guards against decompression bombs) |

## Import Behavior

The importer runs:
- Automatically every 8 hours (configurable via `IMPORT_EVERY_SECONDS`)
- Manually via the "Trigger Import Scan" button in Settings

### Video Transcoding

Non-MP4 video formats (.mov, .avi, .mkv, .webm, .m4v, .wmv, .flv) are automatically transcoded to H.264 MP4 during import for universal browser playback.

### Folder Structure

The importer uses the folder structure to auto-tag images:

```
/import/
├── ArtistName/           # Tagged as "artist:ArtistName"
│   ├── image1.png
│   ├── image2.jpg
│   └── archive.zip       # Extracted, contents tagged with "archive:archive"
├── AnotherArtist/
│   └── subfolder/
│       └── image3.png    # Still tagged as "artist:AnotherArtist"
```

- Top-level folders become `artist:FolderName` tags
- Archives (ZIP, RAR, 7z, etc.) are extracted and their contents tagged with `archive:filename`
- Nested subfolders inherit the top-level artist tag

### Supported Formats

**Images:** PNG, JPG, JPEG, GIF, WebP, BMP, TIFF
**Videos:** MP4, MOV, AVI, MKV, WebM, M4V, WMV, FLV (transcoded to MP4)
**Archives:** ZIP, RAR, 7z, and others supported by `unar`/`7z`

## Settings Page

The Settings page (`/settings`) provides:

### Admin Actions
- **Trigger Import Scan** - Manually start an import scan
- **Regenerate Thumbnails** - Rebuild all thumbnails
- **Find Duplicates** - Scan for visually similar images using pHash
- **Reset Image Database** - Clear all image records (files remain on disk)

### ML Maintenance Tools
- **Run ML backfill** - Enqueue `tag_and_embed` for every image missing predictions or embeddings for the current model versions. Safe to re-run; paginates forward and drops already-processed images. Expected runtime on a fresh DB: hours to days on a single ml-worker.
- **Recompute all centroids** - Rebuild per-tag SigLIP centroids from current image tags. Run after the initial backfill drains, or whenever bulk manual tagging has drifted suggestions.
- **Sync character fandoms** - Additively re-apply each character's fandom tag to every image already tagged with that character. Never removes existing fandom tags.

### Import Filters
Configure filtering rules that apply during import:
- Minimum dimensions
- Transparency filtering
- Duplicate detection sensitivity

### Duplicate Management
- Scan for potential duplicates based on perceptual hash distance
- Review and confirm duplicates for deletion
- Keep the higher resolution version automatically

### Delete Images by Tag
Bulk delete all images associated with a specific tag (e.g., remove an entire artist's collection).

### Clean Up Existing Images
Scan and selectively delete images that match filter criteria (transparency or dimensions).

## API Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/random-images` | GET | Get random images for showcase |
| `/api/tags/search` | GET | Search tags for autocomplete |
| `/api/import-settings` | GET/POST | Read/write import filter settings |
| `/api/import/trigger` | POST | Trigger directory scan |
| `/api/import/status` | GET | Get import queue status |
| `/api/duplicates/scan` | POST | Scan for duplicate images |
| `/api/duplicates/delete` | POST | Delete confirmed duplicates |
| `/api/filter-scan` | GET | Scan images matching filter criteria |
| `/api/filter-delete` | POST | Delete selected images |
| `/api/delete-by-tag` | POST | Delete all images with a tag |
| `/image/<id>/tags` | GET | List tags for an image |
| `/image/<id>/tags/add` | POST | Add a tag to an image |
| `/image/<id>/tags/remove` | POST | Remove a tag from an image |

## Keyboard Shortcuts

In the image modal:
- **Arrow Left/Right** - Previous/Next image
- **Escape** - Close modal
- **Click image** - Toggle zoom
- **Drag** (when zoomed) - Pan around image

## Scaling Workers

To increase import throughput:

```bash
# Scale worker containers
docker-compose up --scale celery-worker=4

# Or increase concurrency per worker
CELERY_WORKER_CONCURRENCY=4 docker-compose up
```

Total parallelism = containers × concurrency

## Development

### Local Setup

```bash
# Clone the repository
git clone https://git.fabledsword.com/bvandeusen/imagerepo.git
cd imagerepo

# Start PostgreSQL and Redis
docker-compose up -d postgres redis

# Create virtual environment
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

# Install dependencies
pip install -r requirements.txt

# Run migrations
flask db upgrade

# Run with Flask dev server
flask run
```

### Docker Build

```bash
docker build -t imagerepo .
```

## License

This is a personal project. Use at your own discretion.