imagerepo/summary.md

# ImageRepo - Codebase Summary

A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.

---

## Architecture Overview

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Flask Web     │     │  Celery Worker  │     │  Celery Beat    │
│   (port 5000)   │     │  (task queues)  │     │  (scheduler)    │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌─────▼─────┐
        │ PostgreSQL│     │    Redis    │    │  /images  │
        │ (database)│     │   (broker)  │    │ /import   │
        └───────────┘     └─────────────┘    └───────────┘
```

---

## File Structure

### Core Application Files

| File | Purpose |
|------|---------|
| `run.py` | Entry point - creates Flask app via `create_app()` |
| `config.py` | Configuration class with DB, Redis, Celery settings |
| `docker-compose.yml` | Container orchestration for all services |

### Application Package (`app/`)

| File | Purpose |
|------|---------|
| `app/__init__.py:56-86` | Flask app factory `create_app()`, CLI commands, initializes extensions |
| `app/main.py` | Blueprint with all HTTP routes and API endpoints |
| `app/models.py` | SQLAlchemy models for database schema |
| `app/celery_app.py` | Celery configuration and task routing |

---

## Database Models (`app/models.py`)

### ImageRecord (lines 19-40)
Primary model for images/videos.
```
Fields: id, filename, filepath, thumb_path, hash (SHA256), perceptual_hash (256-bit),
        file_size, width, height, format, camera_model, taken_at, imported_at, is_thumbnail
Relationships: tags (many-to-many via image_tags)
```

### Tag (lines 42-51)
Tagging system for organization.
```
Fields: id, name (unique), kind (artist|archive|user|source|post|character|series|rating)
Relationships: images (many-to-many)
```

### ArchiveRecord (lines 53-66)
Tracks processed archives to prevent re-importing.
```
Fields: id, filename, file_size, hash, imported_at, artist, tag_id
```

### AppSettings (lines 69-78)
Key-value store for application configuration.
```
Fields: id, key, value, updated_at
```

### ImportBatch (lines 81-101)
Groups import tasks for batch tracking.
```
Fields: id, source_directory, status, total_files, processed_files,
        imported_count, skipped_count, failed_count, started_at, completed_at
```

### PostMetadata (lines 104-133)
Stores metadata about posts from external platforms (Patreon, SubscribeStar, HentaiFoundry).
```
Fields: id, tag_id (FK to post tag), platform, post_id, artist,
        title, description (may contain HTML), source_url,
        attachment_count, published_at, created_at, updated_at
Relationships: tag (one-to-one with Tag where kind='post')
```

### SeriesPage (lines 136-168)
Links images to series tags with page numbers for ordered reading.
```
Fields: id, series_tag_id (FK to tag), image_id (FK to image, unique),
        page_number, created_at, updated_at
Constraints: unique(series_tag_id, page_number), unique(image_id)
Relationships: series_tag, image
```

### ImportTask (lines 171-222)
Individual file import task tracking.
```
Fields: id, source_path, file_hash, file_size, task_type, status, celery_task_id,
        context (JSON), batch_id, result_image_id, error_message, retry_count, timestamps
Status values: pending, queued, processing, complete, failed, skipped
Task types: scan, import_image, import_archive, thumbnail, sidecar
```

---

## Celery Tasks (`app/tasks/`)

### Scan Tasks (`app/tasks/scan.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `scan_directory()` | 29-154 | Walks `/import` directory, creates ImportTask records, queues import tasks |
| `_create_task_if_new()` | 157-228 | Creates ImportTask if file not already processed |
| `recover_interrupted_tasks()` | 231-302 | Re-queues stuck tasks (called periodically by Beat) |
| `update_batch_stats()` | 305-349 | Updates ImportBatch statistics |

### Import Tasks (`app/tasks/import_file.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `import_media_file()` | 23-252 | Main import task - filters, deduplicates, copies, creates thumbnail |
| `import_archive()` | 255-407 | Extracts archive, queues child import tasks |
| `_fail_task()` | 410-425 | Marks task as failed |
| `_skip_task()` | 428-443 | Marks task as skipped |
| `_add_artist_tag()` | 446-459 | Adds artist:name tag to image |
| `_supersede_existing()` | 472-539 | Replaces smaller image with larger version |

### Thumbnail Tasks (`app/tasks/thumbnail.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `generate_thumbnail_task()` | 18-69 | Generate thumbnail for single image/video |
| `regenerate_all_thumbnails()` | 72-108 | Bulk regenerate all thumbnails |
| `regenerate_missing_thumbnails()` | 111-153 | Generate only missing thumbnails |

### Sidecar Tasks (`app/tasks/sidecar.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `apply_sidecar_metadata()` | 18-97 | Apply Gallery-DL JSON metadata to image |
| `scan_and_match_sidecars()` | 100-147 | Find and match sidecar files to images |
| `reprocess_sidecars_for_tag()` | 150-181 | Re-run sidecar extraction for tagged images |

---

## Utility Functions (`app/utils/`)

### Image Importer (`app/utils/image_importer.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `load_import_settings()` | 50-88 | Load filter settings from DB/file/env |
| `is_mostly_transparent()` | 91-146 | Check if image is mostly transparent |
| `is_mostly_single_color()` | 149-215 | Check if image is solid color |
| `supersede_image()` | 218-275 | Replace smaller image with larger version |
| `extract_archive_resilient()` | 359-384 | Extract archive using unar/7z fallback |
| `build_hashed_dest_path()` | 391-408 | Generate content-addressed file path |
| `generate_thumbnail()` | 415-500 | Create thumbnail for image |
| `generate_video_thumbnail_mirrored()` | 524-532 | Create thumbnail for video |
| `transcode_video_to_mp4()` | 535-609 | Transcode video to H.264 MP4 |
| `needs_transcode()` | 612-615 | Check if video needs transcoding |
| `calculate_hash()` | 618-623 | SHA256 content hash |
| `calculate_perceptual_hash()` | 626-643 | 256-bit pHash for similarity |
| `find_similar_image()` | 646-678 | Find similar images by pHash |
| `extract_metadata()` | 691-754 | Extract EXIF/video metadata |
| `is_first_volume()` | 788-823 | Check if archive is first volume of multi-part |
| `compute_next_archive_tag_name()` | 825-890 | Generate unique archive:artist/NNNN tag |

### Metadata Enrichment (`app/utils/metadata_enrichment.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `find_sidecar_json()` | 27-83 | Find Gallery-DL sidecar JSON for image |
| `load_sidecar_metadata()` | 86-93 | Parse sidecar JSON file |
| `extract_platform()` | 100-117 | Get platform from metadata |
| `extract_artist()` | 120-136 | Get artist/creator name |
| `extract_post_id()` | 139-155 | Get post ID |
| `extract_post_title()` | 158-164 | Get post title |
| `extract_description()` | 167-194 | Get post description/content (HTML or plain text) |
| `extract_source_url()` | 197-228 | Get URL to original post (constructs for SS/HF) |
| `extract_attachment_count()` | 231-248 | Get number of attachments in post |
| `extract_date()` | 251-271 | Extract publication date |
| `generate_source_tags()` | 338-369 | Generate source:platform and post:... tags |
| `resolve_date_conflict()` | 376-405 | Keep earliest date |
| `enrich_from_sidecar()` | 412-470 | Main enrichment function (returns platform, artist, title, description, source_url, etc.) |

### Version Migration (`app/utils/version_migration.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `get_setting()` / `set_setting()` | 43-74 | Read/write AppSettings |
| `get_setting_int/float/bool()` | 77-104 | Typed setting getters |
| `get_import_settings()` | 111-122 | Load import filter settings from DB |
| `calculate_phash_256()` | 129-139 | Calculate 256-bit perceptual hash |
| `recompute_all_phashes()` | 146-218 | Batch recompute all pHashes |
| `check_and_run_migrations()` | 221-268 | Run data migrations on version change |

---

## HTTP Routes (`app/main.py`)

### Page Routes

| Route | Line | Purpose |
|-------|------|---------|
| `GET /` | 18-30 | Showcase - random images masonry |
| `GET /gallery` | 80-124 | Main gallery with infinite scroll |
| `GET /tags` | 391-475 | Tag explorer (filterable by kind) |
| `GET /artists` | 477-480 | Redirect to tag_list?kind=artist |
| `GET /settings` | 488-490 | Settings page |

### Gallery API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/random-images` | 33-77 | Random images for showcase shuffle |
| `GET /api/gallery/scroll` | 188-259 | Infinite scroll pagination |
| `GET /api/gallery/timeline` | 262-316 | Year-month groups for sidebar |
| `GET /api/gallery/jump` | 319-388 | Jump to specific year-month |
| `GET /images/<path:filename>` | 127-129 | Serve image files |

### Tag API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/tags/search` | 497-537 | Tag autocomplete search |
| `GET /image/<id>/tags` | 540-543 | List tags for image |
| `POST /image/<id>/tags/add` | 546-573 | Add tag to image |
| `POST /image/<id>/tags/remove` | 576-587 | Remove tag from image |
| `GET /api/tag/<id>` | 594-605 | Get tag details |
| `GET /api/post-metadata/<id>` | 608-651 | Get post metadata for a post tag |
| `GET /api/post-metadata/by-tag-name/<name>` | 654-685 | Get post metadata by tag name |
| `POST /api/tag/<id>/update` | 688-764 | Update/merge tag |
| `POST /api/tag/<id>/delete` | 767-776 | Delete tag |
| `GET /api/tag/<id>/image-count` | 842-847 | Count images with tag |

### Bulk Operations

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/delete-by-tag` | 703-759 | Delete all images with tag |
| `POST /api/images/common-tags` | 1394-1425 | Get common tags across images |
| `POST /api/images/bulk-add-tag` | 1428-1474 | Add tag to multiple images |
| `POST /api/images/bulk-remove-tag` | 1477-1514 | Remove tag from multiple images |

### Series / Reader API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /read/<tag_id>` | 1620-1650 | Reader view for a series (vertical scroll) |
| `GET /api/series/<tag_id>/pages` | 1653-1685 | Get all pages in a series |
| `POST /api/series/<tag_id>/pages/add` | 1688-1737 | Add single image to series with page number |
| `POST /api/series/<tag_id>/pages/bulk-add` | 1740-1800 | Add multiple images with sequential page numbers |
| `POST /api/series/page/<page_id>/update` | 1803-1828 | Update a page's page number |
| `POST /api/series/page/<page_id>/delete` | 1831-1841 | Remove image from series |
| `GET /api/image/<image_id>/series-info` | 1844-1858 | Check if image is in a series |

### Import Queue API

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/import/trigger` | 961-981 | Start directory scan |
| `GET /api/import/status` | 984-1042 | Queue status and recent tasks |
| `GET /api/import/task/<id>` | 1045-1082 | Task details |
| `POST /api/import/retry-failed` | 1085-1122 | Retry failed tasks |
| `POST /api/import/clear-completed` | 1125-1144 | Clear completed tasks |
| `GET /api/import/batches` | 1222-1250 | List import batches |

### Settings API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/import-settings` | 774-788 | Get import filter settings |
| `POST /api/import-settings` | 791-821 | Save import filter settings |
| `GET /api/filter-scan` | 828-902 | Scan images against filters |
| `POST /api/filter-delete` | 905-954 | Delete filtered images |

### Thumbnail API

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/thumbnails/regenerate` | 1147-1164 | Regenerate all thumbnails |
| `POST /api/thumbnails/regenerate-missing` | 1167-1183 | Generate missing thumbnails |

### Celery Status

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/celery/status` | 1186-1219 | Worker and queue status |

### Duplicate Detection

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/duplicates/scan` | 1257-1343 | Scan for similar images by pHash |
| `POST /api/duplicates/delete` | 1346-1387 | Delete duplicate images |

---

## Frontend Templates (`app/templates/`)

| Template | Purpose |
|----------|---------|
| `layout.html` | Base template with navbar |
| `showcase.html` | Random image masonry view |
| `gallery.html` | Main gallery with infinite scroll, timeline sidebar, bulk editor, series editor |
| `reader.html` | Series reader with vertical scroll, page jump, thumbnail navigation |
| `tags_list.html` | Tag browser with preview images |
| `settings.html` | Import settings, deletion tools, duplicate detection |
| `_gallery_item.html` | Single gallery item partial |
| `_gallery_modal.html` | Image modal with tag editor |
| `_gallery_grid.html` | Gallery grid partial |
| `_pagination.html` | Pagination controls |

### JavaScript Files (`app/static/js/`)

| File | Purpose |
|------|---------|
| `gallery-infinite.js` | Infinite scroll, timeline navigation |
| `modal-pagination.js` | Modal image navigation |
| `bulk-select.js` | Multi-select and bulk tag editing |
| `tag-editor.js` | Tag autocomplete and editing |
| `showcase.js` | Showcase shuffle functionality |

---

## Docker Services

| Service | Purpose |
|---------|---------|
| `redis` | Message broker for Celery |
| `postgres` | PostgreSQL database |
| `web` | Flask application (port 5000) |
| `celery-worker` | Task processor (queues: scan, import, thumbnail, sidecar, default) |
| `celery-beat` | Periodic task scheduler |

---

## Key Concepts

### Content-Addressed Storage
Files are stored with hash suffix: `{filename}__{hash[:10]}.{ext}`

### Perceptual Hashing
256-bit pHash for duplicate detection (threshold default: 10)

### Tag Kinds
- `artist` - Artist/creator (prefix: `artist:name`)
- `archive` - Archive source (prefix: `archive:artist/NNNN`)
- `source` - Platform source (prefix: `source:platform`)
- `post` - Post reference (prefix: `post:platform:artist:id`)
- `user` - User-created tags (no prefix)
- `character`, `series`, `rating` - Content tags

### Import Pipeline
1. `scan_directory` walks `/import/{artist}/` folders
2. Creates ImportTask for each file
3. `import_media_file` or `import_archive` processes file
4. Applies filters (dimensions, transparency, duplicates)
5. Copies to `/images/{artist}/` with hash suffix
6. Generates thumbnail
7. Applies sidecar metadata if found
8. Creates ImageRecord with tags

### Sidecar Metadata
Gallery-DL JSON files provide:
- Platform (patreon, hentaifoundry, etc.)
- Artist name
- Post ID and title
- Publication date
- Source URL

---

## Environment Variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `DB_USER` | imagerepo | PostgreSQL user |
| `DB_PASS` | postgres | PostgreSQL password |
| `DB_HOST` | postgres | PostgreSQL host |
| `DB_NAME` | imagerepo | Database name |
| `CELERY_BROKER_URL` | redis://redis:6379/0 | Redis broker |
| `CELERY_WORKER_CONCURRENCY` | 2 | Worker processes |
| `IMPORT_EVERY_SECONDS` | 28800 | Auto-scan interval (8h) |
| `IMPORT_MIN_WIDTH` | 0 | Minimum image width |
| `IMPORT_MIN_HEIGHT` | 0 | Minimum image height |
| `IMPORT_SKIP_TRANSPARENT` | false | Skip transparent images |
| `IMPORT_TRANSPARENCY_THRESHOLD` | 0.9 | Transparency % threshold |
| `IMPORT_PHASH_THRESHOLD` | 10 | pHash similarity threshold |

---

## CLI Commands

```bash
flask version-check    # Run data migrations
flask celery-worker    # Start Celery worker
flask celery-beat      # Start Celery Beat scheduler
```