imagerepo/summary.md

# ImageRepo - Codebase Summary

A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.

> **IMPORTANT**: This summary must be kept up to date with any code changes. Update the timestamp below when making modifications.
>
> **Last Updated**: 2026-02-05 (Consolidated JS utilities, shared modal template, Tags dropdown gap fix)

---

## Architecture Overview

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Flask Web     │     │  Celery Worker  │     │  Celery Beat    │
│   (port 5000)   │     │  (task queues)  │     │  (scheduler)    │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌─────▼─────┐
        │ PostgreSQL│     │    Redis    │    │  /images  │
        │ (database)│     │   (broker)  │    │ /import   │
        └───────────┘     └─────────────┘    └───────────┘
```

---

## File Structure

### Core Application Files

| File | Purpose |
|------|---------|
| `run.py` | Entry point - creates Flask app via `create_app()` |
| `config.py` | Configuration class with DB, Redis, Celery settings |
| `docker-compose.yml` | Container orchestration for all services |

### Application Package (`app/`)

| File | Purpose |
|------|---------|
| `app/__init__.py:56-86` | Flask app factory `create_app()`, CLI commands, initializes extensions |
| `app/main.py` | Blueprint with all HTTP routes and API endpoints |
| `app/models.py` | SQLAlchemy models for database schema |
| `app/celery_app.py` | Celery configuration and task routing |

---

## Database Models (`app/models.py`)

### ImageRecord (lines 19-40)
Primary model for images/videos.
```
Fields: id, filename, filepath, thumb_path, hash (SHA256), perceptual_hash (256-bit),
        file_size, width, height, format, camera_model, taken_at, imported_at, is_thumbnail
Relationships: tags (many-to-many via image_tags)
```

### Tag (lines 42-51)
Tagging system for organization.
```
Fields: id, name (unique), kind (artist|archive|user|source|post|character|series|fandom|rating)
Relationships: images (many-to-many)
```

### ArchiveRecord (lines 53-66)
Tracks processed archives to prevent re-importing.
```
Fields: id, filename, file_size, hash, imported_at, artist, tag_id
```

### AppSettings (lines 69-78)
Key-value store for application configuration.
```
Fields: id, key, value, updated_at
```

### ImportBatch (lines 81-101)
Groups import tasks for batch tracking.
```
Fields: id, source_directory, status, total_files, processed_files,
        imported_count, skipped_count, failed_count, started_at, completed_at
```

### PostMetadata (lines 104-133)
Stores metadata about posts from external platforms (Patreon, SubscribeStar, HentaiFoundry).
```
Fields: id, tag_id (FK to post tag), platform, post_id, artist,
        title, description (may contain HTML), source_url,
        attachment_count, published_at, created_at, updated_at
Relationships: tag (one-to-one with Tag where kind='post')
```

### SeriesPage (lines 136-168)
Links images to series tags with page numbers for ordered reading.
```
Fields: id, series_tag_id (FK to tag), image_id (FK to image, unique),
        page_number, created_at, updated_at
Constraints: unique(series_tag_id, page_number), unique(image_id)
Relationships: series_tag, image
```

### ImportTask (lines 171-222)
Individual file import task tracking.
```
Fields: id, source_path, file_hash, file_size, task_type, status, celery_task_id,
        context (JSON), batch_id, result_image_id, error_message, retry_count, timestamps
Status values: pending, queued, processing, complete, failed, skipped
Task types: scan, import_image, import_archive, thumbnail, sidecar
```

---

## Celery Tasks (`app/tasks/`)

### Scan Tasks (`app/tasks/scan.py`)

Two scan modes are available:
- **Quick scan** (default): Pre-loads existing task/hash data for O(1) lookups. Only queues truly new files. Skips pHash comparison for speed.
- **Deep scan** (on-demand): Full reprocessing of all files. Re-extracts metadata, runs pHash comparison, optionally regenerates thumbnails and re-applies sidecars.

| Function | Purpose |
|----------|---------|
| `_build_task_lookup()` | Pre-loads all ImportTask records into dict for O(1) checks (single query) |
| `_should_skip_file()` | Check if file should be skipped using pre-loaded lookup |
| `scan_directory()` | Quick scan - walks `/import`, uses O(1) lookups, batches task creation |
| `_flush_pending_tasks()` | Batch add tasks and commit for efficiency |
| `deep_scan_directory()` | Deep scan - full reprocessing, pHash comparison, optional thumbnail regen |
| `recover_interrupted_tasks()` | Re-queues stuck tasks (called periodically by Beat) |
| `cleanup_old_tasks()` | Deletes failed/skipped tasks older than 7 days (daily) |
| `update_batch_stats()` | Updates ImportBatch statistics |

### Import Tasks (`app/tasks/import_file.py`)

Import tasks support two modes based on `context.deep_scan`:
- **Quick mode** (default): Fast duplicate detection by content hash only. Skips pHash comparison.
- **Deep mode**: Full reprocessing with pHash comparison, optional thumbnail regen, sidecar re-application.

**Archive tag merging**: When a duplicate is found (by hash or pHash similarity), if the file came from an archive, the archive tag is added to the existing image. This ensures images are tagged with all archives they appear in.

| Function | Purpose |
|----------|---------|
| `_get_cached_settings()` | Cached import settings (5-min TTL) to avoid repeated DB reads |
| `import_media_file()` | Main import task - supports quick/deep modes, filters, deduplicates |
| `import_archive()` | Extracts archive, queues child import tasks with deep scan context |
| `_fail_task()` | Marks task as failed |
| `_skip_task()` | Marks task as skipped |
| `_add_artist_tag()` | Adds artist:name tag to image |
| `_add_archive_tag()` | Adds archive tag to image (for files from archives) |
| `_regenerate_thumbnail()` | Regenerate thumbnail for existing image (deep scan) |
| `_supersede_existing()` | Replaces smaller image with larger version |

### Thumbnail Tasks (`app/tasks/thumbnail.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `generate_thumbnail_task()` | 18-69 | Generate thumbnail for single image/video |
| `regenerate_all_thumbnails()` | 72-108 | Bulk regenerate all thumbnails |
| `regenerate_missing_thumbnails()` | 111-153 | Generate only missing thumbnails |

### Sidecar Tasks (`app/tasks/sidecar.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `apply_sidecar_metadata()` | 18-97 | Apply Gallery-DL JSON metadata to image |
| `scan_and_match_sidecars()` | 100-147 | Find and match sidecar files to images |
| `reprocess_sidecars_for_tag()` | 150-181 | Re-run sidecar extraction for tagged images |

---

## Utility Functions (`app/utils/`)

### Image Importer (`app/utils/image_importer.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `load_import_settings()` | 50-88 | Load filter settings from DB/file/env |
| `is_mostly_transparent()` | 91-146 | Check if image is mostly transparent |
| `is_mostly_single_color()` | 149-215 | Check if image is solid color |
| `supersede_image()` | 218-275 | Replace smaller image with larger version |
| `extract_archive_resilient()` | 359-384 | Extract archive using unar/7z fallback |
| `build_hashed_dest_path()` | 391-408 | Generate content-addressed file path |
| `generate_thumbnail()` | 415-500 | Create thumbnail for image |
| `generate_video_thumbnail_mirrored()` | 524-532 | Create thumbnail for video |
| `transcode_video_to_mp4()` | 535-609 | Transcode video to H.264 MP4 |
| `needs_transcode()` | 612-615 | Check if video needs transcoding |
| `calculate_hash()` | 618-623 | SHA256 content hash |
| `calculate_perceptual_hash()` | 626-643 | 256-bit pHash for similarity |
| `find_similar_image()` | 646-678 | Find similar images by pHash |
| `extract_metadata()` | 691-754 | Extract EXIF/video metadata |
| `is_first_volume()` | 788-823 | Check if archive is first volume of multi-part |
| `compute_next_archive_tag_name()` | 825-890 | Generate unique archive:artist/NNNN tag |

### Metadata Enrichment (`app/utils/metadata_enrichment.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `find_sidecar_json()` | 27-83 | Find Gallery-DL sidecar JSON for image |
| `load_sidecar_metadata()` | 86-93 | Parse sidecar JSON file |
| `extract_platform()` | 100-117 | Get platform from metadata |
| `extract_artist()` | 120-136 | Get artist/creator name |
| `extract_post_id()` | 139-155 | Get post ID |
| `extract_post_title()` | 158-164 | Get post title |
| `extract_description()` | 167-194 | Get post description/content (HTML or plain text) |
| `extract_source_url()` | 197-228 | Get URL to original post (constructs for SS/HF) |
| `extract_attachment_count()` | 231-248 | Get number of attachments in post |
| `extract_date()` | 251-271 | Extract publication date |
| `generate_source_tags()` | 338-369 | Generate source:platform and post:... tags |
| `resolve_date_conflict()` | 376-405 | Keep earliest date |
| `enrich_from_sidecar()` | 412-470 | Main enrichment function (returns platform, artist, title, description, source_url, etc.) |

### Version Migration (`app/utils/version_migration.py`)

| Function | Line | Purpose |
|----------|------|---------|
| `get_setting()` / `set_setting()` | 43-74 | Read/write AppSettings |
| `get_setting_int/float/bool()` | 77-104 | Typed setting getters |
| `get_import_settings()` | 111-122 | Load import filter settings from DB |
| `calculate_phash_256()` | 129-139 | Calculate 256-bit perceptual hash |
| `recompute_all_phashes()` | 146-218 | Batch recompute all pHashes |
| `check_and_run_migrations()` | 221-268 | Run data migrations on version change |

---

## HTTP Routes (`app/main.py`)

### Page Routes

| Route | Line | Purpose |
|-------|------|---------|
| `GET /` | 18-30 | Showcase - random images masonry |
| `GET /gallery` | 80-160 | Main gallery with infinite scroll |
| `GET /tags` | 426-510 | Tag explorer (filterable by kind) |
| `GET /artists` | 512-515 | Redirect to tag_list?kind=artist |
| `GET /settings` | 523-529 | Settings page |

### Gallery API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/random-images` | 33-77 | Random images for showcase shuffle |
| `GET /api/gallery/scroll` | 223-295 | Infinite scroll pagination |
| `GET /api/gallery/timeline` | 297-352 | Year-month groups for sidebar |
| `GET /api/gallery/jump` | 354-423 | Jump to specific year-month |
| `GET /images/<path:filename>` | 162-220 | Serve image files |

### Tag API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/tags/search` | 532-579 | Tag autocomplete search (supports `kind` filter) |
| `GET /api/tags/list` | 535-555 | Paginated tag list for infinite scroll |
| `GET /image/<id>/tags` | 581-585 | List tags for image |
| `POST /image/<id>/tags/add` | 587-615 | Add tag to image |
| `POST /image/<id>/tags/remove` | 617-633 | Remove tag from image |
| `GET /api/tag/<id>` | 635-647 | Get tag details |
| `GET /api/post-metadata/<id>` | 649-680 | Get post metadata for a post tag |
| `GET /api/post-metadata/by-tag-name/<name>` | 682-714 | Get post metadata by tag name |
| `POST /api/tag/<id>/update` | 716-793 | Update/merge tag |
| `POST /api/tag/<id>/delete` | 795-809 | Delete tag |
| `GET /api/tag/<id>/image-count` | 870-880 | Count images with tag |

### Bulk Operations

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/delete-by-tag` | 811-868 | Delete all images with tag |
| `POST /api/images/common-tags` | 1604-1636 | Get common tags across images |
| `POST /api/images/bulk-add-tag` | 1638-1685 | Add tag to multiple images |
| `POST /api/images/bulk-remove-tag` | 1687-1729 | Remove tag from multiple images |

### Series / Reader API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /read/<tag_id>` | 1731-1763 | Reader view for a series (vertical scroll) |
| `GET /api/series/<tag_id>/pages` | 1765-1795 | Get all pages in a series |
| `POST /api/series/<tag_id>/pages/add` | 1797-1853 | Add single image to series with page number |
| `POST /api/series/<tag_id>/pages/bulk-add` | 1855-1915 | Add multiple images with sequential page numbers |
| `POST /api/series/page/<page_id>/update` | 1917-1946 | Update a page's page number |
| `POST /api/series/page/<page_id>/delete` | 1948-1958 | Remove image from series |
| `GET /api/image/<image_id>/series-info` | 1960-1976 | Check if image is in a series |
| `POST /api/image/<image_id>/add-to-series` | 1978-2070 | Add/update/move image in series |

### Import Queue API

| Route | Purpose |
|-------|---------|
| `POST /api/import/trigger` | Start scan (params: `deep=true` for deep scan, `regenerate_thumbnails`, `reapply_sidecars`, `verify_hashes`) |
| `GET /api/import/status` | Queue status counts and active batch info |
| `GET /api/import/tasks` | List tasks with filtering (params: `status`, `task_type`, `search`, `limit`, `offset`) |
| `GET /api/import/task/<id>` | Task details |
| `POST /api/import/retry-failed` | Retry failed tasks |
| `POST /api/import/clear-completed` | Clear tasks (params: `statuses`, `older_than_days` for age-based cleanup) |
| `GET /api/import/task-stats` | Task counts with age breakdown (today, this week, older) |
| `GET /api/import/batches` | List import batches |

### System Stats API

| Route | Purpose |
|-------|---------|
| `GET /api/system/stats` | System overview: total images, tags, storage used, pending imports |

### Settings API

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/import-settings` | 882-897 | Get import filter settings |
| `POST /api/import-settings` | 899-934 | Save import filter settings |
| `GET /api/filter-scan` | 936-1011 | Scan images against filters |
| `POST /api/filter-delete` | 1013-1067 | Delete filtered images |

### Thumbnail API

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/thumbnails/regenerate` | 1255-1273 | Regenerate all thumbnails |
| `POST /api/thumbnails/regenerate-missing` | 1275-1292 | Generate missing thumbnails |

### Maintenance API

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/reapply-artist-tags` | 1294-1358 | Re-apply artist tags from file paths |
| `POST /api/cleanup-orphaned-tags` | 1360-1394 | Delete tags with no associated images |

### Celery Status

| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/celery/status` | 1396-1430 | Worker and queue status |

### Duplicate Detection

| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/duplicates/scan` | 1467-1554 | Scan for similar images by pHash |
| `POST /api/duplicates/delete` | 1556-1602 | Delete duplicate images |

---

## Frontend Templates (`app/templates/`)

| Template | Purpose |
|----------|---------|
| `layout.html` | Base template with navbar (includes Tags dropdown) |
| `showcase.html` | Random image masonry view (extends layout, includes `_gallery_modal.html`) |
| `gallery.html` | Main gallery with infinite scroll, timeline sidebar, bulk editor (includes `_gallery_modal.html`) |
| `reader.html` | Series reader with vertical scroll, page jump, thumbnail navigation |
| `tags_list.html` | Tag browser with preview images and infinite scroll |
| `settings.html` | Import settings, deletion tools, duplicate detection, maintenance tools |
| `_gallery_item.html` | Single gallery item partial (with tag overlay icons) |
| `_gallery_modal.html` | **Shared** image modal with tag editor and series management (used by gallery & showcase) |
| `_tag_cards.html` | Tag card grid partial for tag list |
| `_pagination.html` | Legacy pagination controls |
| `_pagination_floating.html` | Floating pagination indicator |

**Template Inheritance**:
```
layout.html (base)
├── gallery.html (includes: _gallery_item.html, _gallery_modal.html)
├── showcase.html (includes: _gallery_modal.html)
├── reader.html
├── tags_list.html (includes: _tag_cards.html)
└── settings.html
```

### JavaScript Files (`app/static/js/`)

| File | Purpose |
|------|---------|
| `tag-utils.js` | **Shared utility module** - `getTagIcon()`, `getTagDisplayName()`, `escapeHtml()`, `createTagChipHtml()` used by all tag-related JS |
| `gallery-infinite.js` | Infinite scroll, timeline navigation (uses TagUtils) |
| `view-modal.js` | Modal image navigation, tag editing, series management (uses TagUtils) |
| `bulk-select.js` | Multi-select with ordered selection, bulk tag editing, tag refresh (uses TagUtils) |
| `tag-editor.js` | Tag autocomplete and editing |
| `showcase.js` | Showcase masonry layout and shuffle functionality (uses TagUtils) |

**Script Loading Order**: `tag-utils.js` must be loaded before other JS files that depend on `window.TagUtils`.

---

## Docker Services

| Service | Purpose |
|---------|---------|
| `redis` | Message broker for Celery |
| `postgres` | PostgreSQL database |
| `web` | Flask application (port 5000) |
| `celery-worker` | Task processor (queues: scan, import, thumbnail, sidecar, default) |
| `celery-beat` | Periodic task scheduler |

---

## Key Concepts

### Content-Addressed Storage
Files are stored with hash suffix: `{filename}__{hash[:10]}.{ext}`

### Perceptual Hashing
256-bit pHash for duplicate detection (threshold default: 10)

### Tag Kinds
- `artist` - Artist/creator (prefix: `artist:name`)
- `archive` - Archive source (prefix: `archive:artist/NNNN`)
- `source` - Platform source (prefix: `source:platform`)
- `post` - Post reference (prefix: `post:platform:artist:id`)
- `user` - User-created tags (no prefix)
- `character` - Character tags (prefix: `character:name`)
- `series` - Series for ordered reading (prefix: `series:name`)
- `fandom` - Fandom/franchise tags (prefix: `fandom:name`)
- `rating` - Content rating tags (prefix: `rating:level`)

### Import Pipeline

**Quick Scan (default)** - Optimized for speed:
1. `scan_directory` pre-loads all existing tasks in memory (single query)
2. Walks `/import/{artist}/` folders with O(1) duplicate checks
3. Batches task creation for efficiency
4. `import_media_file` uses cached settings, skips pHash comparison
5. Content hash duplicate detection only
6. Copies new files to `/images/{artist}/` with hash suffix
7. Generates thumbnail and applies sidecar metadata

**Deep Scan (on-demand)** - Full reprocessing:
1. `deep_scan_directory` queues all files (only skips currently active)
2. Full metadata re-extraction
3. pHash calculation and similarity comparison
4. Optional thumbnail regeneration
5. Optional sidecar metadata re-application
6. Content hash verification

Trigger via API: `POST /api/import/trigger` with `deep=true`

### Sidecar Metadata
Gallery-DL JSON files provide:
- Platform (patreon, hentaifoundry, subscribestar, pixiv)
- Artist name
- Post ID and title
- Publication date
- Source URL

Pixiv-specific: Uses `{id}_p{num}.json` naming pattern, extracts from `user.account`, `create_date`, `caption`, constructs URL as `https://www.pixiv.net/artworks/{id}`

---

## Environment Variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `DB_USER` | imagerepo | PostgreSQL user |
| `DB_PASS` | postgres | PostgreSQL password |
| `DB_HOST` | postgres | PostgreSQL host |
| `DB_NAME` | imagerepo | Database name |
| `CELERY_BROKER_URL` | redis://redis:6379/0 | Redis broker |
| `CELERY_WORKER_CONCURRENCY` | 2 | Worker processes |
| `TZ` | America/New_York | Container timezone |
| `IMPORT_EVERY_SECONDS` | 28800 | Auto-scan interval (8h) |
| `IMPORT_MIN_WIDTH` | 0 | Minimum image width |
| `IMPORT_MIN_HEIGHT` | 0 | Minimum image height |
| `IMPORT_SKIP_TRANSPARENT` | false | Skip transparent images |
| `IMPORT_TRANSPARENCY_THRESHOLD` | 0.9 | Transparency % threshold |
| `IMPORT_PHASH_THRESHOLD` | 10 | pHash similarity threshold |

---

## CLI Commands

```bash
flask version-check    # Run data migrations
flask celery-worker    # Start Celery worker
flask celery-beat      # Start Celery Beat scheduler
```