imagerepo/summary.md

ImageRepo - Codebase Summary

A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.

Important

: This summary must be kept up to date with any code changes. Update the timestamp below when making modifications.

Last Updated: 2026-02-11 (Character-fandom tag association system)

---

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Flask Web     │     │  Celery Worker  │     │  Celery Beat    │
│   (port 5000)   │     │  (task queues)  │     │  (scheduler)    │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌─────▼─────┐
        │ PostgreSQL│     │    Redis    │    │  /images  │
        │ (database)│     │   (broker)  │    │ /import   │
        └───────────┘     └─────────────┘    └───────────┘

---

File Structure

Core Application Files

File	Purpose
run.py	Entry point - creates Flask app via create_app()
config.py	Configuration class with DB, Redis, Celery settings
docker-compose.yml	Container orchestration for all services

Application Package (app/)

File	Purpose
app/__init__.py:56-86	Flask app factory create_app(), CLI commands, initializes extensions
app/main.py	Blueprint with all HTTP routes and API endpoints
app/models.py	SQLAlchemy models for database schema
app/celery_app.py	Celery configuration and task routing

---

Database Models (app/models.py)

ImageRecord (lines 19-40)

Primary model for images/videos.

Fields: id, filename, filepath, thumb_path, hash (SHA256), perceptual_hash (256-bit),
        file_size, width, height, format, camera_model, taken_at, imported_at, is_thumbnail
Relationships: tags (many-to-many via image_tags)

Tag (lines 42-53)

Tagging system for organization.

Fields: id, name (unique), kind (artist|archive|user|source|post|character|series|fandom|rating),
        fandom_id (nullable FK to tag.id, self-referential for character→fandom link)
Relationships: images (many-to-many), fandom (self-referential, character tags point to fandom tags)

ArchiveRecord (lines 53-66)

Tracks processed archives to prevent re-importing.

Fields: id, filename, file_size, hash, imported_at, artist, tag_id

AppSettings (lines 69-78)

Key-value store for application configuration.

Fields: id, key, value, updated_at

ImportBatch (lines 81-101)

Groups import tasks for batch tracking.

Fields: id, source_directory, status, total_files, processed_files,
        imported_count, skipped_count, failed_count, started_at, completed_at

PostMetadata (lines 104-133)

Stores metadata about posts from external platforms (Patreon, SubscribeStar, HentaiFoundry).

Fields: id, tag_id (FK to post tag), platform, post_id, artist,
        title, description (may contain HTML), source_url,
        attachment_count, published_at, created_at, updated_at
Relationships: tag (one-to-one with Tag where kind='post')

SeriesPage (lines 136-168)

Links images to series tags with page numbers for ordered reading.

Fields: id, series_tag_id (FK to tag), image_id (FK to image, unique),
        page_number, created_at, updated_at
Constraints: unique(series_tag_id, page_number), unique(image_id)
Relationships: series_tag, image

ImportTask (lines 171-222)

Individual file import task tracking.

Fields: id, source_path, file_hash, file_size, task_type, status, celery_task_id,
        context (JSON), batch_id, result_image_id, error_message, retry_count, timestamps
Status values: pending, queued, processing, complete, failed, skipped
Task types: scan, import_image, import_archive, thumbnail, sidecar

---

Celery Tasks (app/tasks/)

Scan Tasks (app/tasks/scan.py)

Two scan modes are available:

• Quick scan (default): Pre-loads existing task/hash data for O(1) lookups. Only queues truly new files. Skips pHash comparison for speed.
• Deep scan (on-demand): Full reprocessing of all files. Re-extracts metadata, runs pHash comparison, optionally regenerates thumbnails and re-applies sidecars.

Function	Purpose
_build_task_lookup()	Pre-loads all ImportTask records into dict for O(1) checks (single query)
_should_skip_file()	Check if file should be skipped using pre-loaded lookup
scan_directory()	Quick scan - walks /import, uses O(1) lookups, batches task creation
_flush_pending_tasks()	Batch add tasks and commit for efficiency
deep_scan_directory()	Deep scan - full reprocessing, pHash comparison, optional thumbnail regen
recover_interrupted_tasks()	Re-queues stuck tasks (called periodically by Beat)
cleanup_old_tasks()	Deletes failed/skipped tasks older than 7 days (daily)
update_batch_stats()	Updates ImportBatch statistics

Import Tasks (app/tasks/import_file.py)

Import tasks support two modes based on context.deep_scan:

• Quick mode (default): Fast duplicate detection by content hash only. Skips pHash comparison.
• Deep mode: Full reprocessing with pHash comparison, optional thumbnail regen, sidecar re-application.

Archive tag merging: When a duplicate is found (by hash or pHash similarity), if the file came from an archive, the archive tag is added to the existing image. This ensures images are tagged with all archives they appear in.

Function	Purpose
_get_cached_settings()	Cached import settings (5-min TTL) to avoid repeated DB reads
import_media_file()	Main import task - supports quick/deep modes, filters, deduplicates
import_archive()	Extracts archive, queues child import tasks with deep scan context
_fail_task()	Marks task as failed
_skip_task()	Marks task as skipped
_add_artist_tag()	Adds artist:name tag to image
_add_archive_tag()	Adds archive tag to image (for files from archives)
_regenerate_thumbnail()	Regenerate thumbnail for existing image (deep scan)
_supersede_existing()	Replaces smaller image with larger version

Thumbnail Tasks (app/tasks/thumbnail.py)

Function	Line	Purpose
generate_thumbnail_task()	18-69	Generate thumbnail for single image/video
regenerate_all_thumbnails()	72-108	Bulk regenerate all thumbnails
regenerate_missing_thumbnails()	111-153	Generate only missing thumbnails

Sidecar Tasks (app/tasks/sidecar.py)

Function	Line	Purpose
apply_sidecar_metadata()	18-97	Apply Gallery-DL JSON metadata to image
scan_and_match_sidecars()	100-147	Find and match sidecar files to images
reprocess_sidecars_for_tag()	150-181	Re-run sidecar extraction for tagged images

---

Utility Functions (app/utils/)

Image Importer (app/utils/image_importer.py)

Function	Line	Purpose
load_import_settings()	50-88	Load filter settings from DB/file/env
is_mostly_transparent()	91-146	Check if image is mostly transparent
is_mostly_single_color()	149-215	Check if image is solid color
supersede_image()	218-275	Replace smaller image with larger version
extract_archive_resilient()	359-384	Extract archive using unar/7z fallback
build_hashed_dest_path()	391-408	Generate content-addressed file path
generate_thumbnail()	415-500	Create thumbnail for image
generate_video_thumbnail_mirrored()	524-532	Create thumbnail for video
transcode_video_to_mp4()	535-609	Transcode video to H.264 MP4
needs_transcode()	612-615	Check if video needs transcoding
calculate_hash()	618-623	SHA256 content hash
calculate_perceptual_hash()	626-643	256-bit pHash for similarity
find_similar_image()	646-678	Find similar images by pHash
extract_metadata()	691-754	Extract EXIF/video metadata
is_first_volume()	788-823	Check if archive is first volume of multi-part
compute_next_archive_tag_name()	825-890	Generate unique archive:artist/NNNN tag

Metadata Enrichment (app/utils/metadata_enrichment.py)

Function	Line	Purpose
find_sidecar_json()	27-83	Find Gallery-DL sidecar JSON for image
load_sidecar_metadata()	86-93	Parse sidecar JSON file
extract_platform()	100-117	Get platform from metadata
extract_artist()	120-136	Get artist/creator name
extract_post_id()	139-155	Get post ID
extract_post_title()	158-164	Get post title
extract_description()	167-194	Get post description/content (HTML or plain text)
extract_source_url()	197-228	Get URL to original post (constructs for SS/HF)
extract_attachment_count()	231-248	Get number of attachments in post
extract_date()	251-271	Extract publication date
generate_source_tags()	338-369	Generate source:platform and post:... tags
resolve_date_conflict()	376-405	Keep earliest date
enrich_from_sidecar()	412-470	Main enrichment function (returns platform, artist, title, description, source_url, etc.)

Version Migration (app/utils/version_migration.py)

Function	Line	Purpose
get_setting() / set_setting()	43-74	Read/write AppSettings
get_setting_int/float/bool()	77-104	Typed setting getters
get_import_settings()	111-122	Load import filter settings from DB
calculate_phash_256()	129-139	Calculate 256-bit perceptual hash
recompute_all_phashes()	146-218	Batch recompute all pHashes
check_and_run_migrations()	221-268	Run data migrations on version change

---

HTTP Routes (app/main.py)

Page Routes

Route	Line	Purpose
GET /	18-30	Showcase - random images masonry
GET /gallery	80-160	Main gallery with infinite scroll
GET /tags	426-510	Tag explorer (filterable by kind)
GET /artists	512-515	Redirect to tag_list?kind=artist
GET /settings	523-529	Settings page

Gallery API

Route	Line	Purpose
GET /api/random-images	33-77	Random images for showcase shuffle
GET /api/gallery/scroll	223-295	Infinite scroll pagination
GET /api/gallery/timeline	297-352	Year-month groups for sidebar
GET /api/gallery/jump	354-423	Jump to specific year-month
GET /images/<path:filename>	162-220	Serve image files

Tag API

Route	Line	Purpose
GET /api/tags/search	532-579	Tag autocomplete search (supports kind filter)
GET /api/tags/list	535-555	Paginated tag list for infinite scroll
GET /image/<id>/tags	581-585	List tags for image
POST /image/<id>/tags/add	587-615	Add tag to image
POST /image/<id>/tags/remove	617-633	Remove tag from image
GET /api/tag/<id>	635-647	Get tag details
GET /api/post-metadata/<id>	649-680	Get post metadata for a post tag
GET /api/post-metadata/by-tag-name/<name>	682-714	Get post metadata by tag name
POST /api/tag/<id>/update	716-793	Update/merge tag (cascades fandom renames to linked characters)
POST /api/tag/<id>/set-fandom	—	Set/clear character tag's fandom association (updates name suffix)
POST /api/tag/<id>/delete	795-809	Delete tag
GET /api/tag/<id>/image-count	870-880	Count images with tag

Bulk Operations

Route	Line	Purpose
POST /api/delete-by-tag	811-868	Delete all images with tag
POST /api/images/common-tags	1604-1636	Get common tags across images
POST /api/images/bulk-add-tag	1638-1685	Add tag to multiple images
POST /api/images/bulk-remove-tag	1687-1729	Remove tag from multiple images

Series / Reader API

Route	Line	Purpose
GET /read/<tag_id>	1731-1763	Reader view for a series (vertical scroll)
GET /api/series/<tag_id>/pages	1765-1795	Get all pages in a series
POST /api/series/<tag_id>/pages/add	1797-1853	Add single image to series with page number
POST /api/series/<tag_id>/pages/bulk-add	1855-1915	Add multiple images with sequential page numbers
POST /api/series/page/<page_id>/update	1917-1946	Update a page's page number
POST /api/series/page/<page_id>/delete	1948-1958	Remove image from series
GET /api/image/<image_id>/series-info	1960-1976	Check if image is in a series
POST /api/image/<image_id>/add-to-series	1978-2070	Add/update/move image in series

Import Queue API

Route	Purpose
POST /api/import/trigger	Start scan (params: deep=true for deep scan, regenerate_thumbnails, reapply_sidecars, verify_hashes)
GET /api/import/status	Queue status counts and active batch info
GET /api/import/tasks	List tasks with filtering (params: status, task_type, search, limit, offset)
GET /api/import/task/<id>	Task details
POST /api/import/retry-failed	Retry failed tasks
POST /api/import/clear-completed	Clear tasks (params: statuses, older_than_days for age-based cleanup)
GET /api/import/task-stats	Task counts with age breakdown (today, this week, older)
GET /api/import/batches	List import batches

System Stats API

Route	Purpose
GET /api/system/stats	System overview: total images, tags, storage used, pending imports

Settings API

Route	Line	Purpose
GET /api/import-settings	882-897	Get import filter settings
POST /api/import-settings	899-934	Save import filter settings
GET /api/filter-scan	936-1011	Scan images against filters
POST /api/filter-delete	1013-1067	Delete filtered images

Thumbnail API

Route	Line	Purpose
POST /api/thumbnails/regenerate	1255-1273	Regenerate all thumbnails
POST /api/thumbnails/regenerate-missing	1275-1292	Generate missing thumbnails

Maintenance API

Route	Line	Purpose
POST /api/reapply-artist-tags	1294-1358	Re-apply artist tags from file paths
POST /api/cleanup-orphaned-tags	1360-1394	Delete tags with no associated images

Celery Status

Route	Line	Purpose
GET /api/celery/status	1396-1430	Worker and queue status

Duplicate Detection

Route	Line	Purpose
POST /api/duplicates/scan	1467-1554	Scan for similar images by pHash
POST /api/duplicates/delete	1556-1602	Delete duplicate images

---

Frontend Templates (app/templates/)

Template	Purpose
layout.html	Base template with navbar (includes Tags dropdown)
showcase.html	Random image masonry view (extends layout, includes _gallery_modal.html)
gallery.html	Main gallery with infinite scroll, timeline sidebar, bulk editor (includes _gallery_modal.html)
reader.html	Series reader with vertical scroll, page jump, thumbnail navigation
tags_list.html	Tag browser with preview images and infinite scroll
settings.html	Import settings, deletion tools, duplicate detection, maintenance tools
_gallery_item.html	Single gallery item partial (with tag overlay icons)
_gallery_modal.html	Shared image modal with tag editor and series management (used by gallery & showcase)
_tag_cards.html	Tag card grid partial for tag list
_pagination.html	Legacy pagination controls
_pagination_floating.html	Floating pagination indicator

Template Inheritance:

layout.html (base)
├── gallery.html (includes: _gallery_item.html, _gallery_modal.html)
├── showcase.html (includes: _gallery_modal.html)
├── reader.html
├── tags_list.html (includes: _tag_cards.html)
└── settings.html

JavaScript Files (app/static/js/)

File	Purpose
tag-utils.js	Shared utility module - getTagIcon(), getTagDisplayName(), escapeHtml(), createTagChipHtml() used by all tag-related JS
gallery-infinite.js	Infinite scroll, timeline navigation (uses TagUtils)
view-modal.js	Modal image navigation, tag editing, series management (uses TagUtils)
bulk-select.js	Multi-select with ordered selection, bulk tag editing, tag refresh (uses TagUtils)
tag-editor.js	Tag autocomplete and editing (includes fandom selector for character tags)
showcase.js	Showcase masonry layout and shuffle functionality (uses TagUtils)

Script Loading Order: tag-utils.js must be loaded before other JS files that depend on window.TagUtils.

---

Docker Services

Service	Purpose
redis	Message broker for Celery
postgres	PostgreSQL database
web	Flask application (port 5000)
celery-worker	Task processor (queues: scan, import, thumbnail, sidecar, default)
celery-beat	Periodic task scheduler

---

Key Concepts

Content-Addressed Storage

Files are stored with hash suffix: {filename}__{hash[:10]}.{ext}

Perceptual Hashing

256-bit pHash for duplicate detection (threshold default: 10)

Tag Kinds

• artist - Artist/creator (prefix: artist:name)
• archive - Archive source (prefix: archive:artist/NNNN)
• source - Platform source (prefix: source:platform)
• post - Post reference (prefix: post:platform:artist:id)
• user - User-created tags (no prefix)
• character - Character tags (prefix: character:name or character:name (fandom)) — linked to fandom via fandom_id FK; adding a character auto-applies its fandom tag
• series - Series for ordered reading (prefix: series:name)
• fandom - Fandom/franchise tags (prefix: fandom:name) — renaming cascades to all linked character tags
• rating - Content rating tags (prefix: rating:level)

Import Pipeline

Quick Scan (default) - Optimized for speed:

1. scan_directory pre-loads all existing tasks in memory (single query)
2. Walks /import/{artist}/ folders with O(1) duplicate checks
3. Batches task creation for efficiency
4. import_media_file uses cached settings, skips pHash comparison
5. Content hash duplicate detection only
6. Copies new files to /images/{artist}/ with hash suffix
7. Generates thumbnail and applies sidecar metadata

Deep Scan (on-demand) - Full reprocessing:

1. deep_scan_directory queues all files (only skips currently active)
2. Full metadata re-extraction
3. pHash calculation and similarity comparison
4. Optional thumbnail regeneration
5. Optional sidecar metadata re-application
6. Content hash verification

Trigger via API: POST /api/import/trigger with deep=true

Sidecar Metadata

Gallery-DL JSON files provide:

• Platform (patreon, hentaifoundry, subscribestar, pixiv)
• Artist name
• Post ID and title
• Publication date
• Source URL

Pixiv-specific: Uses {id}_p{num}.json naming pattern, extracts from user.account, create_date, caption, constructs URL as https://www.pixiv.net/artworks/{id}

---

Environment Variables

Variable	Default	Purpose
DB_USER	imagerepo	PostgreSQL user
DB_PASS	postgres	PostgreSQL password
DB_HOST	postgres	PostgreSQL host
DB_NAME	imagerepo	Database name
CELERY_BROKER_URL	redis://redis:6379/0	Redis broker
CELERY_WORKER_CONCURRENCY	2	Worker processes
TZ	America/New_York	Container timezone
IMPORT_EVERY_SECONDS	28800	Auto-scan interval (8h)
IMPORT_MIN_WIDTH	0	Minimum image width
IMPORT_MIN_HEIGHT	0	Minimum image height
IMPORT_SKIP_TRANSPARENT	false	Skip transparent images
IMPORT_TRANSPARENCY_THRESHOLD	0.9	Transparency % threshold
IMPORT_PHASH_THRESHOLD	10	pHash similarity threshold

---

CLI Commands

flask version-check    # Run data migrations
flask celery-worker    # Start Celery worker
flask celery-beat      # Start Celery Beat scheduler