Archived

This repository has been archived on 2026-05-31. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

Bryan Van Deusen 9ebeeed133 series management tuning and import archive extraction troubleshooting

2026-01-23 23:05:25 -05:00

17 KiB

Raw Blame History

ImageRepo - Codebase Summary

A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Flask Web     │     │  Celery Worker  │     │  Celery Beat    │
│   (port 5000)   │     │  (task queues)  │     │  (scheduler)    │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌─────▼─────┐
        │ PostgreSQL│     │    Redis    │    │  /images  │
        │ (database)│     │   (broker)  │    │ /import   │
        └───────────┘     └─────────────┘    └───────────┘

File Structure

Core Application Files

File	Purpose
`run.py`	Entry point - creates Flask app via `create_app()`
`config.py`	Configuration class with DB, Redis, Celery settings
`docker-compose.yml`	Container orchestration for all services

Application Package (`app/`)

File	Purpose
`app/__init__.py:56-86`	Flask app factory `create_app()`, CLI commands, initializes extensions
`app/main.py`	Blueprint with all HTTP routes and API endpoints
`app/models.py`	SQLAlchemy models for database schema
`app/celery_app.py`	Celery configuration and task routing

Database Models (`app/models.py`)

ImageRecord (lines 19-40)

Primary model for images/videos.

Fields: id, filename, filepath, thumb_path, hash (SHA256), perceptual_hash (256-bit),
        file_size, width, height, format, camera_model, taken_at, imported_at, is_thumbnail
Relationships: tags (many-to-many via image_tags)

Tag (lines 42-51)

Tagging system for organization.

Fields: id, name (unique), kind (artist|archive|user|source|post|character|series|fandom|rating)
Relationships: images (many-to-many)

ArchiveRecord (lines 53-66)

Tracks processed archives to prevent re-importing.

Fields: id, filename, file_size, hash, imported_at, artist, tag_id

AppSettings (lines 69-78)

Key-value store for application configuration.

Fields: id, key, value, updated_at

ImportBatch (lines 81-101)

Groups import tasks for batch tracking.

Fields: id, source_directory, status, total_files, processed_files,
        imported_count, skipped_count, failed_count, started_at, completed_at

PostMetadata (lines 104-133)

Stores metadata about posts from external platforms (Patreon, SubscribeStar, HentaiFoundry).

Fields: id, tag_id (FK to post tag), platform, post_id, artist,
        title, description (may contain HTML), source_url,
        attachment_count, published_at, created_at, updated_at
Relationships: tag (one-to-one with Tag where kind='post')

SeriesPage (lines 136-168)

Links images to series tags with page numbers for ordered reading.

Fields: id, series_tag_id (FK to tag), image_id (FK to image, unique),
        page_number, created_at, updated_at
Constraints: unique(series_tag_id, page_number), unique(image_id)
Relationships: series_tag, image

ImportTask (lines 171-222)

Individual file import task tracking.

Fields: id, source_path, file_hash, file_size, task_type, status, celery_task_id,
        context (JSON), batch_id, result_image_id, error_message, retry_count, timestamps
Status values: pending, queued, processing, complete, failed, skipped
Task types: scan, import_image, import_archive, thumbnail, sidecar

Celery Tasks (`app/tasks/`)

Scan Tasks (`app/tasks/scan.py`)

Function	Line	Purpose
`scan_directory()`	29-154	Walks `/import` directory, creates ImportTask records, queues import tasks
`_create_task_if_new()`	157-228	Creates ImportTask if file not already processed
`recover_interrupted_tasks()`	231-302	Re-queues stuck tasks (called periodically by Beat)
`update_batch_stats()`	305-349	Updates ImportBatch statistics

Import Tasks (`app/tasks/import_file.py`)

Function	Line	Purpose
`import_media_file()`	23-252	Main import task - filters, deduplicates, copies, creates thumbnail
`import_archive()`	255-407	Extracts archive, queues child import tasks
`_fail_task()`	410-425	Marks task as failed
`_skip_task()`	428-443	Marks task as skipped
`_add_artist_tag()`	446-459	Adds artist:name tag to image
`_supersede_existing()`	472-539	Replaces smaller image with larger version

Thumbnail Tasks (`app/tasks/thumbnail.py`)

Function	Line	Purpose
`generate_thumbnail_task()`	18-69	Generate thumbnail for single image/video
`regenerate_all_thumbnails()`	72-108	Bulk regenerate all thumbnails
`regenerate_missing_thumbnails()`	111-153	Generate only missing thumbnails

Sidecar Tasks (`app/tasks/sidecar.py`)

Function	Line	Purpose
`apply_sidecar_metadata()`	18-97	Apply Gallery-DL JSON metadata to image
`scan_and_match_sidecars()`	100-147	Find and match sidecar files to images
`reprocess_sidecars_for_tag()`	150-181	Re-run sidecar extraction for tagged images

Utility Functions (`app/utils/`)

Image Importer (`app/utils/image_importer.py`)

Function	Line	Purpose
`load_import_settings()`	50-88	Load filter settings from DB/file/env
`is_mostly_transparent()`	91-146	Check if image is mostly transparent
`is_mostly_single_color()`	149-215	Check if image is solid color
`supersede_image()`	218-275	Replace smaller image with larger version
`extract_archive_resilient()`	359-384	Extract archive using unar/7z fallback
`build_hashed_dest_path()`	391-408	Generate content-addressed file path
`generate_thumbnail()`	415-500	Create thumbnail for image
`generate_video_thumbnail_mirrored()`	524-532	Create thumbnail for video
`transcode_video_to_mp4()`	535-609	Transcode video to H.264 MP4
`needs_transcode()`	612-615	Check if video needs transcoding
`calculate_hash()`	618-623	SHA256 content hash
`calculate_perceptual_hash()`	626-643	256-bit pHash for similarity
`find_similar_image()`	646-678	Find similar images by pHash
`extract_metadata()`	691-754	Extract EXIF/video metadata
`is_first_volume()`	788-823	Check if archive is first volume of multi-part
`compute_next_archive_tag_name()`	825-890	Generate unique archive:artist/NNNN tag

Metadata Enrichment (`app/utils/metadata_enrichment.py`)

Function	Line	Purpose
`find_sidecar_json()`	27-83	Find Gallery-DL sidecar JSON for image
`load_sidecar_metadata()`	86-93	Parse sidecar JSON file
`extract_platform()`	100-117	Get platform from metadata
`extract_artist()`	120-136	Get artist/creator name
`extract_post_id()`	139-155	Get post ID
`extract_post_title()`	158-164	Get post title
`extract_description()`	167-194	Get post description/content (HTML or plain text)
`extract_source_url()`	197-228	Get URL to original post (constructs for SS/HF)
`extract_attachment_count()`	231-248	Get number of attachments in post
`extract_date()`	251-271	Extract publication date
`generate_source_tags()`	338-369	Generate source:platform and post:... tags
`resolve_date_conflict()`	376-405	Keep earliest date
`enrich_from_sidecar()`	412-470	Main enrichment function (returns platform, artist, title, description, source_url, etc.)

Version Migration (`app/utils/version_migration.py`)

Function	Line	Purpose
`get_setting()` / `set_setting()`	43-74	Read/write AppSettings
`get_setting_int/float/bool()`	77-104	Typed setting getters
`get_import_settings()`	111-122	Load import filter settings from DB
`calculate_phash_256()`	129-139	Calculate 256-bit perceptual hash
`recompute_all_phashes()`	146-218	Batch recompute all pHashes
`check_and_run_migrations()`	221-268	Run data migrations on version change

HTTP Routes (`app/main.py`)

Page Routes

Route	Line	Purpose
`GET /`	18-30	Showcase - random images masonry
`GET /gallery`	80-160	Main gallery with infinite scroll
`GET /tags`	426-510	Tag explorer (filterable by kind)
`GET /artists`	512-515	Redirect to tag_list?kind=artist
`GET /settings`	523-529	Settings page

Gallery API

Route	Line	Purpose
`GET /api/random-images`	33-77	Random images for showcase shuffle
`GET /api/gallery/scroll`	223-295	Infinite scroll pagination
`GET /api/gallery/timeline`	297-352	Year-month groups for sidebar
`GET /api/gallery/jump`	354-423	Jump to specific year-month
`GET /images/<path:filename>`	162-220	Serve image files

Tag API

Route	Line	Purpose
`GET /api/tags/search`	532-579	Tag autocomplete search (supports `kind` filter)
`GET /image/<id>/tags`	581-585	List tags for image
`POST /image/<id>/tags/add`	587-615	Add tag to image
`POST /image/<id>/tags/remove`	617-633	Remove tag from image
`GET /api/tag/<id>`	635-647	Get tag details
`GET /api/post-metadata/<id>`	649-680	Get post metadata for a post tag
`GET /api/post-metadata/by-tag-name/<name>`	682-714	Get post metadata by tag name
`POST /api/tag/<id>/update`	716-793	Update/merge tag
`POST /api/tag/<id>/delete`	795-809	Delete tag
`GET /api/tag/<id>/image-count`	870-880	Count images with tag

Bulk Operations

Route	Line	Purpose
`POST /api/delete-by-tag`	811-868	Delete all images with tag
`POST /api/images/common-tags`	1604-1636	Get common tags across images
`POST /api/images/bulk-add-tag`	1638-1685	Add tag to multiple images
`POST /api/images/bulk-remove-tag`	1687-1729	Remove tag from multiple images

Series / Reader API

Route	Line	Purpose
`GET /read/<tag_id>`	1731-1763	Reader view for a series (vertical scroll)
`GET /api/series/<tag_id>/pages`	1765-1795	Get all pages in a series
`POST /api/series/<tag_id>/pages/add`	1797-1853	Add single image to series with page number
`POST /api/series/<tag_id>/pages/bulk-add`	1855-1915	Add multiple images with sequential page numbers
`POST /api/series/page/<page_id>/update`	1917-1946	Update a page's page number
`POST /api/series/page/<page_id>/delete`	1948-1958	Remove image from series
`GET /api/image/<image_id>/series-info`	1960-1976	Check if image is in a series
`POST /api/image/<image_id>/add-to-series`	1978-2070	Add/update/move image in series

Import Queue API

Route	Line	Purpose
`POST /api/import/trigger`	1069-1090	Start directory scan
`GET /api/import/status`	1092-1151	Queue status and recent tasks
`GET /api/import/task/<id>`	1153-1191	Task details
`POST /api/import/retry-failed`	1193-1231	Retry failed tasks
`POST /api/import/clear-completed`	1233-1253	Clear completed tasks
`GET /api/import/batches`	1432-1465	List import batches

Settings API

Route	Line	Purpose
`GET /api/import-settings`	882-897	Get import filter settings
`POST /api/import-settings`	899-934	Save import filter settings
`GET /api/filter-scan`	936-1011	Scan images against filters
`POST /api/filter-delete`	1013-1067	Delete filtered images

Thumbnail API

Route	Line	Purpose
`POST /api/thumbnails/regenerate`	1255-1273	Regenerate all thumbnails
`POST /api/thumbnails/regenerate-missing`	1275-1292	Generate missing thumbnails

Maintenance API

Route	Line	Purpose
`POST /api/reapply-artist-tags`	1294-1358	Re-apply artist tags from file paths
`POST /api/cleanup-orphaned-tags`	1360-1394	Delete tags with no associated images

Celery Status

Route	Line	Purpose
`GET /api/celery/status`	1396-1430	Worker and queue status

Duplicate Detection

Route	Line	Purpose
`POST /api/duplicates/scan`	1467-1554	Scan for similar images by pHash
`POST /api/duplicates/delete`	1556-1602	Delete duplicate images

Frontend Templates (`app/templates/`)

Template	Purpose
`layout.html`	Base template with navbar
`showcase.html`	Random image masonry view
`gallery.html`	Main gallery with infinite scroll, timeline sidebar, bulk editor, series editor
`reader.html`	Series reader with vertical scroll, page jump, thumbnail navigation
`tags_list.html`	Tag browser with preview images
`settings.html`	Import settings, deletion tools, duplicate detection, maintenance tools
`_gallery_item.html`	Single gallery item partial (with tag overlay)
`_gallery_modal.html`	Image modal with tag editor and series management
`_gallery_grid.html`	Gallery grid partial
`_pagination.html`	Pagination controls

JavaScript Files (`app/static/js/`)

File	Purpose
`gallery-infinite.js`	Infinite scroll, timeline navigation
`modal-pagination.js`	Modal image navigation, tag editing, series management (add/update/move)
`bulk-select.js`	Multi-select with ordered selection, bulk tag editing, tag refresh, series bulk-add
`tag-editor.js`	Tag autocomplete and editing
`showcase.js`	Showcase shuffle functionality

Docker Services

Service	Purpose
`redis`	Message broker for Celery
`postgres`	PostgreSQL database
`web`	Flask application (port 5000)
`celery-worker`	Task processor (queues: scan, import, thumbnail, sidecar, default)
`celery-beat`	Periodic task scheduler

Key Concepts

Content-Addressed Storage

Files are stored with hash suffix: {filename}__{hash[:10]}.{ext}

Perceptual Hashing

256-bit pHash for duplicate detection (threshold default: 10)

Tag Kinds

artist - Artist/creator (prefix: artist:name)
archive - Archive source (prefix: archive:artist/NNNN)
source - Platform source (prefix: source:platform)
post - Post reference (prefix: post:platform:artist:id)
user - User-created tags (no prefix)
character - Character tags (prefix: character:name)
series - Series for ordered reading (prefix: series:name)
fandom - Fandom/franchise tags (prefix: fandom:name)
rating - Content rating tags (prefix: rating:level)

Import Pipeline

scan_directory walks /import/{artist}/ folders
Creates ImportTask for each file
import_media_file or import_archive processes file
Applies filters (dimensions, transparency, duplicates)
Copies to /images/{artist}/ with hash suffix
Generates thumbnail
Applies sidecar metadata if found
Creates ImageRecord with tags

Sidecar Metadata

Gallery-DL JSON files provide:

Platform (patreon, hentaifoundry, etc.)
Artist name
Post ID and title
Publication date
Source URL

Environment Variables

Variable	Default	Purpose
`DB_USER`	imagerepo	PostgreSQL user
`DB_PASS`	postgres	PostgreSQL password
`DB_HOST`	postgres	PostgreSQL host
`DB_NAME`	imagerepo	Database name
`CELERY_BROKER_URL`	redis://redis:6379/0	Redis broker
`CELERY_WORKER_CONCURRENCY`	2	Worker processes
`IMPORT_EVERY_SECONDS`	28800	Auto-scan interval (8h)
`IMPORT_MIN_WIDTH`	0	Minimum image width
`IMPORT_MIN_HEIGHT`	0	Minimum image height
`IMPORT_SKIP_TRANSPARENT`	false	Skip transparent images
`IMPORT_TRANSPARENCY_THRESHOLD`	0.9	Transparency % threshold
`IMPORT_PHASH_THRESHOLD`	10	pHash similarity threshold

CLI Commands

flask version-check    # Run data migrations
flask celery-worker    # Start Celery worker
flask celery-beat      # Start Celery Beat scheduler

17 KiB Raw Blame History