bvandeusen/imagerepo
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-05-31. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
007592827c74245332e019a6c58e1d9fd8497381
imagerepo/summary.md
T
bvandeusen d05bbd983c docs: update summary.md for UI/UX improvements feature 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-20 13:43:03 -04:00

21 KiB
Raw Blame History

ImageRepo - Codebase Summary

A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.

Important

: This summary must be kept up to date with any code changes. Update the timestamp below when making modifications.

Last Updated: 2026-03-20 (Content-first UI/UX improvements)

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Flask Web     │     │  Celery Worker  │     │  Celery Beat    │
│   (port 5000)   │     │  (task queues)  │     │  (scheduler)    │
└────────┬────────┘     └────────┬────────┘     └────────┬────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌─────▼─────┐
        │ PostgreSQL│     │    Redis    │    │  /images  │
        │ (database)│     │   (broker)  │    │ /import   │
        └───────────┘     └─────────────┘    └───────────┘

File Structure

Core Application Files

File Purpose
run.py Entry point - creates Flask app via create_app()
config.py Configuration class with DB, Redis, Celery settings
docker-compose.yml Container orchestration for all services

Application Package (app/)

File Purpose
app/__init__.py:56-86 Flask app factory create_app(), CLI commands, initializes extensions
app/main.py Blueprint with all HTTP routes and API endpoints
app/models.py SQLAlchemy models for database schema
app/celery_app.py Celery configuration and task routing

Database Models (app/models.py)

ImageRecord (lines 19-40)

Primary model for images/videos.

Fields: id, filename, filepath, thumb_path, hash (SHA256), perceptual_hash (256-bit),
        file_size, width, height, format, camera_model, taken_at, imported_at, is_thumbnail
Relationships: tags (many-to-many via image_tags)

Tag (lines 42-53)

Tagging system for organization.

Fields: id, name (unique), kind (artist|archive|user|source|post|character|series|fandom|rating),
        fandom_id (nullable FK to tag.id, self-referential for character→fandom link)
Relationships: images (many-to-many), fandom (self-referential, character tags point to fandom tags)

ArchiveRecord (lines 53-66)

Tracks processed archives to prevent re-importing.

Fields: id, filename, file_size, hash, imported_at, artist, tag_id

AppSettings (lines 69-78)

Key-value store for application configuration.

Fields: id, key, value, updated_at

ImportBatch (lines 81-101)

Groups import tasks for batch tracking.

Fields: id, source_directory, status, total_files, processed_files,
        imported_count, skipped_count, failed_count, started_at, completed_at

PostMetadata (lines 104-133)

Stores metadata about posts from external platforms (Patreon, SubscribeStar, HentaiFoundry).

Fields: id, tag_id (FK to post tag), platform, post_id, artist,
        title, description (may contain HTML), source_url,
        attachment_count, published_at, created_at, updated_at
Relationships: tag (one-to-one with Tag where kind='post')

SeriesPage (lines 136-168)

Links images to series tags with page numbers for ordered reading.

Fields: id, series_tag_id (FK to tag), image_id (FK to image, unique),
        page_number, created_at, updated_at
Constraints: unique(series_tag_id, page_number), unique(image_id)
Relationships: series_tag, image

ImportTask (lines 171-222)

Individual file import task tracking.

Fields: id, source_path, file_hash, file_size, task_type, status, celery_task_id,
        context (JSON), batch_id, result_image_id, error_message, retry_count, timestamps
Status values: pending, queued, processing, complete, failed, skipped
Task types: scan, import_image, import_archive, thumbnail, sidecar

Celery Tasks (app/tasks/)

Scan Tasks (app/tasks/scan.py)

Two scan modes are available:

  • Quick scan (default): Pre-loads existing task/hash data for O(1) lookups. Only queues truly new files. Skips pHash comparison for speed.
  • Deep scan (on-demand): Full reprocessing of all files. Re-extracts metadata, runs pHash comparison, optionally regenerates thumbnails and re-applies sidecars.
Function Purpose
_build_task_lookup() Pre-loads all ImportTask records into dict for O(1) checks (single query)
_should_skip_file() Check if file should be skipped using pre-loaded lookup
scan_directory() Quick scan - walks /import, uses O(1) lookups, batches task creation
_flush_pending_tasks() Batch add tasks and commit for efficiency
deep_scan_directory() Deep scan - full reprocessing, pHash comparison, optional thumbnail regen
recover_interrupted_tasks() Re-queues stuck tasks (called periodically by Beat)
cleanup_old_tasks() Deletes failed/skipped tasks older than 7 days (daily)
update_batch_stats() Updates ImportBatch statistics

Import Tasks (app/tasks/import_file.py)

Import tasks support two modes based on context.deep_scan:

  • Quick mode (default): Fast duplicate detection by content hash only. Skips pHash comparison.
  • Deep mode: Full reprocessing with pHash comparison, optional thumbnail regen, sidecar re-application.

Archive tag merging: When a duplicate is found (by hash or pHash similarity), if the file came from an archive, the archive tag is added to the existing image. This ensures images are tagged with all archives they appear in.

Function Purpose
_get_cached_settings() Cached import settings (5-min TTL) to avoid repeated DB reads
import_media_file() Main import task - supports quick/deep modes, filters, deduplicates
import_archive() Extracts archive, queues child import tasks with deep scan context
_fail_task() Marks task as failed
_skip_task() Marks task as skipped
_add_artist_tag() Adds artist:name tag to image
_add_archive_tag() Adds archive tag to image (for files from archives)
_regenerate_thumbnail() Regenerate thumbnail for existing image (deep scan)
_supersede_existing() Replaces smaller image with larger version

Thumbnail Tasks (app/tasks/thumbnail.py)

Function Line Purpose
generate_thumbnail_task() 18-69 Generate thumbnail for single image/video
regenerate_all_thumbnails() 72-108 Bulk regenerate all thumbnails
regenerate_missing_thumbnails() 111-153 Generate only missing thumbnails

Sidecar Tasks (app/tasks/sidecar.py)

Function Line Purpose
apply_sidecar_metadata() 18-97 Apply Gallery-DL JSON metadata to image
scan_and_match_sidecars() 100-147 Find and match sidecar files to images
reprocess_sidecars_for_tag() 150-181 Re-run sidecar extraction for tagged images

Utility Functions (app/utils/)

Image Importer (app/utils/image_importer.py)

Function Line Purpose
load_import_settings() 50-88 Load filter settings from DB/file/env
is_mostly_transparent() 91-146 Check if image is mostly transparent
is_mostly_single_color() 149-215 Check if image is solid color
supersede_image() 218-275 Replace smaller image with larger version
extract_archive_resilient() 359-384 Extract archive using unar/7z fallback
build_hashed_dest_path() 391-408 Generate content-addressed file path
generate_thumbnail() 415-500 Create thumbnail for image
generate_video_thumbnail_mirrored() 524-532 Create thumbnail for video
transcode_video_to_mp4() 535-609 Transcode video to H.264 MP4
needs_transcode() 612-615 Check if video needs transcoding
calculate_hash() 618-623 SHA256 content hash
calculate_perceptual_hash() 626-643 256-bit pHash for similarity
find_similar_image() 646-678 Find similar images by pHash
extract_metadata() 691-754 Extract EXIF/video metadata
is_first_volume() 788-823 Check if archive is first volume of multi-part
compute_next_archive_tag_name() 825-890 Generate unique archive:artist/NNNN tag

Metadata Enrichment (app/utils/metadata_enrichment.py)

Function Line Purpose
find_sidecar_json() 27-83 Find Gallery-DL sidecar JSON for image
load_sidecar_metadata() 86-93 Parse sidecar JSON file
extract_platform() 100-117 Get platform from metadata
extract_artist() 120-136 Get artist/creator name
extract_post_id() 139-155 Get post ID
extract_post_title() 158-164 Get post title
extract_description() 167-194 Get post description/content (HTML or plain text)
extract_source_url() 197-228 Get URL to original post (constructs for SS/HF)
extract_attachment_count() 231-248 Get number of attachments in post
extract_date() 251-271 Extract publication date
generate_source_tags() 338-369 Generate source:platform and post:... tags
resolve_date_conflict() 376-405 Keep earliest date
enrich_from_sidecar() 412-470 Main enrichment function (returns platform, artist, title, description, source_url, etc.)

Version Migration (app/utils/version_migration.py)

Function Line Purpose
get_setting() / set_setting() 43-74 Read/write AppSettings
get_setting_int/float/bool() 77-104 Typed setting getters
get_import_settings() 111-122 Load import filter settings from DB
calculate_phash_256() 129-139 Calculate 256-bit perceptual hash
recompute_all_phashes() 146-218 Batch recompute all pHashes
check_and_run_migrations() 221-268 Run data migrations on version change

HTTP Routes (app/main.py)

Page Routes

Route Line Purpose
GET / 18-30 Showcase - random images masonry
GET /gallery 80-160 Main gallery with infinite scroll
GET /tags 426-510 Tag explorer (filterable by kind)
GET /artists 512-515 Redirect to tag_list?kind=artist
GET /settings 523-529 Settings page

Gallery API

Route Line Purpose
GET /api/random-images 33-77 Random images for showcase shuffle
GET /api/gallery/scroll 223-295 Infinite scroll pagination
GET /api/gallery/timeline 297-352 Year-month groups for sidebar
GET /api/gallery/jump 354-423 Jump to specific year-month
GET /images/<path:filename> 162-220 Serve image files

Tag API

Route Line Purpose
GET /api/tags/search 532-579 Tag autocomplete search (supports kind filter)
GET /api/tags/list 535-555 Paginated tag list for infinite scroll
GET /image/<id>/tags 581-585 List tags for image
POST /image/<id>/tags/add 587-615 Add tag to image
POST /image/<id>/tags/remove 617-633 Remove tag from image
GET /api/tag/<id> 635-647 Get tag details
GET /api/post-metadata/<id> 649-680 Get post metadata for a post tag
GET /api/post-metadata/by-tag-name/<name> 682-714 Get post metadata by tag name
POST /api/tag/<id>/update 716-793 Update/merge tag (cascades fandom renames to linked characters)
POST /api/tag/<id>/set-fandom — Set/clear character tag's fandom association (updates name suffix)
POST /api/tag/<id>/delete 795-809 Delete tag
GET /api/tag/<id>/image-count 870-880 Count images with tag

Bulk Operations

Route Line Purpose
POST /api/delete-by-tag 811-868 Delete all images with tag
POST /api/images/common-tags 1604-1636 Get common tags across images
POST /api/images/bulk-add-tag 1638-1685 Add tag to multiple images
POST /api/images/bulk-remove-tag 1687-1729 Remove tag from multiple images

Series / Reader API

Route Line Purpose
GET /read/<tag_id> 1731-1763 Reader view for a series (vertical scroll)
GET /api/series/<tag_id>/pages 1765-1795 Get all pages in a series
POST /api/series/<tag_id>/pages/add 1797-1853 Add single image to series with page number
POST /api/series/<tag_id>/pages/bulk-add 1855-1915 Add multiple images with sequential page numbers
POST /api/series/page/<page_id>/update 1917-1946 Update a page's page number
POST /api/series/page/<page_id>/delete 1948-1958 Remove image from series
GET /api/image/<image_id>/series-info 1960-1976 Check if image is in a series
POST /api/image/<image_id>/add-to-series 1978-2070 Add/update/move image in series

Import Queue API

Route Purpose
POST /api/import/trigger Start scan (params: deep=true for deep scan, regenerate_thumbnails, reapply_sidecars, verify_hashes)
GET /api/import/status Queue status counts and active batch info
GET /api/import/tasks List tasks with filtering (params: status, task_type, search, limit, offset)
GET /api/import/task/<id> Task details
POST /api/import/retry-failed Retry failed tasks
POST /api/import/clear-completed Clear tasks (params: statuses, older_than_days for age-based cleanup)
GET /api/import/task-stats Task counts with age breakdown (today, this week, older)
GET /api/import/batches List import batches

System Stats API

Route Purpose
GET /api/system/stats System overview: total images, tags, storage used, pending imports

Settings API

Route Line Purpose
GET /api/import-settings 882-897 Get import filter settings
POST /api/import-settings 899-934 Save import filter settings
GET /api/filter-scan 936-1011 Scan images against filters
POST /api/filter-delete 1013-1067 Delete filtered images

Thumbnail API

Route Line Purpose
POST /api/thumbnails/regenerate 1255-1273 Regenerate all thumbnails
POST /api/thumbnails/regenerate-missing 1275-1292 Generate missing thumbnails

Maintenance API

Route Line Purpose
POST /api/reapply-artist-tags 1294-1358 Re-apply artist tags from file paths
POST /api/cleanup-orphaned-tags 1360-1394 Delete tags with no associated images

Celery Status

Route Line Purpose
GET /api/celery/status 1396-1430 Worker and queue status

Duplicate Detection

Route Line Purpose
POST /api/duplicates/scan 1467-1554 Scan for similar images by pHash
POST /api/duplicates/delete 1556-1602 Delete duplicate images

Frontend Templates (app/templates/)

Template Purpose
layout.html Base template with navbar (includes Tags dropdown)
showcase.html Random image masonry view (extends layout, includes _gallery_modal.html)
gallery.html Main gallery with infinite scroll, timeline sidebar, bulk editor (includes _gallery_modal.html)
reader.html Series reader with vertical scroll, progress bar, auto-hiding header, active thumbnail tracking; bridges data to reader.js via window.readerConfig
tags_list.html Tag browser with preview images and infinite scroll
settings.html Three-tab layout (Overview / Import / Maintenance) — import settings, deletion tools, duplicate detection, maintenance tools
_gallery_item.html Single gallery item partial (with tag overlay icons)
_gallery_modal.html Shared image modal with tag editor and series management (used by gallery & showcase)
_tag_cards.html Tag card grid partial for tag list
_pagination.html Legacy pagination controls
_pagination_floating.html Floating pagination indicator

Template Inheritance:

layout.html (base)
├── gallery.html (includes: _gallery_item.html, _gallery_modal.html)
├── showcase.html (includes: _gallery_modal.html)
├── reader.html
├── tags_list.html (includes: _tag_cards.html)
└── settings.html

JavaScript Files (app/static/js/)

File Purpose
tag-utils.js Shared utility module - getTagIcon(), getTagDisplayName(), escapeHtml(), createTagChipHtml() used by all tag-related JS
gallery-infinite.js Infinite scroll, timeline navigation (uses TagUtils)
view-modal.js Modal image navigation, tag editing, series management; in-context showFeedback() pattern, "Add to Series" toggle with lazy-load (uses TagUtils)
bulk-select.js Multi-select with ordered selection, bulk tag editing, tag refresh (uses TagUtils)
tag-editor.js Tag autocomplete and editing (includes fandom selector for character tags)
showcase.js Showcase masonry layout, shuffle, stagger animation, ghost #N tag count badge (uses TagUtils)
reader.js Series reader logic — progress bar, auto-hiding header, active thumbnail tracking via scroll-position heuristic, floating page indicator, keyboard shortcuts

Script Loading Order: tag-utils.js must be loaded before other JS files that depend on window.TagUtils.

Docker Services

Service Purpose
redis Message broker for Celery
postgres PostgreSQL database
web Flask application (port 5000)
celery-worker Task processor (queues: scan, import, thumbnail, sidecar, default)
celery-beat Periodic task scheduler

Key Concepts

Content-Addressed Storage

Files are stored with hash suffix: {filename}__{hash[:10]}.{ext}

Perceptual Hashing

256-bit pHash for duplicate detection (threshold default: 10)

Tag Kinds

  • artist - Artist/creator (prefix: artist:name)
  • archive - Archive source (prefix: archive:artist/NNNN)
  • source - Platform source (prefix: source:platform)
  • post - Post reference (prefix: post:platform:artist:id)
  • user - User-created tags (no prefix)
  • character - Character tags (prefix: character:name or character:name (fandom)) — linked to fandom via fandom_id FK; adding a character auto-applies its fandom tag
  • series - Series for ordered reading (prefix: series:name)
  • fandom - Fandom/franchise tags (prefix: fandom:name) — renaming cascades to all linked character tags
  • rating - Content rating tags (prefix: rating:level)

Import Pipeline

Quick Scan (default) - Optimized for speed:

  1. scan_directory pre-loads all existing tasks in memory (single query)
  2. Walks /import/{artist}/ folders with O(1) duplicate checks
  3. Batches task creation for efficiency
  4. import_media_file uses cached settings, skips pHash comparison
  5. Content hash duplicate detection only
  6. Copies new files to /images/{artist}/ with hash suffix
  7. Generates thumbnail and applies sidecar metadata

Deep Scan (on-demand) - Full reprocessing:

  1. deep_scan_directory queues all files (only skips currently active)
  2. Full metadata re-extraction
  3. pHash calculation and similarity comparison
  4. Optional thumbnail regeneration
  5. Optional sidecar metadata re-application
  6. Content hash verification

Trigger via API: POST /api/import/trigger with deep=true

Sidecar Metadata

Gallery-DL JSON files provide:

  • Platform (patreon, hentaifoundry, subscribestar, pixiv)
  • Artist name
  • Post ID and title
  • Publication date
  • Source URL

Pixiv-specific: Uses {id}_p{num}.json naming pattern, extracts from user.account, create_date, caption, constructs URL as https://www.pixiv.net/artworks/{id}

Environment Variables

Variable Default Purpose
DB_USER imagerepo PostgreSQL user
DB_PASS postgres PostgreSQL password
DB_HOST postgres PostgreSQL host
DB_NAME imagerepo Database name
CELERY_BROKER_URL redis://redis:6379/0 Redis broker
CELERY_WORKER_CONCURRENCY 2 Worker processes
TZ America/New_York Container timezone
IMPORT_EVERY_SECONDS 28800 Auto-scan interval (8h)
IMPORT_MIN_WIDTH 0 Minimum image width
IMPORT_MIN_HEIGHT 0 Minimum image height
IMPORT_SKIP_TRANSPARENT false Skip transparent images
IMPORT_TRANSPARENCY_THRESHOLD 0.9 Transparency % threshold
IMPORT_PHASH_THRESHOLD 10 pHash similarity threshold

CLI Commands

flask version-check    # Run data migrations
flask celery-worker    # Start Celery worker
flask celery-beat      # Start Celery Beat scheduler
Reference in New Issue View Git Blame Copy Permalink