bvandeusen/imagerepo
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Optimize scan process with quick/deep modes and archive tag merging

Browse Source 
Quick scan (default):
- Pre-load all ImportTask records for O(1) duplicate checks
- Batch task creation commits (50 at a time)
- Cache import settings (5-min TTL)
- Skip pHash comparison for speed

Deep scan (on-demand):
- Full reprocessing of all files
- pHash similarity detection
- Optional thumbnail regeneration
- Optional sidecar re-application

Archive tag merging:
- Add archive tags to existing images when duplicates found
- Works for both hash and pHash duplicate detection

Also fixes:
- Add missing source/post tag icons to gallery templates
This commit is contained in:
Bryan Van Deusen
2026-01-31 15:18:43 -05:00
parent d0fcde38e8
commit 042a69f9c3
7 changed files with 595 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files
summary.md
+64 -30
View File
@@ -2,6 +2,10 @@
A Flask-based image repository with Celery task processing for importing, organizing, and managing images/videos with tagging, duplicate detection, and Gallery-DL metadata integration.
> **IMPORTANT**: This summary must be kept up to date with any code changes. Update the timestamp below when making modifications.
>
> **Last Updated**: 2026-01-31 (Scan optimization: quick/deep scan modes, archive tag merging)
---
## Architecture Overview
@@ -114,23 +118,40 @@ Task types: scan, import_image, import_archive, thumbnail, sidecar
### Scan Tasks (`app/tasks/scan.py`)
| Function | Line | Purpose |
|----------|------|---------|
| `scan_directory()` | 29-154 | Walks `/import` directory, creates ImportTask records, queues import tasks |
| `_create_task_if_new()` | 157-228 | Creates ImportTask if file not already processed |
| `recover_interrupted_tasks()` | 231-302 | Re-queues stuck tasks (called periodically by Beat) |
| `update_batch_stats()` | 305-349 | Updates ImportBatch statistics |
Two scan modes are available:
- **Quick scan** (default): Pre-loads existing task/hash data for O(1) lookups. Only queues truly new files. Skips pHash comparison for speed.
- **Deep scan** (on-demand): Full reprocessing of all files. Re-extracts metadata, runs pHash comparison, optionally regenerates thumbnails and re-applies sidecars.
| Function | Purpose |
|----------|---------|
| `_build_task_lookup()` | Pre-loads all ImportTask records into dict for O(1) checks (single query) |
| `_should_skip_file()` | Check if file should be skipped using pre-loaded lookup |
| `scan_directory()` | Quick scan - walks `/import`, uses O(1) lookups, batches task creation |
| `_flush_pending_tasks()` | Batch add tasks and commit for efficiency |
| `deep_scan_directory()` | Deep scan - full reprocessing, pHash comparison, optional thumbnail regen |
| `recover_interrupted_tasks()` | Re-queues stuck tasks (called periodically by Beat) |
| `cleanup_old_tasks()` | Deletes failed/skipped tasks older than 7 days (daily) |
| `update_batch_stats()` | Updates ImportBatch statistics |
### Import Tasks (`app/tasks/import_file.py`)
| Function | Line | Purpose |
|----------|------|---------|
| `import_media_file()` | 23-252 | Main import task - filters, deduplicates, copies, creates thumbnail |
| `import_archive()` | 255-407 | Extracts archive, queues child import tasks |
| `_fail_task()` | 410-425 | Marks task as failed |
| `_skip_task()` | 428-443 | Marks task as skipped |
| `_add_artist_tag()` | 446-459 | Adds artist:name tag to image |
| `_supersede_existing()` | 472-539 | Replaces smaller image with larger version |
Import tasks support two modes based on `context.deep_scan`:
- **Quick mode** (default): Fast duplicate detection by content hash only. Skips pHash comparison.
- **Deep mode**: Full reprocessing with pHash comparison, optional thumbnail regen, sidecar re-application.
**Archive tag merging**: When a duplicate is found (by hash or pHash similarity), if the file came from an archive, the archive tag is added to the existing image. This ensures images are tagged with all archives they appear in.
| Function | Purpose |
|----------|---------|
| `_get_cached_settings()` | Cached import settings (5-min TTL) to avoid repeated DB reads |
| `import_media_file()` | Main import task - supports quick/deep modes, filters, deduplicates |
| `import_archive()` | Extracts archive, queues child import tasks with deep scan context |
| `_fail_task()` | Marks task as failed |
| `_skip_task()` | Marks task as skipped |
| `_add_artist_tag()` | Adds artist:name tag to image |
| `_add_archive_tag()` | Adds archive tag to image (for files from archives) |
| `_regenerate_thumbnail()` | Regenerate thumbnail for existing image (deep scan) |
| `_supersede_existing()` | Replaces smaller image with larger version |
### Thumbnail Tasks (`app/tasks/thumbnail.py`)
@@ -231,6 +252,7 @@ Task types: scan, import_image, import_archive, thumbnail, sidecar
| Route | Line | Purpose |
|-------|------|---------|
| `GET /api/tags/search` | 532-579 | Tag autocomplete search (supports `kind` filter) |
| `GET /api/tags/list` | 535-555 | Paginated tag list for infinite scroll |
| `GET /image/<id>/tags` | 581-585 | List tags for image |
| `POST /image/<id>/tags/add` | 587-615 | Add tag to image |
| `POST /image/<id>/tags/remove` | 617-633 | Remove tag from image |
@@ -265,14 +287,14 @@ Task types: scan, import_image, import_archive, thumbnail, sidecar
### Import Queue API
| Route | Line | Purpose |
|-------|------|---------|
| `POST /api/import/trigger` | 1069-1090 | Start directory scan |
| `GET /api/import/status` | 1092-1151 | Queue status and recent tasks |
| `GET /api/import/task/<id>` | 1153-1191 | Task details |
| `POST /api/import/retry-failed` | 1193-1231 | Retry failed tasks |
| `POST /api/import/clear-completed` | 1233-1253 | Clear completed tasks |
| `GET /api/import/batches` | 1432-1465 | List import batches |
| Route | Purpose |
|-------|---------|
| `POST /api/import/trigger` | Start scan (params: `deep=true` for deep scan, `regenerate_thumbnails`, `reapply_sidecars`, `verify_hashes`) |
| `GET /api/import/status` | Queue status and recent tasks |
| `GET /api/import/task/<id>` | Task details |
| `POST /api/import/retry-failed` | Retry failed tasks |
| `POST /api/import/clear-completed` | Clear completed tasks |
| `GET /api/import/batches` | List import batches |
### Settings API
@@ -370,14 +392,25 @@ Files are stored with hash suffix: `{filename}__{hash[:10]}.{ext}`
- `rating` - Content rating tags (prefix: `rating:level`)
### Import Pipeline
1. `scan_directory` walks `/import/{artist}/` folders
2. Creates ImportTask for each file
3. `import_media_file` or `import_archive` processes file
4. Applies filters (dimensions, transparency, duplicates)
5. Copies to `/images/{artist}/` with hash suffix
6. Generates thumbnail
7. Applies sidecar metadata if found
8. Creates ImageRecord with tags
**Quick Scan (default)** - Optimized for speed:
1. `scan_directory` pre-loads all existing tasks in memory (single query)
2. Walks `/import/{artist}/` folders with O(1) duplicate checks
3. Batches task creation for efficiency
4. `import_media_file` uses cached settings, skips pHash comparison
5. Content hash duplicate detection only
6. Copies new files to `/images/{artist}/` with hash suffix
7. Generates thumbnail and applies sidecar metadata
**Deep Scan (on-demand)** - Full reprocessing:
1. `deep_scan_directory` queues all files (only skips currently active)
2. Full metadata re-extraction
3. pHash calculation and similarity comparison
4. Optional thumbnail regeneration
5. Optional sidecar metadata re-application
6. Content hash verification
Trigger via API: `POST /api/import/trigger` with `deep=true`
### Sidecar Metadata
Gallery-DL JSON files provide:
@@ -399,6 +432,7 @@ Gallery-DL JSON files provide:
| `DB_NAME` | imagerepo | Database name |
| `CELERY_BROKER_URL` | redis://redis:6379/0 | Redis broker |
| `CELERY_WORKER_CONCURRENCY` | 2 | Worker processes |
| `TZ` | America/New_York | Container timezone |
| `IMPORT_EVERY_SECONDS` | 28800 | Auto-scan interval (8h) |
| `IMPORT_MIN_WIDTH` | 0 | Minimum image width |
| `IMPORT_MIN_HEIGHT` | 0 | Minimum image height |