feat(integrity): structural verification + supersede-on-replace pipeline
Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.
Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries
Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints
Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
a flagged-corrupt record routes through _supersede_existing,
preserving tags/series/embeddings
Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
get_suggestions all filter integrity_status IN ('ok', 'unknown') so
flagged rows don't poison the gallery, ML, or suggestion math.
'unknown' is treated as healthy so post-migration data stays visible
until the sweep runs.
UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -100,6 +100,15 @@ def tag_and_embed(self, image_id: int):
|
||||
log.warning(f"tag_and_embed: file missing at {image.filepath}")
|
||||
return {'status': 'file_missing'}
|
||||
|
||||
# Skip flagged-corrupt rows. The verifier already saw the file fail
|
||||
# structurally; running inference would either crash or produce noise.
|
||||
# 'unknown' (pre-sweep / mid-import) still proceeds.
|
||||
if image.integrity_status not in ('ok', 'unknown'):
|
||||
log.info(
|
||||
f"tag_and_embed: skipping image {image_id} — integrity={image.integrity_status}"
|
||||
)
|
||||
return {'status': 'skipped_integrity', 'integrity': image.integrity_status}
|
||||
|
||||
is_video = image.filepath.lower().endswith(VIDEO_EXTS)
|
||||
|
||||
try:
|
||||
@@ -187,6 +196,7 @@ def backfill(self, batch_size: int = 50, pause_seconds: float = 0.5):
|
||||
),
|
||||
)
|
||||
.filter(ImageRecord.id > last_id)
|
||||
.filter(ImageRecord.integrity_status.in_(('ok', 'unknown')))
|
||||
.filter(
|
||||
(ImageTagPrediction.image_id.is_(None))
|
||||
| (ImageEmbedding.image_id.is_(None))
|
||||
|
||||
Reference in New Issue
Block a user