feat(integrity): structural verification + supersede-on-replace pipeline

Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.

Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries

Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints

Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
  paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
  the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
  verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
  a flagged-corrupt record routes through _supersede_existing,
  preserving tags/series/embeddings

Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
  get_suggestions all filter integrity_status IN ('ok', 'unknown') so
  flagged rows don't poison the gallery, ML, or suggestion math.
  'unknown' is treated as healthy so post-migration data stays visible
  until the sweep runs.

UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-04-26 00:16:06 -04:00
parent 8af9f12544
commit ce560d09a1
11 changed files with 499 additions and 4 deletions
+24
View File
@@ -470,6 +470,30 @@
</section>
<section class="settings-section">
<h3>Verify image integrity</h3>
<p class="settings-hint">
Walks every image and runs a structural check (PIL for images,
ffprobe for video, zipfile for archives). Flagged rows are excluded
from random/showcase, ML retag, and the suggestion modal until a
fresh copy lands in <code>/import/&lt;artist&gt;/&lt;filename&gt;</code>
— at which point the import pipeline supersedes the corrupt row,
preserving its tags. The default mode only checks rows whose status
is still <code>unknown</code> (post-migration default + freshly
imported); use "force re-verify all" after disk events to recheck
everything. Flagged rows are listed at
<code>/api/integrity/failed</code>.
</p>
<form action="{{ url_for('main.trigger_verify_unverified_images') }}" method="post" style="display:inline-block;">
<button type="submit" class="btn secondary-btn">Verify unknown</button>
</form>
<form action="{{ url_for('main.trigger_verify_unverified_images') }}" method="post" style="display:inline-block; margin-left:0.5rem;"
onsubmit="return confirm('Re-verify every image in the library? Cheap per-file but enqueues one task per image.');">
<input type="hidden" name="mode" value="all">
<button type="submit" class="btn secondary-btn">Force re-verify all</button>
</form>
</section>
<section class="settings-section">
<h3>Auto-accept threshold (general tags)</h3>
<p class="settings-hint">