feat(integrity): structural verification + supersede-on-replace pipeline
Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.
Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries
Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints
Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
a flagged-corrupt record routes through _supersede_existing,
preserving tags/series/embeddings
Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
get_suggestions all filter integrity_status IN ('ok', 'unknown') so
flagged rows don't poison the gallery, ML, or suggestion math.
'unknown' is treated as healthy so post-migration data stays visible
until the sweep runs.
UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -470,6 +470,30 @@
|
||||
|
||||
</section>
|
||||
|
||||
<section class="settings-section">
|
||||
<h3>Verify image integrity</h3>
|
||||
<p class="settings-hint">
|
||||
Walks every image and runs a structural check (PIL for images,
|
||||
ffprobe for video, zipfile for archives). Flagged rows are excluded
|
||||
from random/showcase, ML retag, and the suggestion modal until a
|
||||
fresh copy lands in <code>/import/<artist>/<filename></code>
|
||||
— at which point the import pipeline supersedes the corrupt row,
|
||||
preserving its tags. The default mode only checks rows whose status
|
||||
is still <code>unknown</code> (post-migration default + freshly
|
||||
imported); use "force re-verify all" after disk events to recheck
|
||||
everything. Flagged rows are listed at
|
||||
<code>/api/integrity/failed</code>.
|
||||
</p>
|
||||
<form action="{{ url_for('main.trigger_verify_unverified_images') }}" method="post" style="display:inline-block;">
|
||||
<button type="submit" class="btn secondary-btn">Verify unknown</button>
|
||||
</form>
|
||||
<form action="{{ url_for('main.trigger_verify_unverified_images') }}" method="post" style="display:inline-block; margin-left:0.5rem;"
|
||||
onsubmit="return confirm('Re-verify every image in the library? Cheap per-file but enqueues one task per image.');">
|
||||
<input type="hidden" name="mode" value="all">
|
||||
<button type="submit" class="btn secondary-btn">Force re-verify all</button>
|
||||
</form>
|
||||
</section>
|
||||
|
||||
<section class="settings-section">
|
||||
<h3>Auto-accept threshold (general tags)</h3>
|
||||
<p class="settings-hint">
|
||||
|
||||
Reference in New Issue
Block a user