bvandeusen
ce560d09a1
Adds per-image integrity tracking so corrupt files are detected, excluded
from random/showcase/ML/suggestion paths, and recoverable by dropping a
fresh copy in /import — closing the gap that surfaced as the WD14
'6 bytes not processed' OSError.
Schema (migration l26042501)
- image_record.integrity_status: unknown | ok | truncated | unreadable | missing
- image_record.integrity_checked_at: timestamptz
- partial index on status <> 'ok' for cheap report/filter queries
Verifier
- app/services/integrity.py: verify_path() dispatches by extension
- PIL two-stage (verify + load with LOAD_TRUNCATED_IMAGES disabled)
- ffprobe for video, zipfile.testzip for archives
- Truncation-vs-unreadable distinction via PIL message hints
Pipeline
- verify_media_integrity Celery task: per-image, idempotent
- verify_unverified_images sweep: only_unknown by default, skips
paths in active import tasks
- Hooked into the end of import_media_file (new + archive paths) and
the supersede branch
- supersede_image() resets status to 'unknown' so the post-supersede
verify writes a fresh truth
- Supersede-on-replace: a fresh /import/<artist>/<filename> matching
a flagged-corrupt record routes through _supersede_existing,
preserving tags/series/embeddings
Exclusions
- /, /api/random-images, tag_and_embed, ml.backfill enqueue, and
get_suggestions all filter integrity_status IN ('ok', 'unknown') so
flagged rows don't poison the gallery, ML, or suggestion math.
'unknown' is treated as healthy so post-migration data stays visible
until the sweep runs.
UI / report
- Settings -> Maintenance: 'Verify unknown' + 'Force re-verify all'
- GET /api/integrity/failed (paginated list of flagged rows)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
64 lines
1.8 KiB
Python
64 lines
1.8 KiB
Python
"""Add integrity_status + integrity_checked_at to image_record.
|
|
|
|
Tracks per-image structural verification state so corrupt files (truncated
|
|
downloads, broken containers, partial archives) can be flagged and excluded
|
|
from random/showcase/ML/suggestion paths instead of blowing up downstream
|
|
preprocessing.
|
|
|
|
Status values:
|
|
- unknown: never verified (default for existing rows + freshly imported
|
|
until the verifier runs)
|
|
- ok: passed marker check + format-level verify
|
|
- truncated: missing trailing bytes (PIL EOI/IEND, ffprobe end-of-stream)
|
|
- unreadable: header/structure invalid; can't even open the file
|
|
- missing: filepath doesn't exist on disk
|
|
|
|
Revision ID: l26042501
|
|
Revises: k26042201
|
|
Create Date: 2026-04-25
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from alembic import op
|
|
import sqlalchemy as sa
|
|
|
|
|
|
revision = 'l26042501'
|
|
down_revision = 'k26042201'
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
|
|
def upgrade():
|
|
op.add_column(
|
|
'image_record',
|
|
sa.Column(
|
|
'integrity_status',
|
|
sa.String(16),
|
|
nullable=False,
|
|
server_default='unknown',
|
|
),
|
|
)
|
|
op.add_column(
|
|
'image_record',
|
|
sa.Column(
|
|
'integrity_checked_at',
|
|
sa.DateTime(timezone=True),
|
|
nullable=True,
|
|
),
|
|
)
|
|
# Partial index so the "list flagged rows" report and the
|
|
# exclude-from-random/ML filters stay cheap on a million-row table.
|
|
op.create_index(
|
|
'image_record_integrity_status_idx',
|
|
'image_record',
|
|
['integrity_status'],
|
|
postgresql_where=sa.text("integrity_status <> 'ok'"),
|
|
)
|
|
|
|
|
|
def downgrade():
|
|
op.drop_index('image_record_integrity_status_idx', table_name='image_record')
|
|
op.drop_column('image_record', 'integrity_checked_at')
|
|
op.drop_column('image_record', 'integrity_status')
|