Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.
New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):
- probe_video(path): validates the container + first video stream via
ffprobe (a separate binary — a decoder crash kills only ffprobe, not
the worker). Returns width/height, which the importer didn't capture
for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
(MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
spawned child process. A decompression-bomb OOM or native-lib
segfault on a malformed archive shows up as a non-zero child exit
code → crashed=True, never a dead worker.
ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).
Wired:
- importer._import_media video branch: probe_video before the pipeline;
crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
→ failed, clean reject → still preserve the archive as a
PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
so a corrupt video is rejected (status='bad_video') instead of
crashing the ml-worker on frame decode.
Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.
Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.
Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff lint surfaced 23 violations across three rules; all addressed:
UP017 (Use datetime.UTC alias):
Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC),
also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for
timezone.utc that ruff's pyupgrade rules prefer.
UP042 (StrEnum):
Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)`
with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and
is the modern idiom. Behavior is equivalent for our usage (the .value
attribute, str(member) semantics).
I001 (Import sorting):
Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so
ruff groups `backend.*` imports correctly. Without it, ruff treated
them as third-party and demanded a different grouping. The existing
import order is stdlib → third-party → first-party → local relative,
which ruff now accepts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-file pipeline: validate as image, apply filter rules (min dimensions,
transparency), SHA256 hash dedup, atomic copy to /images/{subdir}/, create
ImageRecord with origin='imported_filesystem' and integrity_status='unknown',
auto-derive top-level folder name as Artist + artist tag.
Importer is intentionally sync (consumed by a Celery worker process); the
async Quart side uses the same ORM through its own async session. The
db_sync fixture in conftest.py was added in Task 3 to support these tests.
Thumbnail generation is NOT inlined; the calling Celery task enqueues a
separate thumbnail task so the import queue keeps moving on big batches.
pHash dedup is FC-2d, not here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen