feat(import-resilience L3): subprocess-isolated probes for video + archive
Layer 3 — prevent the hard worker crash rather than just recovering from it. The realistic process-crash vectors (operator's observed slow/heavy tasks) are video decode and archive extraction; images decode in-process and Pillow raises-and-skips cleanly, and a subprocess per image would wreck deep-scan throughput, so images are intentionally not probed. New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so the spawned child stays light): - probe_video(path): validates the container + first video stream via ffprobe (a separate binary — a decoder crash kills only ffprobe, not the worker). Returns width/height, which the importer didn't capture for videos before. crashed=True only on ffprobe timeout. - probe_archive(path): an uncompressed-size bomb guard (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a spawned child process. A decompression-bomb OOM or native-lib segfault on a malformed archive shows up as a non-zero child exit code → crashed=True, never a dead worker. ProbeResult.crashed distinguishes a HARD failure (subprocess killed / timed out — the poison-pill signature → caller returns terminal 'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap, integrity mismatch → caller's choice of skipped/attached). Wired: - importer._import_media video branch: probe_video before the pipeline; crash → failed, clean reject → invalid_image skip, ok → capture dims. - importer._import_archive: probe_archive before extract_archive; crash → failed, clean reject → still preserve the archive as a PostAttachment (matches extract_archive's fail-soft contract). - ml.tag_and_embed video branch: probe_video before sampling 10 frames, so a corrupt video is rejected (status='bad_video') instead of crashing the ml-worker on frame decode. Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct _inspect_archive size+integrity, in-process _archive_probe_target bomb guard (monkeypatch can't reach a spawned child, so the target is called directly), and a non-video → ok=False that's robust to ffprobe presence in CI.
This commit is contained in:
@@ -71,6 +71,18 @@ def tag_and_embed(self, image_id: int) -> dict:
|
||||
embedder = get_embedder()
|
||||
|
||||
if _is_video(src):
|
||||
# Layer-3 isolation: ffprobe (a separate process) validates
|
||||
# the container before we burn ~20 GPU ops sampling frames
|
||||
# from it. A corrupt video that would crash the frame
|
||||
# decoder is rejected cleanly here instead of taking down
|
||||
# the ml-worker. Operator-flagged 2026-05-28.
|
||||
from ..utils import safe_probe
|
||||
vprobe = safe_probe.probe_video(src)
|
||||
if not vprobe.ok:
|
||||
return {
|
||||
"status": "bad_video", "image_id": image_id,
|
||||
"reason": vprobe.reason,
|
||||
}
|
||||
frames = _sample_video_frames(
|
||||
src, int(os.environ.get("VIDEO_ML_FRAMES", "10"))
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user