Agent: short-video sampling fix + "Retry errored jobs" recovery action #185

Merged

bvandeusen merged 2 commits from dev into main 2026-07-01 21:13:48 -04:00

Conversation 0 Commits 2 Files Changed 6

+149 -24

bvandeusen commented 2026-07-01 21:13:40 -04:00

Owner

Copy Link

Copy Source

Two changes that ship together — the fix and its recovery path. CI-green on 686808d (run 1850, incl. the new integration test).

3a683d7 — short videos failed as "unprocessable" (agent, root cause)

The sampler used -vf fps=1/4; ffmpeg's fps filter emits round(duration/4) frames — zero for any clip shorter than ~2 s. Short animation loops (verified against two originals: a 0.5 s 4440×2760 clip and a 1.75 s clip — both complete, valid h264) decoded fine, produced no frames, and were failed as unprocessable: the source of the "no frames sampled from video" flood (~2.8k errored jobs).

• Sample via select (first frame always, then one per interval) + -fps_mode vfr; scale=out_range=full fixes a secondary mjpeg full-range strictness rejection. Verified on both failing originals + a synthetic 15 s video (long-video behavior unchanged: 4 frames at t=0/4/8/12).
• ffmpeg stderr now goes to a temp file and its tail is logged on any no-frames/timeout failure — failures name their real reason from now on (this bug hid for weeks behind a discarded stderr).
• Frames written before a mid-stream ffmpeg error are kept (partial > nothing). Agent VERSION → 2026-07-02.1.

686808d — "Retry errored jobs" (curator web/backend)

Errored jobs exhaust their 3 attempts and park: backfills skip them, and /reprocess is nuclear (resets the 179k done jobs too). New scoped recovery:

• POST /api/gpu/retry_errors — resets every status='error' job (all task types) to pending, attempts=0, error cleared; returns {requeued: n}.
• UI: Retry errored jobs button on the GPU-agent Settings card under the queue tiles (disabled at 0; toast shows the count).
• Integration test: requeues errored, leaves done untouched.

Deploy note

Both the agent image and web image need a re-pull. After deploy: click Retry errored jobs once — the short clips will process under the fixed sampler, and anything still failing will log its actual ffmpeg error. No migrations.

🤖 Generated with Claude Code

Two changes that ship together — the fix and its recovery path. CI-green on `686808d` (run 1850, incl. the new integration test). ## `3a683d7` — short videos failed as "unprocessable" (agent, root cause) The sampler used `-vf fps=1/4`; ffmpeg's fps filter emits `round(duration/4)` frames — **zero for any clip shorter than ~2 s**. Short animation loops (verified against two originals: a 0.5 s 4440×2760 clip and a 1.75 s clip — both complete, valid h264) decoded fine, produced no frames, and were failed as unprocessable: the source of the "no frames sampled from video" flood (~2.8k errored jobs). - Sample via `select` (first frame **always**, then one per interval) + `-fps_mode vfr`; `scale=out_range=full` fixes a secondary mjpeg full-range strictness rejection. Verified on both failing originals + a synthetic 15 s video (long-video behavior unchanged: 4 frames at t=0/4/8/12). - ffmpeg stderr now goes to a temp file and its tail is **logged on any no-frames/timeout failure** — failures name their real reason from now on (this bug hid for weeks behind a discarded stderr). - Frames written before a mid-stream ffmpeg error are kept (partial > nothing). Agent VERSION → `2026-07-02.1`. ## `686808d` — "Retry errored jobs" (curator web/backend) Errored jobs exhaust their 3 attempts and park: backfills skip them, and `/reprocess` is nuclear (resets the 179k done jobs too). New scoped recovery: - `POST /api/gpu/retry_errors` — resets every `status='error'` job (all task types) to pending, attempts=0, error cleared; returns `{requeued: n}`. - UI: **Retry errored jobs** button on the GPU-agent Settings card under the queue tiles (disabled at 0; toast shows the count). - Integration test: requeues errored, leaves done untouched. ## Deploy note Both the **agent** image and **web** image need a re-pull. After deploy: click *Retry errored jobs* once — the short clips will process under the fixed sampler, and anything still failing will log its actual ffmpeg error. No migrations. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-07-01 21:13:40 -04:00

fix(agent): short videos failed as "unprocessable" — fps filter emits 0 frames ... 3a683d7feb

Root cause of the "no frames sampled from video (unprocessable)" flood
(operator-flagged 2026-07-02, whole 62k-70k image block + others): the
sampler used `-vf fps=1/4`, and ffmpeg's fps filter emits round(duration/4)
frames — which is ZERO for any clip shorter than ~2s. Short animation loops
(0.5s, 1.75s — verified against two originals from different artists) are
complete, valid h264 videos; ffmpeg decoded them fine, emitted no frames,
exited 0, and the agent failed the job as unprocessable. Long videos worked,
so only the short-clip class flooded.

Fix: sample with select ("first frame always, then one per interval of
timestamp") + -fps_mode vfr, and scale=out_range=full so limited-range
yuv420p sources don't trip the mjpeg encoder's full-range strictness
(secondary failure observed on a 4440x2760 clip). Verified locally against
both failing originals (frames extracted, PIL-clean) and a synthetic 15s
video (4 frames at t=0/4/8/12 — long-video behavior unchanged).

Observability (why this hid for weeks): ffmpeg's stderr was discarded, so
every failure logged only "no frames sampled". stderr now goes to a temp
file and its tail is logged on any produced-no-frames/timeout failure — the
log names the actual ffmpeg reason from now on. Also: frames written before
a mid-stream ffmpeg error are now kept (partial > nothing).

VERSION → 2026-07-02.1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(gpu): "Retry errored jobs" — scoped requeue of errors only ... 686808d3f3

After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.

POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.

UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.

Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bvandeusen merged commit d9354ac1e1 into main 2026-07-01 21:13:48 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#185