feat(gpu): "Retry errored jobs" — scoped requeue of errors only
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m26s

After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.

POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.

UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.

Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-01 21:09:07 -04:00
parent 3a683d7feb
commit 686808d3f3
4 changed files with 92 additions and 3 deletions
@@ -52,6 +52,17 @@
<div class="fc-q"><div class="fc-q__n" :class="queue.error ? 'fc-weak' : ''">{{ queue.error }}</div><div class="fc-q__l">errored</div></div>
</div>
<v-btn
class="mt-3" color="accent" variant="tonal" rounded="pill" size="small"
prepend-icon="mdi-restart-alert" :loading="retrying"
:disabled="!queue.error" @click="onRetryErrors"
>Retry errored jobs</v-btn>
<p class="fc-muted text-caption mt-2 mb-0">
Errored jobs park after 3 failed attempts. This requeues just those (their
errors cleared, attempts reset) use after updating the agent, without
re-running the whole done library.
</p>
<v-btn
class="mt-4" color="accent" variant="tonal" rounded="pill" size="small"
prepend-icon="mdi-account-box-multiple" :loading="backfilling" @click="onBackfill"
@@ -164,6 +175,7 @@ const rotating = ref(false)
const backfilling = ref(false)
const backfillingSiglip = ref(false)
const reprocessing = ref(false)
const retrying = ref(false)
const threshold = ref(0.85)
const savingThreshold = ref(false)
const autoApply = ref(true)
@@ -323,6 +335,19 @@ async function onBackfillSiglip() {
}
}
async function onRetryErrors() {
retrying.value = true
try {
const { requeued } = await store.retryErrors()
toast({ text: `Requeued ${requeued} errored job${requeued === 1 ? '' : 's'} — run the agent to process them`, type: 'success' })
await refreshQueue()
} catch (e) {
toast({ text: `Could not retry errored jobs: ${e.message}`, type: 'error' })
} finally {
retrying.value = false
}
}
async function onReprocess() {
if (!window.confirm('Re-process the ENTIRE library (re-detect + re-crop every image)? This is heavy and runs on the GPU agent.')) return
reprocessing.value = true
+7 -1
View File
@@ -35,5 +35,11 @@ export const useGpuStore = defineStore('gpu', () => {
return await api.post('/api/gpu/reprocess', { body: { task } })
}
return { token, rotateToken, status, backfill, reprocess }
// Requeue ONLY the errored jobs (all task types) — the scoped recovery after
// an agent fix, without re-running the done library. Returns { requeued }.
async function retryErrors() {
return await api.post('/api/gpu/retry_errors')
}
return { token, rotateToken, status, backfill, reprocess, retryErrors }
})