feat(gpu): "Retry errored jobs" — scoped requeue of errors only
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -52,6 +52,17 @@
|
||||
<div class="fc-q"><div class="fc-q__n" :class="queue.error ? 'fc-weak' : ''">{{ queue.error }}</div><div class="fc-q__l">errored</div></div>
|
||||
</div>
|
||||
|
||||
<v-btn
|
||||
class="mt-3" color="accent" variant="tonal" rounded="pill" size="small"
|
||||
prepend-icon="mdi-restart-alert" :loading="retrying"
|
||||
:disabled="!queue.error" @click="onRetryErrors"
|
||||
>Retry errored jobs</v-btn>
|
||||
<p class="fc-muted text-caption mt-2 mb-0">
|
||||
Errored jobs park after 3 failed attempts. This requeues just those (their
|
||||
errors cleared, attempts reset) — use after updating the agent, without
|
||||
re-running the whole done library.
|
||||
</p>
|
||||
|
||||
<v-btn
|
||||
class="mt-4" color="accent" variant="tonal" rounded="pill" size="small"
|
||||
prepend-icon="mdi-account-box-multiple" :loading="backfilling" @click="onBackfill"
|
||||
@@ -164,6 +175,7 @@ const rotating = ref(false)
|
||||
const backfilling = ref(false)
|
||||
const backfillingSiglip = ref(false)
|
||||
const reprocessing = ref(false)
|
||||
const retrying = ref(false)
|
||||
const threshold = ref(0.85)
|
||||
const savingThreshold = ref(false)
|
||||
const autoApply = ref(true)
|
||||
@@ -323,6 +335,19 @@ async function onBackfillSiglip() {
|
||||
}
|
||||
}
|
||||
|
||||
async function onRetryErrors() {
|
||||
retrying.value = true
|
||||
try {
|
||||
const { requeued } = await store.retryErrors()
|
||||
toast({ text: `Requeued ${requeued} errored job${requeued === 1 ? '' : 's'} — run the agent to process them`, type: 'success' })
|
||||
await refreshQueue()
|
||||
} catch (e) {
|
||||
toast({ text: `Could not retry errored jobs: ${e.message}`, type: 'error' })
|
||||
} finally {
|
||||
retrying.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function onReprocess() {
|
||||
if (!window.confirm('Re-process the ENTIRE library (re-detect + re-crop every image)? This is heavy and runs on the GPU agent.')) return
|
||||
reprocessing.value = true
|
||||
|
||||
@@ -35,5 +35,11 @@ export const useGpuStore = defineStore('gpu', () => {
|
||||
return await api.post('/api/gpu/reprocess', { body: { task } })
|
||||
}
|
||||
|
||||
return { token, rotateToken, status, backfill, reprocess }
|
||||
// Requeue ONLY the errored jobs (all task types) — the scoped recovery after
|
||||
// an agent fix, without re-running the done library. Returns { requeued }.
|
||||
async function retryErrors() {
|
||||
return await api.post('/api/gpu/retry_errors')
|
||||
}
|
||||
|
||||
return { token, rotateToken, status, backfill, reprocess, retryErrors }
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user