fix: task runs record the lane Celery really routes them to, and no healthy long job is swept as stalled (#4432)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Failing after 2m24s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

celery_signals._queue_for was a hand-kept copy of task_routes and had
drifted:

- backup, admin and library_audit jobs, and backfill_phash, run on
  maintenance_long but were recorded as `maintenance`;
- translation and gpu_queue jobs were recorded as `default`, where the
  5-minute stall sweep failed healthy 35-minute translation runs.

It now asks the router, cached per task name.

A new guard test checks every registered task's hard time limit against
the stall threshold the sweep would use for it. It also caught these
sweeps, which failed healthy runs mid-flight and are fixed here:

- ml's scheduled sweeps (35 min, previously swept at 25);
- train_heads and apply_head_tags (65 min);
- import_media_file (6 min, previously swept at 5);
- the long lane, which now has its own 45-minute threshold.

UI changes:

- the admin job poller follows a job by celery_task_id, via a new filter
  on /runs, instead of by lane;
- the archive re-extract and missing-file repair cards show the long
  lane's backlog, where their jobs actually wait;
- the queue table lists maintenance_long.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-25 09:46:39 -04:00
co-authored by Claude Opus 5.5
parent dfd28a0aa6
commit a360d69ee8
9 changed files with 112 additions and 50 deletions
@@ -17,7 +17,7 @@
<v-icon start>mdi-folder-zip-outline</v-icon> Re-extract archives now
</v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
<QueueStatusBar queue="maintenance_long" queue-label="Long maintenance" />
</MaintenanceTile>
</template>
@@ -18,7 +18,7 @@
<v-icon start>mdi-file-remove-outline</v-icon> Repair missing-file records
</v-btn>
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
<QueueStatusBar queue="maintenance" queue-label="Maintenance" />
<QueueStatusBar queue="maintenance_long" queue-label="Long maintenance" />
</MaintenanceTile>
</template>
@@ -41,7 +41,7 @@ const props = defineProps({
const QUEUE_NAMES = [
'default', 'import', 'thumbnail', 'ml',
'download', 'scan', 'maintenance',
'download', 'scan', 'maintenance', 'maintenance_long',
]
function formatDepth(name) {
+4 -2
View File
@@ -119,7 +119,7 @@ export const useAdminStore = defineStore('admin', () => {
// --- Task progress polling (taps FC-3i activity dashboard) --------
/**
* Polls /api/system/activity/runs?queue=maintenance every 3s,
* Polls /api/system/activity/runs?celery_task_id=<id> every 3s,
* resolves when a task_run row with the given celery task_id
* reaches a terminal status (ok / error / timeout). Returns the
* row. Times out after 30 min by default.
@@ -129,7 +129,9 @@ export const useAdminStore = defineStore('admin', () => {
while (Date.now() < deadline) {
const body = await api.get(
'/api/system/activity/runs',
{ params: { queue: 'maintenance', limit: 20 } },
// By id, not by lane: these jobs run on `maintenance_long`, and a
// lane filter here is one more copy of the routing table (#4432).
{ params: { celery_task_id: taskId, limit: 1 } },
)
const row = (body.runs || []).find(r => r.celery_task_id === taskId)
if (row && ['ok', 'error', 'timeout'].includes(row.status)) {