fix: task runs record the lane Celery really routes them to, and no healthy long job is swept as stalled (#4432)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Failing after 2m24s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 28s
CI and images / backend-lint-and-test (push) Successful in 34s
CI and images / integration (push) Failing after 2m24s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
celery_signals._queue_for was a hand-kept copy of task_routes and had drifted: - backup, admin and library_audit jobs, and backfill_phash, run on maintenance_long but were recorded as `maintenance`; - translation and gpu_queue jobs were recorded as `default`, where the 5-minute stall sweep failed healthy 35-minute translation runs. It now asks the router, cached per task name. A new guard test checks every registered task's hard time limit against the stall threshold the sweep would use for it. It also caught these sweeps, which failed healthy runs mid-flight and are fixed here: - ml's scheduled sweeps (35 min, previously swept at 25); - train_heads and apply_head_tags (65 min); - import_media_file (6 min, previously swept at 5); - the long lane, which now has its own 45-minute threshold. UI changes: - the admin job poller follows a job by celery_task_id, via a new filter on /runs, instead of by lane; - the archive re-extract and missing-file repair cards show the long lane's backlog, where their jobs actually wait; - the queue table lists maintenance_long. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -137,7 +137,13 @@ IMPORT_BATCH_KEEP_DAYS = 30
|
||||
# (the import queue itself stays at the 5-min default for single
|
||||
# files); time_limit=2100.
|
||||
QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
"ml": 25,
|
||||
# ml: the scheduled auto-apply sweeps and refresh_character_prototypes run
|
||||
# to a 35-min hard limit (2100s); 25 swept them mid-run (#4432). The two
|
||||
# 65-min jobs have their own entries below.
|
||||
"ml": 40,
|
||||
# import: import_media_file's hard limit is 6 min (360s), one past the
|
||||
# 5-min default this queue fell to (#4432).
|
||||
"import": 10,
|
||||
# download_source legitimately walks 5-25 min (Patreon/gallery-dl
|
||||
# deep creators); its hard time_limit is DOWNLOAD_HARD_TIME_LIMIT
|
||||
# (1500s = 25m). The 5-min default flagged healthy in-flight walks as
|
||||
@@ -154,6 +160,12 @@ QUEUE_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
# overrides below cover the outliers (backups, library audit).
|
||||
"maintenance": 75,
|
||||
"scan": 75,
|
||||
# The long lane (#4432). Until TaskRun.queue asked the router, nothing was
|
||||
# recorded here: these runs read as `maintenance` (75) or, for
|
||||
# translation, `default` (5 — which failed healthy 35-min runs). The
|
||||
# longest task without its own entry below is the admin family at a
|
||||
# 40-min hard limit; 45 = 40 + 5.
|
||||
"maintenance_long": 45,
|
||||
}
|
||||
TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
"backend.app.tasks.import_file.import_archive_file": 40,
|
||||
@@ -179,6 +191,10 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
# external-fetch entry above — without an override a healthy in-flight walk
|
||||
# is swept 'RecoverySweep' at the bare 5-min default. 30 = 25 + 5.
|
||||
"backend.app.tasks.admin.reclaim_orphaned_attachments_task": 30,
|
||||
# Head training and the manual head apply run to 65 min (3900s) — past the
|
||||
# ml queue's threshold (#4432). 70 = 65 + 5.
|
||||
"backend.app.tasks.ml.train_heads": 70,
|
||||
"backend.app.tasks.ml.apply_head_tags": 70,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user