fix: a native chunk stops walking when its import work would overrun the task
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m15s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped

The walk's time budget covered the walk alone, but phase 3 runs in the same
Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked
for about two minutes and handed phase 3 431 orphan imports and ~3000
relinks. Phase 3 ran for 20 minutes and died at the soft limit (event
90808). This predates the worker consolidation; the limits are unchanged
since June.

The walk now also stops when elapsed time plus phase 3's estimated cost
(2.5s per import, 0.25s per relink, measured on the live instance) passes
CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so
such a stop is a PARTIAL chunk boundary and the next chunk resumes the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 16:13:47 -04:00
co-authored by Claude Opus 5.5
parent ee59133b42
commit 31020d9395
3 changed files with 67 additions and 2 deletions
+9
View File
@@ -46,6 +46,15 @@ def test_timeout_ladder_keeps_subprocess_budgets_under_soft_limit():
assert DOWNLOAD_SOFT_TIME_LIMIT < DOWNLOAD_HARD_TIME_LIMIT
def test_the_native_chunk_leaves_room_to_tear_down_before_the_soft_limit():
"""The native walk sizes itself against CHUNK_TOTAL_SECONDS, walk plus
phase 3; that total has to leave the task time to finalize its event."""
from backend.app.services.ingest_core import CHUNK_TOTAL_SECONDS
from backend.app.tasks.download import DOWNLOAD_SOFT_TIME_LIMIT
assert CHUNK_TOTAL_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120
def test_decorated_limits_match_module_constants():
"""The @celery.task decorator must use the audited constants, not
drifted literals."""