fix: phase 3 stops importing at a deadline and leaves the rest for the next chunk
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
A Discord recapture chunk on #banana-land was killed at the 1350s soft limit twice (events 92354 on 2026-09-28 and 92623 on 2026-09-29). The walk's budget (CHUNK_TOTAL_SECONDS, #4407) assumed 2.5s per import; measured on those chunks, a real import of the source's doodles took about 5s, so ~290 imports the walk had planned to fit took about twice the time reserved. - PHASE3_DEADLINE_SECONDS (1200s from task start): on native platforms phase 3 starts no import past it. The files left stay on disk with no record and are not marked seen; the next chunk resumes on the page that holds the first of them and imports them through the existing "on disk but never imported" path. A walk that reached its end becomes PARTIAL so the backfill is not marked complete while files wait. - DownloadResult.written_cursors records each written file's page cursor; mark_seen_after_import takes skip_paths. - PHASE3_IMPORT_SECONDS 2.5 -> 5.0, the slow end of what was measured, so the walk rarely leans on the deadline. Task #4493. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -55,6 +55,15 @@ def test_the_native_chunk_leaves_room_to_tear_down_before_the_soft_limit():
|
||||
assert CHUNK_TOTAL_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120
|
||||
|
||||
|
||||
def test_phase_3_stops_importing_with_room_to_finalize():
|
||||
"""Past its deadline phase 3 starts no import; what remains after it (the
|
||||
import in hand, post records, the event) has to fit before the soft limit."""
|
||||
from backend.app.services.ingest_core import PHASE3_DEADLINE_SECONDS
|
||||
from backend.app.tasks.download import DOWNLOAD_SOFT_TIME_LIMIT
|
||||
|
||||
assert PHASE3_DEADLINE_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120
|
||||
|
||||
|
||||
def test_decorated_limits_match_module_constants():
|
||||
"""The @celery.task decorator must use the audited constants, not
|
||||
drifted literals."""
|
||||
|
||||
Reference in New Issue
Block a user