fix: phase 3 stops importing at a deadline and leaves the rest for the next chunk
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

A Discord recapture chunk on #banana-land was killed at the 1350s soft limit
twice (events 92354 on 2026-09-28 and 92623 on 2026-09-29). The walk's
budget (CHUNK_TOTAL_SECONDS, #4407) assumed 2.5s per import; measured on
those chunks, a real import of the source's doodles took about 5s, so ~290
imports the walk had planned to fit took about twice the time reserved.

- PHASE3_DEADLINE_SECONDS (1200s from task start): on native platforms
  phase 3 starts no import past it. The files left stay on disk with no
  record and are not marked seen; the next chunk resumes on the page that
  holds the first of them and imports them through the existing "on disk
  but never imported" path. A walk that reached its end becomes PARTIAL so
  the backfill is not marked complete while files wait.
- DownloadResult.written_cursors records each written file's page cursor;
  mark_seen_after_import takes skip_paths.
- PHASE3_IMPORT_SECONDS 2.5 -> 5.0, the slow end of what was measured, so
  the walk rarely leans on the deadline.

Task #4493.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 09:53:57 -05:00
co-authored by Claude Opus 5.5
parent 3c3045190d
commit 3f5224800e
6 changed files with 197 additions and 12 deletions
+9
View File
@@ -55,6 +55,15 @@ def test_the_native_chunk_leaves_room_to_tear_down_before_the_soft_limit():
assert CHUNK_TOTAL_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120
def test_phase_3_stops_importing_with_room_to_finalize():
"""Past its deadline phase 3 starts no import; what remains after it (the
import in hand, post records, the event) has to fit before the soft limit."""
from backend.app.services.ingest_core import PHASE3_DEADLINE_SECONDS
from backend.app.tasks.download import DOWNLOAD_SOFT_TIME_LIMIT
assert PHASE3_DEADLINE_SECONDS <= DOWNLOAD_SOFT_TIME_LIMIT - 120
def test_decorated_limits_match_module_constants():
"""The @celery.task decorator must use the audited constants, not
drifted literals."""