fix: a native chunk stops walking when its import work would overrun the task
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m15s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m15s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped
The walk's time budget covered the walk alone, but phase 3 runs in the same Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked for about two minutes and handed phase 3 431 orphan imports and ~3000 relinks. Phase 3 ran for 20 minutes and died at the soft limit (event 90808). This predates the worker consolidation; the limits are unchanged since June. The walk now also stops when elapsed time plus phase 3's estimated cost (2.5s per import, 0.25s per relink, measured on the live instance) passes CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so such a stop is a PARTIAL chunk boundary and the next chunk resumes the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -846,6 +846,30 @@ async def test_recapture_does_not_refetch_seen_media_missing_from_disk(
|
||||
assert result.relink_source_paths == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_chunk_stops_walking_when_phase_3_would_overrun_the_task(
|
||||
source_id, sync_engine, tmp_path, monkeypatch,
|
||||
):
|
||||
"""Phase 3 shares the task's soft limit with the walk. TamadaHeijun's
|
||||
recapture walked for two minutes and then died importing what it found.
|
||||
With room for about one import, the walk takes two posts and stops."""
|
||||
import backend.app.services.ingest_core as core
|
||||
monkeypatch.setattr(core, "CHUNK_TOTAL_SECONDS", core.PHASE3_IMPORT_SECONDS + 0.5)
|
||||
|
||||
pages = [("CUR1", [(f"p{i}", [_media(f"p{i}", 1)]) for i in range(1, 4)])]
|
||||
downloader = _FakeDownloader(tmp_path)
|
||||
ing = _ingester(sync_engine, tmp_path, _FakeClient(pages), downloader)
|
||||
result = ing.run(
|
||||
source_id=source_id, campaign_id="c1", artist_slug="ingest",
|
||||
url="https://patreon.com/ingest", mode="backfill", time_budget_seconds=600.0,
|
||||
)
|
||||
|
||||
assert result.files_downloaded == 2
|
||||
assert downloader.download_calls == 2 # p3 left for the next chunk
|
||||
assert result.error_type == ErrorType.PARTIAL # a chunk boundary, not a failure
|
||||
assert result.cursor == "CUR1" # resumes the page it was cut on
|
||||
|
||||
|
||||
# --- a run that dies between download and import ---------------------------
|
||||
# TamadaHeijun's 【12PCG】 post, 2026-09-24: 13 files on disk, 5 in the library.
|
||||
# A run wrote 01–08, marked them seen, and was killed before phase 3 imported
|
||||
|
||||
Reference in New Issue
Block a user