fix: a native chunk stops walking when its import work would overrun the task
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m15s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 7s
CI and images / build-web (push) Successful in 1m41s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Skipped

The walk's time budget covered the walk alone, but phase 3 runs in the same
Celery task under the same 1350s soft limit. TamadaHeijun's recapture walked
for about two minutes and handed phase 3 431 orphan imports and ~3000
relinks. Phase 3 ran for 20 minutes and died at the soft limit (event
90808). This predates the worker consolidation; the limits are unchanged
since June.

The walk now also stops when elapsed time plus phase 3's estimated cost
(2.5s per import, 0.25s per relink, measured on the live instance) passes
CHUNK_TOTAL_SECONDS (1200). Work handed to phase 3 counts as progress, so
such a stop is a PARTIAL chunk boundary and the next chunk resumes the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 16:13:47 -04:00
co-authored by Claude Opus 5.5
parent ee59133b42
commit 31020d9395
3 changed files with 67 additions and 2 deletions
+24
View File
@@ -846,6 +846,30 @@ async def test_recapture_does_not_refetch_seen_media_missing_from_disk(
assert result.relink_source_paths == []
@pytest.mark.asyncio
async def test_a_chunk_stops_walking_when_phase_3_would_overrun_the_task(
source_id, sync_engine, tmp_path, monkeypatch,
):
"""Phase 3 shares the task's soft limit with the walk. TamadaHeijun's
recapture walked for two minutes and then died importing what it found.
With room for about one import, the walk takes two posts and stops."""
import backend.app.services.ingest_core as core
monkeypatch.setattr(core, "CHUNK_TOTAL_SECONDS", core.PHASE3_IMPORT_SECONDS + 0.5)
pages = [("CUR1", [(f"p{i}", [_media(f"p{i}", 1)]) for i in range(1, 4)])]
downloader = _FakeDownloader(tmp_path)
ing = _ingester(sync_engine, tmp_path, _FakeClient(pages), downloader)
result = ing.run(
source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="backfill", time_budget_seconds=600.0,
)
assert result.files_downloaded == 2
assert downloader.download_calls == 2 # p3 left for the next chunk
assert result.error_type == ErrorType.PARTIAL # a chunk boundary, not a failure
assert result.cursor == "CUR1" # resumes the page it was cut on
# --- a run that dies between download and import ---------------------------
# TamadaHeijun's 【12PCG】 post, 2026-09-24: 13 files on disk, 5 in the library.
# A run wrote 01–08, marked them seen, and was killed before phase 3 imported