fix: phase 3 stops importing at a deadline and leaves the rest for the next chunk
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

A Discord recapture chunk on #banana-land was killed at the 1350s soft limit
twice (events 92354 on 2026-09-28 and 92623 on 2026-09-29). The walk's
budget (CHUNK_TOTAL_SECONDS, #4407) assumed 2.5s per import; measured on
those chunks, a real import of the source's doodles took about 5s, so ~290
imports the walk had planned to fit took about twice the time reserved.

- PHASE3_DEADLINE_SECONDS (1200s from task start): on native platforms
  phase 3 starts no import past it. The files left stay on disk with no
  record and are not marked seen; the next chunk resumes on the page that
  holds the first of them and imports them through the existing "on disk
  but never imported" path. A walk that reached its end becomes PARTIAL so
  the backfill is not marked complete while files wait.
- DownloadResult.written_cursors records each written file's page cursor;
  mark_seen_after_import takes skip_paths.
- PHASE3_IMPORT_SECONDS 2.5 -> 5.0, the slow end of what was measured, so
  the walk rarely leans on the deadline.

Task #4493.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 09:53:57 -05:00
co-authored by Claude Opus 5.5
parent 3c3045190d
commit 3f5224800e
6 changed files with 197 additions and 12 deletions
+25
View File
@@ -984,6 +984,31 @@ async def test_recapture_imports_a_seen_file_nothing_imported(
assert downloader.download_calls == 0
@pytest.mark.asyncio
async def test_files_phase_3_leaves_unimported_stay_unmarked(
source_id, sync_engine, tmp_path,
):
"""Phase 3 can stop importing at its deadline. Each written file names the
page it came from, so the next chunk can resume there; the files left keep
no ledger mark, so that walk meets them on disk and imports them."""
m1, m2 = _media("p1", 1), _media("p2", 1)
pages = [("CUR1", [("p1", [m1])]), ("CUR2", [("p2", [m2])])]
ing = _ingester(sync_engine, tmp_path, _FakeClient(pages), _FakeDownloader(tmp_path))
result = ing.run(source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="backfill")
assert result.written_cursors == ["CUR1", "CUR2"]
result.mark_seen_after_import(skip_paths={str(tmp_path / "p2_1.jpg")})
assert _count_ledger(sync_engine, source_id) == 3 # m1 and both post records
ing2 = _ingester(sync_engine, tmp_path, _FakeClient(pages),
_FakeDownloader(tmp_path, on_disk={_ledger_key(m1), _ledger_key(m2)}))
again = ing2.run(source_id=source_id, campaign_id="c1", artist_slug="ingest",
url="https://patreon.com/ingest", mode="backfill")
assert again.written_paths == [str(tmp_path / "p2_1.jpg")]
assert "on disk but never imported: p2_1.jpg" in again.stdout
# --- dead-letter ledger (plan #705 #7) ------------------------------------