fix: phase 3 stops importing at a deadline and leaves the rest for the next chunk
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 22s
CI and images / extension-test (push) Successful in 22s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Failing after 2m34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

A Discord recapture chunk on #banana-land was killed at the 1350s soft limit
twice (events 92354 on 2026-09-28 and 92623 on 2026-09-29). The walk's
budget (CHUNK_TOTAL_SECONDS, #4407) assumed 2.5s per import; measured on
those chunks, a real import of the source's doodles took about 5s, so ~290
imports the walk had planned to fit took about twice the time reserved.

- PHASE3_DEADLINE_SECONDS (1200s from task start): on native platforms
  phase 3 starts no import past it. The files left stay on disk with no
  record and are not marked seen; the next chunk resumes on the page that
  holds the first of them and imports them through the existing "on disk
  but never imported" path. A walk that reached its end becomes PARTIAL so
  the backfill is not marked complete while files wait.
- DownloadResult.written_cursors records each written file's page cursor;
  mark_seen_after_import takes skip_paths.
- PHASE3_IMPORT_SECONDS 2.5 -> 5.0, the slow end of what was measured, so
  the walk rarely leans on the deadline.

Task #4493.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 09:53:57 -05:00
co-authored by Claude Opus 5.5
parent 3c3045190d
commit 3f5224800e
6 changed files with 197 additions and 12 deletions
+58
View File
@@ -570,6 +570,64 @@ async def test_backfill_chunk_progress_advances_cursor(
assert remaining == 4
@pytest.mark.asyncio
async def test_phase_3_leaves_imports_past_its_deadline_for_the_next_chunk(
db, db_sync, tmp_path, seed_artist_and_source, monkeypatch,
):
"""A walk that reached its end handed phase 3 more imports than fit before
the deadline. Phase 3 imports what fits, leaves the rest on disk and
unmarked, keeps the backfill running, and resumes it on the page holding
the first file it left."""
import backend.app.services.download_service as ds
_artist, source = seed_artist_and_source
source.config_overrides = {"_backfill_state": "running", "_backfill_cursor": "03:PAGE1"}
source.backfill_runs_remaining = 5
await db.commit()
images_root = tmp_path / "images"
f1 = images_root / "alice" / "patreon" / "post" / "a.jpg"
f2 = images_root / "alice" / "patreon" / "post" / "b.jpg"
_make_jpg(f1, split="h")
_make_jpg(f2, split="v")
result = _make_fake_dl_result(
success=True, written_paths=[str(f1), str(f2)], cursor="03:PAGE3",
)
result.written_cursors = ["03:PAGE1", "03:PAGE2"]
left_unmarked = []
result.mark_seen_after_import = lambda skip_paths=(): left_unmarked.append(
set(skip_paths)
)
# The clock is past the deadline once the first import is done.
past = ds.PHASE3_DEADLINE_SECONDS + 1
ticks = iter([0.0, 0.0])
monkeypatch.setattr(ds, "time", SimpleNamespace(monotonic=lambda: next(ticks, past)))
svc, _ = _backfill_svc(db, db_sync, tmp_path, result)
event_id = await svc.download_source(source.id)
ev = (await db.execute(
select(DownloadEvent).where(DownloadEvent.id == event_id)
)).scalar_one()
assert ev.status == "ok"
assert ev.files_count == 1
assert "1 file(s) left for the next chunk" in ev.metadata_["stdout"]
assert left_unmarked == [{str(f2)}]
assert f2.exists()
assert db_sync.execute(
select(func.count(ImageRecord.id)).where(ImageRecord.path == str(f2))
).scalar_one() == 0
remaining, co = (await db.execute(
select(Source.backfill_runs_remaining, Source.config_overrides)
.where(Source.id == source.id)
)).one()
assert co.get("_backfill_state") == "running"
assert co.get("_backfill_cursor") == "03:PAGE2"
assert remaining == 4
@pytest.mark.asyncio
async def test_backfill_state_running_selects_backfill_mode_and_resumes(
db, db_sync, tmp_path, seed_artist_and_source,