perf(gpu-queue): partial indexes + two-phase lease so leasing stays O(batch)
The throughput bottleneck was curator-side, not the network. lease() claimed the lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a plain `status` index Postgres walked the primary key from id=1, skipping the entire prefix of already done/error rows before reaching pending ones. As `done` grew (69k+), every lease became an O(done) scan — leasing crawled, the DB saturated, and even /status (the queue GROUP BY count) stalled the agent. - Migration 0070 adds two partial indexes over just the live slice: pending rows indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery + orphan sweep). They stay tiny no matter how large the done/error history. - lease() split into two phases so each uses a partial index: claim pending first (id-ordered, O(batch)); reclaim expired leases only when pending can't fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim). - Model __table_args__ declares the indexes so ORM and schema agree. - Test: a done-prefix at low ids must not stop the lease reaching pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -141,6 +141,22 @@ async def test_lease_claims_then_skips_when_held(db):
|
||||
assert again == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lease_skips_done_prefix(db):
|
||||
# A long prefix of already-done jobs at the LOW ids must not stop the lease
|
||||
# from reaching the pending ones (the O(done) scan bug this indexing fixes).
|
||||
svc = GpuJobService(db)
|
||||
for i in range(5):
|
||||
j = await svc.enqueue((await _img(db, f"d{i}" * 32)).id, "ccip")
|
||||
j.status = "done"
|
||||
target = await svc.enqueue((await _img(db, "dz" * 32)).id, "ccip") # highest id
|
||||
await db.commit()
|
||||
|
||||
leased = await svc.lease("agent-1", batch_size=8)
|
||||
await db.commit()
|
||||
assert [j.id for j in leased] == [target.id]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_expired_lease_is_reclaimed(db):
|
||||
img = await _img(db, "c" * 64)
|
||||
|
||||
Reference in New Issue
Block a user