perf: GPU-job leasing stays O(batch) — partial indexes + two-phase lease #166

2026-06-30T21:18:19-04:00

bvandeusen commented

2026-06-30 21:18:19 -04:00

The agent's low throughput was a curator-side defect, not network/hardware.

lease() claimed the lowest-id pending jobs with ORDER BY id LIMIT n, but with only a plain status index Postgres walked the primary key from id=1, skipping the entire prefix of already done/error rows before reaching pending ones. With ~69k done, every lease was an O(done) scan → leasing crawled, the DB saturated, and even the /status GROUP-BY count stalled the agent.

Migration 0070: two partial indexes over just the live slice — pending rows by id (hot path), leased rows by lease_expires_at (crash-recovery + orphan sweep). Stay tiny regardless of done/error history.
lease() two-phase: claim pending first (id-ordered, O(batch)); reclaim expired leases only when pending can't fill the batch. Same SKIP LOCKED / attempts / reclaim semantics.
Model __table_args__ declares the indexes; new test asserts a low-id done-prefix doesn't stop the lease reaching pending.

CI green on dev (integration job ran the migration + lease tests).

🤖 Generated with Claude Code

The agent's low throughput was a **curator-side** defect, not network/hardware. `lease()` claimed the lowest-id pending jobs with `ORDER BY id LIMIT n`, but with only a plain `status` index Postgres walked the primary key from id=1, skipping the entire prefix of already done/error rows before reaching pending ones. With ~69k done, every lease was an **O(done) scan** → leasing crawled, the DB saturated, and even the `/status` GROUP-BY count stalled the agent. - **Migration 0070**: two partial indexes over just the live slice — pending rows by id (hot path), leased rows by `lease_expires_at` (crash-recovery + orphan sweep). Stay tiny regardless of done/error history. - **`lease()` two-phase**: claim pending first (id-ordered, O(batch)); reclaim expired leases only when pending can't fill the batch. Same SKIP LOCKED / attempts / reclaim semantics. - Model `__table_args__` declares the indexes; new test asserts a low-id done-prefix doesn't stop the lease reaching pending. CI green on dev (integration job ran the migration + lease tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 1 commit 2026-06-30 21:18:19 -04:00

perf(gpu-queue): partial indexes + two-phase lease so leasing stays O(batch)

CI / lint (push) Successful in 3s

Details

CI / frontend-build (push) Successful in 20s

Details

CI / backend-lint-and-test (push) Successful in 26s

Details

CI / integration (push) Successful in 3m25s

Details

181f1c6a27

The throughput bottleneck was curator-side, not the network. lease() claimed the
lowest-id pending/expired jobs with `... ORDER BY id LIMIT n`, but with only a
plain `status` index Postgres walked the primary key from id=1, skipping the
entire prefix of already done/error rows before reaching pending ones. As `done`
grew (69k+), every lease became an O(done) scan — leasing crawled, the DB
saturated, and even /status (the queue GROUP BY count) stalled the agent.

- Migration 0070 adds two partial indexes over just the live slice: pending rows
  indexed by id (hot path), and leased rows by lease_expires_at (crash-recovery
  + orphan sweep). They stay tiny no matter how large the done/error history.
- lease() split into two phases so each uses a partial index: claim pending
  first (id-ordered, O(batch)); reclaim expired leases only when pending can't
  fill the batch. Same semantics (SKIP LOCKED, attempts++, expired reclaim).
- Model __table_args__ declares the indexes so ORM and schema agree.
- Test: a done-prefix at low ids must not stop the lease reaching pending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

bvandeusen merged commit baac851220 into main

2026-06-30 21:18:23 -04:00

bvandeusen referenced this pull request

2026-06-30 21:28:54 -04:00

Release: agent per-stage timing breakdown #167

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#166