Operator, 2026-09-23: *"something about changing the cap number is blocking to
the website... it shouldn't be"*.
Nothing here was slow in itself. A database connection was held across work
that is slow, and that is why it surfaced as the whole site stalling rather
than as one slow page.
`lane_view` took the session and kept it open through a celery inspect whose
budget is 11s. The System tab polls that endpoint every 15s — and with a lane
not answering, every inspect runs to nearly its full budget, so each poll
pinned a connection for most of the interval. SQLAlchemy's default pool is 5
plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing,
and a cap change adding two more inspects exhausts it, and every OTHER request
then waits for a connection.
Split so the database work finishes before the broker work starts:
- `lane_settings(session)` reads the caps and the oldest running task, then
the session closes. `lane_view(settings)` does the inspect with none held.
- `store_lane_cap(session, …)` validates and commits, then the session closes.
`push_lane_cap(lane, …)` does the live push with none held.
And a second finding while measuring it: **raising a cap now costs no broker
round trip at all.** The first cut only knew on/off, so it inspected on every
raise to find out whether the pool needed lowering — the control meant to be
instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap
so the push knows the direction; only a lowering needs to say anything.
The guard is structural, not timed: `lane_view` and `push_lane_cap` must not
ACCEPT a session. A timing test would be flaky, and a call-order test would
pass against a version that took the session and merely used it early.
`/api/system/health` has the same shape and is NOT fixed here — it is
rate-limited by `refresh_if_stale` so it does not inspect on every request.
Worth doing, separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
bvandeusenandClaude Opus 5
5b6f2ba526