fix: a Postgres connection was held across every celery round trip (4295)

Operator, 2026-09-23: *"something about changing the cap number is blocking to
the website... it shouldn't be"*.

Nothing here was slow in itself. A database connection was held across work
that is slow, and that is why it surfaced as the whole site stalling rather
than as one slow page.

`lane_view` took the session and kept it open through a celery inspect whose
budget is 11s. The System tab polls that endpoint every 15s — and with a lane
not answering, every inspect runs to nearly its full budget, so each poll
pinned a connection for most of the interval. SQLAlchemy's default pool is 5
plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing,
and a cap change adding two more inspects exhausts it, and every OTHER request
then waits for a connection.

Split so the database work finishes before the broker work starts:

- `lane_settings(session)` reads the caps and the oldest running task, then
  the session closes. `lane_view(settings)` does the inspect with none held.
- `store_lane_cap(session, …)` validates and commits, then the session closes.
  `push_lane_cap(lane, …)` does the live push with none held.

And a second finding while measuring it: **raising a cap now costs no broker
round trip at all.** The first cut only knew on/off, so it inspected on every
raise to find out whether the pool needed lowering — the control meant to be
instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap
so the push knows the direction; only a lowering needs to say anything.

The guard is structural, not timed: `lane_view` and `push_lane_cap` must not
ACCEPT a session. A timing test would be flaky, and a call-order test would
pass against a version that took the session and merely used it early.

`/api/system/health` has the same shape and is NOT fixed here — it is
rate-limited by `refresh_if_stale` so it does not inspect on every request.
Worth doing, separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/api/workers.py

@@ -23,7 +23,13 @@ from datetime import UTC, datetime
 from quart import Blueprint, jsonify, request
 
 from ..extensions import get_session
-from ..services.worker_control import LaneUpdateRefused, lane_view, set_lane
+from ..services.worker_control import (
+    LaneUpdateRefused,
+    lane_settings,
+    lane_view,
+    push_lane_cap,
+    store_lane_cap,
+)
 from ..services.worker_lanes import LANES_BY_NAME
 from ._responses import error_response as _bad
 
@@ -41,8 +47,14 @@ async def list_lanes():
     reply would show them the value from before their own change and read as
     the control having failed.
     """
+    # The session closes BEFORE the broker work. Holding a Postgres connection
+    # across a celery inspect is what made this page block the whole site —
+    # see `worker_control.LaneSettings`. This endpoint polls every 15s and the
+    # inspect budget is 11s, so each poll was pinning a connection for most of
+    # the interval.
     async with get_session() as session:
-        lanes = await lane_view(session)
+        settings = await lane_settings(session)
+    lanes = await lane_view(settings)
     return jsonify({
         "lanes": lanes,
         "fetched_at": datetime.now(UTC).isoformat(),
@@ -85,9 +97,13 @@ async def update_lane(name: str):
     if not isinstance(value, int) or isinstance(value, bool):
         return _bad("invalid_body", detail="slots_cap must be an integer")
 
+    # Store, close the session, THEN push. Same reason as the GET above, and
+    # more sharply here: a cap change could do three broker round trips, all
+    # of them previously with a connection held.
     async with get_session() as session:
         try:
-            result = await set_lane(session, lane, slots_cap=value)
+            was_cap = await store_lane_cap(session, lane, value)
         except LaneUpdateRefused as exc:
             return _bad("refused", detail=str(exc))
+    result = await push_lane_cap(lane, value, was_cap=was_cap)
     return jsonify(result)