feat: a saturated lane can grow itself, within the cap the operator set (4297)

Milestone 422 step 7 — the one sweep in this milestone that decides rather
than obeys, so it is off until a lane is opted in, bounded by the operator's
cap, floored at the operator's value, and it reports every decision including
the ones where it did nothing.

Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means
celery is about to pick those up and growing would add idle children (#1253
is that bug in the GPU agent); saturation alone means the lane is busy with
exactly as much work as exists. The backlog is depth PLUS reserved, because
celery prefetches and LLEN reads 0 while a worker holds thirty tasks in
memory — the case an LLEN-only autoscaler misses entirely, and the reason
step 2 plumbed `reserved` through.

The two sweeps had to be taught not to fight. The reconcile drives every
lane to its stored slots every five minutes, which would have reverted each
grow on the next tick: grow, revert, grow, revert, forever. For an
autoscaling lane the stored value is now a FLOOR — restored when a lane
falls below it, never taken back above it.

The operator's "a task that runs for x concurrent time" idea stays a UI
warning rather than a trigger: a long task does not finish sooner because
the lane gained a slot, so scaling on it would spend memory to change
nothing. Read from `task_run` on our own wall clock, not celery's
`time_start`, which is the WORKER's monotonic clock and would produce a
duration that is meaningless in the direction that matters — plausible.

Caught while reading it back: the first version read the stored slots as the
CURRENT pool. The autoscaler never writes that row, so every tick would have
proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica
already past the target is issued no message and reports success), and
capping the lane one slot above its floor forever while claiming otherwise.
It now reads the live pool and keeps the stored value purely as the floor,
and the tests fix the two to different numbers so an equal-fixture pass
cannot hide it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/tasks/maintenance.py

@@ -1387,14 +1387,69 @@ def reconcile_worker_lanes() -> dict:
     from ..models import WorkerLane
     from ..services.worker_control import reconcile_lanes_sync
 
+    # Both dicts built INSIDE the session. Reading a column off a detached
+    # instance happens to work while the attribute is still loaded and stops
+    # working the moment anything expires it — a failure that would appear
+    # long after this line, in a sweep nobody is watching.
     with _sync_session_factory()() as session:
-        desired = {
-            row.name: (row.slots, row.enabled)
-            for row in session.execute(select(WorkerLane)).scalars()
-        }
+        rows = list(session.execute(select(WorkerLane)).scalars())
+        desired = {row.name: (row.slots, row.enabled) for row in rows}
+        # Lanes the autoscaler may move. For these the stored value is a
+        # FLOOR: this sweep restores a lane that fell below it and never takes
+        # back what the autoscaler added, or the two would fight every five
+        # minutes.
+        autoscaling = frozenset(row.name for row in rows if row.autoscale)
     if not desired:
         # Migration 0103 seeds these, so an empty table means it has not run
         # yet. Nothing to assert — and inventing defaults here would let this
         # task disagree with the seed it is supposed to be enforcing.
         return {"changed": [], "skipped": [], "failed": {}}
-    return reconcile_lanes_sync(desired)
+    return reconcile_lanes_sync(desired, autoscaling)
+
+
+@celery.task(name="backend.app.tasks.maintenance.autoscale_worker_lanes")
+def autoscale_worker_lanes() -> dict:
+    """Grow a saturated lane, within the cap the operator set.
+
+    Milestone 422 step 7, and the only sweep in this milestone that decides
+    rather than obeys. Everything else applies what someone pressed.
+
+    OFF unless a lane opts in. A no-op costs one `celery inspect` and one LLEN
+    sweep and sends no control messages — the same fixed point the reconcile
+    holds, and for the same reason: this runs forever, so a settled system has
+    to be silent or the real signal drowns in its own heartbeat.
+
+    Returns every lane's decision INCLUDING the ones it held, each with a
+    reason. An autoscaler that only speaks when it acts cannot be debugged on
+    the day it does not.
+    """
+    from ..models import WorkerLane
+    from ..services.worker_control import autoscale_lanes_sync
+
+    with _sync_session_factory()() as session:
+        lanes = {
+            # The stored `slots` is passed ONLY as the floor. What the
+            # autoscaler moves is the live pool, which it reads itself — this
+            # row is never written, so using it as the current value would pin
+            # every decision to the same number forever.
+            row.name: (row.slots_cap, row.slots, True)
+            for row in session.execute(select(WorkerLane)).scalars()
+            if row.autoscale
+        }
+    if not lanes:
+        return {"decisions": []}
+
+    decisions = autoscale_lanes_sync(lanes)
+    for d in decisions:
+        if d.action != "held":
+            log.info(
+                "autoscale: %s %s to %s slots — %s",
+                d.lane, d.action, d.slots, d.reason,
+            )
+    return {
+        "decisions": [
+            {"lane": d.lane, "action": d.action, "slots": d.slots,
+             "reason": d.reason}
+            for d in decisions
+        ],
+    }