feat: a saturated lane can grow itself, within the cap the operator set (4297)

Milestone 422 step 7 — the one sweep in this milestone that decides rather
than obeys, so it is off until a lane is opted in, bounded by the operator's
cap, floored at the operator's value, and it reports every decision including
the ones where it did nothing.

Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means
celery is about to pick those up and growing would add idle children (#1253
is that bug in the GPU agent); saturation alone means the lane is busy with
exactly as much work as exists. The backlog is depth PLUS reserved, because
celery prefetches and LLEN reads 0 while a worker holds thirty tasks in
memory — the case an LLEN-only autoscaler misses entirely, and the reason
step 2 plumbed `reserved` through.

The two sweeps had to be taught not to fight. The reconcile drives every
lane to its stored slots every five minutes, which would have reverted each
grow on the next tick: grow, revert, grow, revert, forever. For an
autoscaling lane the stored value is now a FLOOR — restored when a lane
falls below it, never taken back above it.

The operator's "a task that runs for x concurrent time" idea stays a UI
warning rather than a trigger: a long task does not finish sooner because
the lane gained a slot, so scaling on it would spend memory to change
nothing. Read from `task_run` on our own wall clock, not celery's
`time_start`, which is the WORKER's monotonic clock and would produce a
duration that is meaningless in the direction that matters — plausible.

Caught while reading it back: the first version read the stored slots as the
CURRENT pool. The autoscaler never writes that row, so every tick would have
proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica
already past the target is issued no message and reports success), and
capping the lane one slot above its floor forever while claiming otherwise.
It now reads the live pool and keeps the stored value purely as the floor,
and the tests fix the two to different numbers so an equal-fixture pass
cannot hide it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

frontend/src/components/settings/WorkerLanesCard.vue

@@ -37,6 +37,7 @@
             <th class="text-right">Pending</th>
             <th class="text-right">Busy</th>
             <th style="width: 200px;">Slots</th>
+            <th class="text-right">Auto</th>
             <th class="text-right">On</th>
           </tr>
         </thead>
@@ -62,6 +63,15 @@
               <div v-else-if="lane.live.replicas > 1" class="text-caption fc-muted">
                 {{ lane.live.replicas }} replicas
               </div>
+              <!-- The operator asked for a lane held at full for a long time
+                   to TRIGGER growth. It reports instead: extra slots do not
+                   make a long task finish sooner, so scaling on this would
+                   spend memory to change nothing. Seeing the lane is wedged
+                   on one slow job is the useful half, and it cannot mislead
+                   a lane into growing for the wrong reason. -->
+              <div v-if="laneStuckFor(lane)" class="text-caption text-warning">
+                all slots busy for {{ laneStuckFor(lane) }}
+              </div>
             </td>
 
             <td class="text-caption fc-muted">{{ lane.queues.join(', ') }}</td>
@@ -106,6 +116,31 @@
               </div>
             </td>
 
+            <!-- The autoscaler is the one control here that acts without
+                 being asked, so it is a separate switch from `On` rather than
+                 a mode folded into it, and it is off until someone opts this
+                 particular lane in.
+
+                 Turning it ON needs room to grow into; turning it OFF never
+                 does. Disabling the whole switch at cap == slots would strand
+                 an already-autoscaling lane as soon as someone raised its
+                 floor to the cap, with the control that would undo it greyed
+                 out. And the reason is on screen, for the same reason the
+                 ceiling justifies itself below: a greyed control with no
+                 explanation reads as a bug. -->
+            <td class="text-right">
+              <v-switch
+                :model-value="lane.autoscale"
+                density="compact" hide-details color="accent"
+                :disabled="busy === lane.name || (!lane.autoscale && !canGrow(lane))"
+                :aria-label="`Let ${lane.display_name} grow itself`"
+                @update:model-value="setAutoscale(lane, $event)"
+              />
+              <div v-if="!canGrow(lane)" class="text-caption fc-muted">
+                raise the cap
+              </div>
+            </td>
+
             <td class="text-right">
               <v-switch
                 :model-value="lane.enabled"
@@ -162,7 +197,7 @@
 <script setup>
 import { computed, ref } from 'vue'
 
-import { useSystemActivityStore } from '../../stores/systemActivity.js'
+import { laneStuckFor, useSystemActivityStore } from '../../stores/systemActivity.js'
 import { formatRelative } from '../../utils/date.js'
 import CardHeading from '../common/CardHeading.vue'
 
@@ -238,6 +273,17 @@ function step(lane, delta) {
 function toggle(lane, value) {
   return apply(lane, { enabled: Boolean(value) })
 }
+
+// Room to grow into. The autoscaler moves the LIVE pool, but the floor it
+// starts from is the stored value, so a lane whose floor already sits at its
+// cap has nowhere to go and enabling it would do nothing at all.
+function canGrow(lane) {
+  return lane.slots_cap > lane.slots
+}
+
+function setAutoscale(lane, value) {
+  return apply(lane, { autoscale: Boolean(value) })
+}
 </script>
 
 <style scoped>

frontend/src/stores/systemActivity.js

@@ -3,6 +3,29 @@ import { ref } from 'vue'
 
 import { useApi } from '../composables/useApi.js'
 
+// Both halves, never one (milestone 422 step 7). Saturation alone is a lane
+// doing its job; a long-running task alone may be the only thing on a lane
+// with slots to spare. Only together do they mean "nothing else on this lane
+// can start", which is the thing worth interrupting someone about.
+//
+// The operator asked for this to TRIGGER growth. It reports instead: a long
+// task does not finish sooner because the lane gained a slot, so autoscaling
+// on it would spend memory to change nothing. Exported rather than left in
+// the card so the AND is pinned by a test — a warning that fires on half the
+// condition is a warning people learn to ignore.
+export const LANE_STUCK_MINUTES = 15
+
+export function laneStuckFor(lane) {
+  const mins = lane?.oldest_running_minutes
+  if (mins === null || mins === undefined || mins < LANE_STUCK_MINUTES) return null
+  // Unknown is not busy: a lane nothing answered for has no live reading to
+  // call saturated, and `pool` of 0 or null is not a full pool.
+  if (!lane.live?.present || !lane.live.pool) return null
+  if (lane.live.active < lane.live.pool) return null
+  if (mins < 120) return `${mins} minutes`
+  return `${Math.floor(mins / 60)} hours`
+}
+
 export const useSystemActivityStore = defineStore('systemActivity', () => {
   const api = useApi()
 

frontend/test/workerLanes.spec.js

@@ -1,6 +1,6 @@
 import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
 import { setActivePinia, createPinia } from 'pinia'
-import { useSystemActivityStore } from '../src/stores/systemActivity.js'
+import { laneStuckFor, useSystemActivityStore } from '../src/stores/systemActivity.js'
 
 // Milestone 422 step 4. Covers the store half of the worker-lane dial — the
 // part that decides what the card can tell the operator.
@@ -139,3 +139,54 @@ describe('worker lanes store', () => {
     expect(err.body.detail).toContain('container can hold')
   })
 })
+
+// --- the long-running-task warning (step 7) ----------------------------------
+//
+// The operator asked for "a task runs for x concurrent time" to TRIGGER
+// growth. It became a warning: extra slots do not make a running task finish
+// sooner. What is pinned here is the AND — a warning that fires on half its
+// condition is one people learn to ignore, and at that point it is worse than
+// not having it.
+
+describe('laneStuckFor', () => {
+  const lane = (over = {}) => ({
+    oldest_running_minutes: 40,
+    live: { present: true, pool: 4, active: 4 },
+    ...over,
+  })
+
+  it('warns when every slot is busy and the oldest task is old', () => {
+    expect(laneStuckFor(lane())).toBe('40 minutes')
+  })
+
+  it('says nothing when the lane has a free slot', () => {
+    expect(laneStuckFor(lane({ live: { present: true, pool: 4, active: 3 } })))
+      .toBeNull()
+  })
+
+  it('says nothing for a saturated lane whose tasks are young', () => {
+    expect(laneStuckFor(lane({ oldest_running_minutes: 2 }))).toBeNull()
+  })
+
+  it('says nothing when nothing is running', () => {
+    expect(laneStuckFor(lane({ oldest_running_minutes: null }))).toBeNull()
+  })
+
+  it('says nothing about a lane that is not answering', () => {
+    // Unknown is not busy. A lane nothing answered for has no live reading to
+    // call saturated, and asserting one would be a verdict from an unswept
+    // read — the same distinction `present` exists for everywhere else here.
+    expect(laneStuckFor(lane({ live: { present: false, pool: null, active: 0 } })))
+      .toBeNull()
+  })
+
+  it('says nothing when the pool is zero', () => {
+    // A lane sized to zero is not "fully busy at zero" — it is off.
+    expect(laneStuckFor(lane({ live: { present: true, pool: 0, active: 0 } })))
+      .toBeNull()
+  })
+
+  it('switches to hours once minutes stop being readable', () => {
+    expect(laneStuckFor(lane({ oldest_running_minutes: 195 }))).toBe('3 hours')
+  })
+})