feat: a saturated lane can grow itself, within the cap the operator set (4297)

Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-22 10:05:43 -04:00
parent 5ca1058fb5
commit a01165365b
11 changed files with 754 additions and 19 deletions
@@ -0,0 +1,51 @@
+"""worker_lane.autoscale — may this lane grow itself?
+
+Milestone 422 step 7. One boolean, defaulting FALSE on every existing row and
+on every new one.
+
+## Why the default is false and not "sensible"
+
+This is the only part of the milestone that acts without anyone watching. The
+manual dial (step 4) and the reconcile (step 3) both do exactly what someone
+asked for; this one decides. Shipping it on would mean every install starts
+with a process that changes its own resource usage based on a heuristic tuned
+against nobody's workload.
+
+Off also makes the failure mode benign: if the signal is wrong, nothing
+happens until an operator opts a lane in, and they opted in while watching.
+
+## Why per lane and not one global switch
+
+The lanes are not alike in what a slot costs. A `worker` slot is a process;
+an `ml` slot is another copy of a ~3.5GB model. A global switch would enable
+growth on a lane whose behaviour under load nobody has observed, and the one
+it would hurt most is the one whose cost is least visible.
+
+Revision ID: 0104
+Revises: 0103
+Create Date: 2026-09-22
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0104"
+down_revision: Union[str, None] = "0103"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "worker_lane",
+        sa.Column(
+            "autoscale", sa.Boolean(),
+            server_default=sa.text("false"), nullable=False,
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("worker_lane", "autoscale")