feat: one number per lane — the cap — and the autoscaler is the mechanism (4295)

Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle
instances quiet down when not running. the number that is visible and
something the user can tweak and manage should be the cap itself the number of
running workers is handled by the autoscaling function which is always on."*

They are right, and the reason it was not built this way is worth stating: the
manual dial came first (steps 2-4) and the autoscaler came last (step 7), as
an opt-in BESIDE a control that already existed. Nothing ever asked whether
the dial should still exist once something could move it automatically. Each
step was defensible; the result was three operator settings over one number.

## `slots`, `enabled` and `autoscale` are gone

`slots` was a MEASUREMENT wearing a preference's clothes. How many workers a
lane runs is read live and moved every minute; storing it meant the operator
had to keep two numbers in agreement and the autoscaler had to be told it was
allowed to touch one of them.

`autoscale` gated the mechanism behind a choice, so a lane nobody opted in
never gave its workers back — which is why an idle instance never quieted
down.

`enabled` is derived: a cap of zero means no consumers. "Off" and "may use no
workers" were two spellings of one fact, stored separately, free to disagree.

## Two sweeps become one

`reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_
sync` moved it away from that same number; and most of step 7's hardest
reasoning — a stored value that is a FLOOR, a target of `max(stored, current)`
— existed only to stop them fighting. Delete the stored number and the problem
is not solved, it is absent.

`size_lanes_sync` runs every minute and owns both consumers and pool size. It
also subsumes what the reconcile was for: a worker restarted at its ENV
concurrency is corrected on the next tick rather than after five.

Growth is immediate, shrink is one worker per tick. Deliberately asymmetric —
"always on" is only pleasant if the ramp keeps up, and +1/minute would take
four minutes to answer a burst. Being one worker too large for a minute costs
a sleeping process; being too small costs work not happening. For ML the
asymmetry matters most: every new slot reloads a multi-GB model, so the slow
shrink is what stops a quiet patch from paying that cost again a minute later.

## The caps ship at one, and zero for ML

Per the operator. Conservative on purpose — and a conservative default nobody
knows how to raise is just a slow product, which is the other half of what
they asked for:

    "there needs to be something that tells the user to bump those numbers to
     improve processing rate or they'd never know the controls exist."

So a lane running everything its cap allows while work piles up says so, in
its own row, with the headroom named: *"4,060 waiting and all 1 worker busy.
Raise the cap to run more at once — this machine allows up to 7."*

It fires only when raising the cap would actually help. Not when the lane is
keeping up, not when the sizing pass has room it has not taken, and not at the
machine ceiling — where "raise the cap" is advice nobody can take.

## Migration 0105 rewrites the caps rather than carrying them

The old defaults (4/2/2/1) bounded a manual control and were loose because
moving within them was the ordinary act. The number now means "the most
workers this lane may use", which is a different promise; carrying the old
figure over would quadruple the worker lane on every existing install at the
moment this deploys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/tasks/maintenance.py

@@ -1349,107 +1349,55 @@ def sync_memberships() -> str:
         parts.append(f"suggested={res['suggested']}")
     return " ".join(parts) or "no platforms"
 
+@celery.task(name="backend.app.tasks.maintenance.size_worker_lanes")
+def size_worker_lanes() -> dict:
+    """Size every lane to its backlog, within the cap the operator set.
 
-@celery.task(name="backend.app.tasks.maintenance.reconcile_worker_lanes")
-def reconcile_worker_lanes() -> dict:
-    """Drive every running lane back to the slots the operator set.
+    ONE sweep, replacing `reconcile_worker_lanes` and `autoscale_worker_lanes`
+    (2026-09-23). They were two enforcers over one number: the reconcile drove
+    the pool to a stored `slots`, the autoscaler moved it away from that same
+    value, and most of the autoscaler's design existed to keep the reconcile
+    from undoing its work. Deleting the stored number deletes the conflict.
 
-    Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by
-    its supervisor comes back at its ENV concurrency, silently below whatever
-    was configured, and nothing on step 2's write path would ever notice.
+    It still does what the reconcile existed for. `pool_grow` is not durable —
+    a worker restarted by its supervisor comes back at its ENV concurrency,
+    silently below what the lane should run — and this reads the LIVE pool
+    every minute, so that worker is corrected on the next tick rather than
+    after five.
 
-    ## Why a beat task and not a hook in web
-
-    Step 3 was written as "web applies the stored values after it starts". It
-    cannot, and `services/service_roster.py` already records why: hypercorn
-    runs `--workers 4`, so anything in `before_serving` becomes FOUR
-    concurrent loops per container, all hammering the broker forever.
-
-    The other option was service_roster's own answer — refresh on demand from
-    whichever request happens to arrive. Rejected here because the two are
-    solving different problems. A stale ROSTER only misleads someone who is
-    looking at it, so recomputing it when they look is exactly right. A lane
-    running at the wrong size is doing less work than it was told to whether
-    or not anyone is watching — and the case that matters is a deploy at 3am
-    followed by a backlog nobody is awake to notice.
-
-    So: unattended, on the quick `maintenance` lane (its module routes it
-    there), beside the other recovery sweeps. The accepted cost is that a dead
-    scheduler stops reconciliation — but a dead scheduler already stops every
-    other sweep, and the roster reports it, so this adds no new blind spot.
-
-    ## Quiet when settled
-
-    One broker round trip per tick, and no control messages at all once every
-    lane matches. Returns the lanes it actually moved, so the log shows a
-    correction rather than a heartbeat.
+    Returns every lane's outcome INCLUDING the ones it held, each with a
+    reason. A pass that only speaks when it acts cannot be debugged on the day
+    it does not.
     """
     from ..models import WorkerLane
-    from ..services.worker_control import reconcile_lanes_sync
+    from ..services.worker_control import size_lanes_sync
 
-    # Both dicts built INSIDE the session. Reading a column off a detached
-    # instance happens to work while the attribute is still loaded and stops
-    # working the moment anything expires it — a failure that would appear
-    # long after this line, in a sweep nobody is watching.
+    # Read INSIDE the session. Reading a column off a detached instance
+    # happens to work while the attribute is still loaded and stops working
+    # the moment anything expires it — a failure that would appear long after
+    # this line, in a sweep nobody is watching.
     with _sync_session_factory()() as session:
-        rows = list(session.execute(select(WorkerLane)).scalars())
-        desired = {row.name: (row.slots, row.enabled) for row in rows}
-        # Lanes the autoscaler may move. For these the stored value is a
-        # FLOOR: this sweep restores a lane that fell below it and never takes
-        # back what the autoscaler added, or the two would fight every five
-        # minutes.
-        autoscaling = frozenset(row.name for row in rows if row.autoscale)
-    if not desired:
-        # Migration 0103 seeds these, so an empty table means it has not run
-        # yet. Nothing to assert — and inventing defaults here would let this
-        # task disagree with the seed it is supposed to be enforcing.
-        return {"changed": [], "skipped": [], "failed": {}}
-    return reconcile_lanes_sync(desired, autoscaling)
-
-
-@celery.task(name="backend.app.tasks.maintenance.autoscale_worker_lanes")
-def autoscale_worker_lanes() -> dict:
-    """Grow a saturated lane, within the cap the operator set.
-
-    Milestone 422 step 7, and the only sweep in this milestone that decides
-    rather than obeys. Everything else applies what someone pressed.
-
-    OFF unless a lane opts in. A no-op costs one `celery inspect` and one LLEN
-    sweep and sends no control messages — the same fixed point the reconcile
-    holds, and for the same reason: this runs forever, so a settled system has
-    to be silent or the real signal drowns in its own heartbeat.
-
-    Returns every lane's decision INCLUDING the ones it held, each with a
-    reason. An autoscaler that only speaks when it acts cannot be debugged on
-    the day it does not.
-    """
-    from ..models import WorkerLane
-    from ..services.worker_control import autoscale_lanes_sync
-
-    with _sync_session_factory()() as session:
-        lanes = {
-            # The stored `slots` is passed ONLY as the floor. What the
-            # autoscaler moves is the live pool, which it reads itself — this
-            # row is never written, so using it as the current value would pin
-            # every decision to the same number forever.
-            row.name: (row.slots_cap, row.slots, True)
+        caps = {
+            row.name: row.slots_cap
             for row in session.execute(select(WorkerLane)).scalars()
-            if row.autoscale
         }
-    if not lanes:
-        return {"decisions": []}
+    if not caps:
+        # Migration 0103/0105 seed these, so an empty table means they have
+        # not run yet. Nothing to assert — and inventing defaults here would
+        # let this task disagree with the seed it is meant to be enforcing.
+        return {"sized": []}
 
-    decisions = autoscale_lanes_sync(lanes)
-    for d in decisions:
-        if d.action != "held":
+    sized = size_lanes_sync(caps)
+    for d in sized:
+        if d.action not in ("held", "skipped"):
             log.info(
-                "autoscale: %s %s to %s slots — %s",
+                "worker lanes: %s %s to %s slots — %s",
                 d.lane, d.action, d.slots, d.reason,
             )
     return {
-        "decisions": [
+        "sized": [
             {"lane": d.lane, "action": d.action, "slots": d.slots,
              "reason": d.reason}
-            for d in decisions
+            for d in sized
         ],
     }