bvandeusenandClaude Opus 5
445164c852
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 24s
CI and images / integration (push) Failing after 24s
CI and images / backend-lint-and-test (push) Failing after 34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle
instances quiet down when not running. the number that is visible and
something the user can tweak and manage should be the cap itself the number of
running workers is handled by the autoscaling function which is always on."*
They are right, and the reason it was not built this way is worth stating: the
manual dial came first (steps 2-4) and the autoscaler came last (step 7), as
an opt-in BESIDE a control that already existed. Nothing ever asked whether
the dial should still exist once something could move it automatically. Each
step was defensible; the result was three operator settings over one number.
## `slots`, `enabled` and `autoscale` are gone
`slots` was a MEASUREMENT wearing a preference's clothes. How many workers a
lane runs is read live and moved every minute; storing it meant the operator
had to keep two numbers in agreement and the autoscaler had to be told it was
allowed to touch one of them.
`autoscale` gated the mechanism behind a choice, so a lane nobody opted in
never gave its workers back — which is why an idle instance never quieted
down.
`enabled` is derived: a cap of zero means no consumers. "Off" and "may use no
workers" were two spellings of one fact, stored separately, free to disagree.
## Two sweeps become one
`reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_
sync` moved it away from that same number; and most of step 7's hardest
reasoning — a stored value that is a FLOOR, a target of `max(stored, current)`
— existed only to stop them fighting. Delete the stored number and the problem
is not solved, it is absent.
`size_lanes_sync` runs every minute and owns both consumers and pool size. It
also subsumes what the reconcile was for: a worker restarted at its ENV
concurrency is corrected on the next tick rather than after five.
Growth is immediate, shrink is one worker per tick. Deliberately asymmetric —
"always on" is only pleasant if the ramp keeps up, and +1/minute would take
four minutes to answer a burst. Being one worker too large for a minute costs
a sleeping process; being too small costs work not happening. For ML the
asymmetry matters most: every new slot reloads a multi-GB model, so the slow
shrink is what stops a quiet patch from paying that cost again a minute later.
## The caps ship at one, and zero for ML
Per the operator. Conservative on purpose — and a conservative default nobody
knows how to raise is just a slow product, which is the other half of what
they asked for:
"there needs to be something that tells the user to bump those numbers to
improve processing rate or they'd never know the controls exist."
So a lane running everything its cap allows while work piles up says so, in
its own row, with the headroom named: *"4,060 waiting and all 1 worker busy.
Raise the cap to run more at once — this machine allows up to 7."*
It fires only when raising the cap would actually help. Not when the lane is
keeping up, not when the sizing pass has room it has not taken, and not at the
machine ceiling — where "raise the cap" is advice nobody can take.
## Migration 0105 rewrites the caps rather than carrying them
The old defaults (4/2/2/1) bounded a manual control and were loose because
moving within them was the ordinary act. The number now means "the most
workers this lane may use", which is a different promise; carrying the old
figure over would quadruple the worker lane on every existing install at the
moment this deploys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
124 lines
5.0 KiB
Python
124 lines
5.0 KiB
Python
"""worker_lane — one number: the cap. `slots`, `enabled` and `autoscale` go.
|
|
|
|
Milestone 422, reshaped by the operator 2026-09-23:
|
|
|
|
"auto should be always on, not a setting, so that idle instances quiet
|
|
down when not running. the number that is visible and something the user
|
|
can tweak and manage should be the cap itself the number of running
|
|
workers is handled by the autoscaling function which is always on."
|
|
|
|
## What each dropped column was, and why it is not needed
|
|
|
|
**`slots`** — how many workers the lane should run. That is a MEASUREMENT,
|
|
not a preference: the autoscaler moves the live pool between one and the cap
|
|
according to the backlog, and reads it back from the worker every minute.
|
|
Storing it made it look like something to keep in agreement with the cap,
|
|
which is exactly what the operator had to do.
|
|
|
|
**`autoscale`** — whether the lane was allowed to size itself. It gated the
|
|
mechanism behind a per-lane opt-in, so a lane nobody enabled simply never
|
|
gave its slots back. Always on now, which is the only way "idle instances
|
|
quiet down" can be true of an install nobody has configured.
|
|
|
|
**`enabled`** — whether the lane consumes its queues. Derived from `cap > 0`.
|
|
It and `slots = 0` were two spellings of one fact and were free to disagree;
|
|
this migration picks the one an operator can see.
|
|
|
|
## Why the caps are rewritten rather than preserved
|
|
|
|
The old defaults were 4 / 2 / 2 / 1, chosen when the number meant "the most
|
|
you may raise SLOTS to" — a bound on a manual control, deliberately loose
|
|
because moving within it was the ordinary act. The number now means "the most
|
|
workers this lane may actually use", which is a different promise, and
|
|
carrying the old figure over would silently quadruple the worker lane on
|
|
every existing install at the moment this deploys.
|
|
|
|
So every row is reset to the new defaults: **one for each required lane, zero
|
|
for ML.** That loses whatever an operator had set — which is the honest
|
|
trade, because what they set was an answer to a different question. The UI
|
|
now tells a busy lane's operator to raise its cap, which is how the number
|
|
gets back up on an install that needs it.
|
|
|
|
ML at zero also keeps rule 164's carve-out intact: no consumers, so no model
|
|
download until someone raises the cap.
|
|
|
|
Revision ID: 0105
|
|
Revises: 0104
|
|
Create Date: 2026-09-23
|
|
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0105"
|
|
down_revision: Union[str, None] = "0104"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
# (name, cap) — the same values `services/worker_lanes.LANES` declares. Seeded
|
|
# here as literals rather than imported: a migration must describe the schema
|
|
# at ITS point in history, and importing the live table would make this file
|
|
# change meaning every time that table does.
|
|
_CAPS = (
|
|
("worker", 1),
|
|
("scheduler", 1),
|
|
("maintenance_long", 1),
|
|
("ml", 0),
|
|
)
|
|
|
|
|
|
def upgrade() -> None:
|
|
# The constraint goes first: it names `slots`, so dropping the column out
|
|
# from under it fails on Postgres.
|
|
op.drop_constraint("ck_worker_lane_slots_within_cap", "worker_lane", type_="check")
|
|
op.drop_constraint(
|
|
"ck_worker_lane_slots_non_negative", "worker_lane", type_="check",
|
|
)
|
|
op.drop_column("worker_lane", "slots")
|
|
op.drop_column("worker_lane", "enabled")
|
|
op.drop_column("worker_lane", "autoscale")
|
|
|
|
# Reset to the new meaning. See the docstring: the old value answered a
|
|
# different question, and carrying it over would raise every lane.
|
|
for name, cap in _CAPS:
|
|
op.execute(
|
|
sa.text("UPDATE worker_lane SET slots_cap = :cap WHERE name = :name")
|
|
.bindparams(cap=cap, name=name)
|
|
)
|
|
|
|
# A lane the old seed never wrote — or one an operator added by hand — is
|
|
# left alone rather than guessed at. `_rows_by_name` creates any missing
|
|
# row at the lane's default on first read.
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.add_column(
|
|
"worker_lane",
|
|
sa.Column("slots", sa.Integer(), nullable=False, server_default="1"),
|
|
)
|
|
op.add_column(
|
|
"worker_lane",
|
|
sa.Column(
|
|
"enabled", sa.Boolean(), nullable=False, server_default=sa.text("true"),
|
|
),
|
|
)
|
|
op.add_column(
|
|
"worker_lane",
|
|
sa.Column(
|
|
"autoscale", sa.Boolean(), nullable=False, server_default=sa.text("false"),
|
|
),
|
|
)
|
|
# Restore the pre-0105 invariants. `slots` comes back as 1 everywhere and
|
|
# the caps are 1/1/1/0, so a lane at cap 0 would violate `slots <= cap` —
|
|
# hence the clamp before the constraint is added.
|
|
op.execute(sa.text("UPDATE worker_lane SET slots = 0 WHERE slots_cap = 0"))
|
|
op.execute(sa.text("UPDATE worker_lane SET enabled = (slots_cap > 0)"))
|
|
op.create_check_constraint(
|
|
op.f("ck_worker_lane_slots_non_negative"), "worker_lane", "slots >= 0",
|
|
)
|
|
op.create_check_constraint(
|
|
op.f("ck_worker_lane_slots_within_cap"), "worker_lane", "slots <= slots_cap",
|
|
)
|