Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 36s
Build images / build-ml (push) Successful in 1m55s
Build images / build-web (push) Successful in 1m54s
CI / integration (push) Successful in 2m16s
Build images / smoke-web (push) Failing after 7m48s
Build images / promote (push) Skipped
Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
99 lines
4.0 KiB
Python
99 lines
4.0 KiB
Python
"""worker_lane — how many slots the operator wants each worker lane to have.
|
|
|
|
Milestone 422 step 1. One row per lane in `services/worker_lanes.LANES`.
|
|
|
|
## What is NOT in here
|
|
|
|
The lane's queues and its display name. Those are decided by `celery_app.py`'s
|
|
`task_routes` — an operator cannot move a backup off `maintenance_long` — so
|
|
storing them would be a row that can contradict the routing table, with
|
|
nothing to notice until a queue had no consumer. They live in
|
|
`services/worker_lanes.py`; this table holds only what an operator may change.
|
|
|
|
The DERIVED CEILING is also absent, and that is deliberate rather than an
|
|
omission. It is computed from the container's cgroup limits on every read, so
|
|
a row written on a 32GB host and later run in a 4GB container is bounded by
|
|
the 4GB — a stored ceiling would quietly authorise what the box can no longer
|
|
hold.
|
|
|
|
## The three numbers
|
|
|
|
slots <= slots_cap <= derived_ceiling
|
|
(live) (this row) (computed)
|
|
|
|
Operator's distinction, 2026-09-22: the derived value is *a cap on the cap*.
|
|
`slots_cap` is theirs and is always lowerable; it simply may not exceed what
|
|
the container can hold. The CHECK constraint below enforces the left half,
|
|
which is a fact about the row; the right half is enforced at write, because
|
|
it depends on a value no database column holds.
|
|
|
|
## enabled
|
|
|
|
Whether the lane consumes its queues at all. This is how ML ships off
|
|
(milestone 422 step 6): `enabled=false` with `slots=0`, so a fresh install
|
|
never loads a model or reaches HuggingFace, and turning tagging on in
|
|
Settings is what triggers the fetch.
|
|
|
|
Not a substitute for `slots=0`. A lane can be enabled with zero slots while
|
|
it is being resized, and the two answer different questions: `enabled` is
|
|
intent, `slots` is capacity.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
Boolean,
|
|
CheckConstraint,
|
|
DateTime,
|
|
Integer,
|
|
String,
|
|
func,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class WorkerLane(Base):
|
|
__tablename__ = "worker_lane"
|
|
__table_args__ = (
|
|
# Bare names — Base.metadata's naming convention prepends
|
|
# ck_worker_lane_. Pre-prefixing here doubles it, which is what
|
|
# alembic 0088 had to rename four constraints for (#3275).
|
|
CheckConstraint("slots >= 0", name="slots_non_negative"),
|
|
CheckConstraint("slots_cap >= 0", name="cap_non_negative"),
|
|
# The invariant that makes the cap mean anything. Enforced in the
|
|
# database rather than only in the service, because a row that
|
|
# violates it is not a rejected request — it is a lane that will be
|
|
# reconciled UP to a value the operator capped.
|
|
CheckConstraint("slots <= slots_cap", name="slots_within_cap"),
|
|
)
|
|
|
|
# The lane name from services/worker_lanes.LANES — never a container
|
|
# hostname. See models/service_seen.py for why: celery's worker names here
|
|
# are `celery@<container id>`, minted fresh on every deploy.
|
|
name: Mapped[str] = mapped_column(String(32), primary_key=True)
|
|
|
|
slots: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
slots_cap: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False)
|
|
|
|
# Whether the autoscaler may raise this lane's slots on its own, up to
|
|
# slots_cap and never past it (milestone 422 step 7).
|
|
#
|
|
# OFF by default and per-lane rather than global. It is the one part of
|
|
# this milestone that acts unattended, so it is opt-in per lane the
|
|
# operator has actually thought about — a global switch would turn it on
|
|
# for lanes whose behaviour under load nobody has watched, including `ml`,
|
|
# where every extra slot is another copy of a multi-GB model.
|
|
autoscale: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, server_default="false",
|
|
)
|
|
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=func.now(),
|
|
onupdate=func.now(),
|
|
)
|