Files
FabledCurator/backend/app/models/worker_lane.py
T
bvandeusenandClaude Opus 5 a01165365b
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 36s
Build images / build-ml (push) Successful in 1m55s
Build images / build-web (push) Successful in 1m54s
CI / integration (push) Successful in 2m16s
Build images / smoke-web (push) Failing after 7m48s
Build images / promote (push) Skipped
feat: a saturated lane can grow itself, within the cap the operator set (4297)
Milestone 422 step 7 — the one sweep in this milestone that decides rather
than obeys, so it is off until a lane is opted in, bounded by the operator's
cap, floored at the operator's value, and it reports every decision including
the ones where it did nothing.

Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means
celery is about to pick those up and growing would add idle children (#1253
is that bug in the GPU agent); saturation alone means the lane is busy with
exactly as much work as exists. The backlog is depth PLUS reserved, because
celery prefetches and LLEN reads 0 while a worker holds thirty tasks in
memory — the case an LLEN-only autoscaler misses entirely, and the reason
step 2 plumbed `reserved` through.

The two sweeps had to be taught not to fight. The reconcile drives every
lane to its stored slots every five minutes, which would have reverted each
grow on the next tick: grow, revert, grow, revert, forever. For an
autoscaling lane the stored value is now a FLOOR — restored when a lane
falls below it, never taken back above it.

The operator's "a task that runs for x concurrent time" idea stays a UI
warning rather than a trigger: a long task does not finish sooner because
the lane gained a slot, so scaling on it would spend memory to change
nothing. Read from `task_run` on our own wall clock, not celery's
`time_start`, which is the WORKER's monotonic clock and would produce a
duration that is meaningless in the direction that matters — plausible.

Caught while reading it back: the first version read the stored slots as the
CURRENT pool. The autoscaler never writes that row, so every tick would have
proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica
already past the target is issued no message and reports success), and
capping the lane one slot above its floor forever while claiming otherwise.
It now reads the live pool and keeps the stored value purely as the floor,
and the tests fix the two to different numbers so an equal-fixture pass
cannot hide it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-22 10:05:43 -04:00

99 lines
4.0 KiB
Python

"""worker_lane — how many slots the operator wants each worker lane to have.
Milestone 422 step 1. One row per lane in `services/worker_lanes.LANES`.
## What is NOT in here
The lane's queues and its display name. Those are decided by `celery_app.py`'s
`task_routes` — an operator cannot move a backup off `maintenance_long` — so
storing them would be a row that can contradict the routing table, with
nothing to notice until a queue had no consumer. They live in
`services/worker_lanes.py`; this table holds only what an operator may change.
The DERIVED CEILING is also absent, and that is deliberate rather than an
omission. It is computed from the container's cgroup limits on every read, so
a row written on a 32GB host and later run in a 4GB container is bounded by
the 4GB — a stored ceiling would quietly authorise what the box can no longer
hold.
## The three numbers
slots <= slots_cap <= derived_ceiling
(live) (this row) (computed)
Operator's distinction, 2026-09-22: the derived value is *a cap on the cap*.
`slots_cap` is theirs and is always lowerable; it simply may not exceed what
the container can hold. The CHECK constraint below enforces the left half,
which is a fact about the row; the right half is enforced at write, because
it depends on a value no database column holds.
## enabled
Whether the lane consumes its queues at all. This is how ML ships off
(milestone 422 step 6): `enabled=false` with `slots=0`, so a fresh install
never loads a model or reaches HuggingFace, and turning tagging on in
Settings is what triggers the fetch.
Not a substitute for `slots=0`. A lane can be enabled with zero slots while
it is being resized, and the two answer different questions: `enabled` is
intent, `slots` is capacity.
"""
from datetime import datetime
from sqlalchemy import (
Boolean,
CheckConstraint,
DateTime,
Integer,
String,
func,
)
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class WorkerLane(Base):
__tablename__ = "worker_lane"
__table_args__ = (
# Bare names — Base.metadata's naming convention prepends
# ck_worker_lane_. Pre-prefixing here doubles it, which is what
# alembic 0088 had to rename four constraints for (#3275).
CheckConstraint("slots >= 0", name="slots_non_negative"),
CheckConstraint("slots_cap >= 0", name="cap_non_negative"),
# The invariant that makes the cap mean anything. Enforced in the
# database rather than only in the service, because a row that
# violates it is not a rejected request — it is a lane that will be
# reconciled UP to a value the operator capped.
CheckConstraint("slots <= slots_cap", name="slots_within_cap"),
)
# The lane name from services/worker_lanes.LANES — never a container
# hostname. See models/service_seen.py for why: celery's worker names here
# are `celery@<container id>`, minted fresh on every deploy.
name: Mapped[str] = mapped_column(String(32), primary_key=True)
slots: Mapped[int] = mapped_column(Integer, nullable=False)
slots_cap: Mapped[int] = mapped_column(Integer, nullable=False)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False)
# Whether the autoscaler may raise this lane's slots on its own, up to
# slots_cap and never past it (milestone 422 step 7).
#
# OFF by default and per-lane rather than global. It is the one part of
# this milestone that acts unattended, so it is opt-in per lane the
# operator has actually thought about — a global switch would turn it on
# for lanes whose behaviour under load nobody has watched, including `ml`,
# where every extra slot is another copy of a multi-GB model.
autoscale: Mapped[bool] = mapped_column(
Boolean, nullable=False, server_default="false",
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
onupdate=func.now(),
)