bvandeusenandClaude Opus 5
84f13135ce
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Failing after 32s
Build images / build-web (push) Successful in 58s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m45s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m13s
Milestone 422 step 1. The data model the rest of the milestone reads. No
behaviour change: nothing consumes these rows yet, and every lane still boots
at its CELERY_CONCURRENCY env value.
Three numbers, not two, per the operator's distinction — the derived value is
a cap ON the cap:
slots <= slots_cap <= derived_ceiling
(live) (operator) (computed)
They can always lower their own cap; they cannot raise it past what the
container can hold. The ceiling is never stored, so a row written on a 32GB
host and later run in a 4GB container is bounded by the 4GB.
`services/worker_lanes.py` is the one place that knows the lane set.
`models/worker_lane.py` holds only what an operator may change.
Two deviations from the step as written, both deliberate:
QUEUES ARE NOT A COLUMN. The step body said the row carries its `-Q` list,
but a lane's queues are decided by celery_app's task_routes, not by
preference — an operator cannot move a backup off maintenance_long. Storing
them would create a row that can contradict the routing table, with nothing
to notice until a queue had no consumer. So queues are code, slots are data.
`test_every_routed_queue_has_a_lane_that_serves_it` reads the real routing
table and fails if a route is ever added without a lane.
ROLE_NAMES IS NOW DERIVED, not left alone. It was a hand-kept second copy of
"queue set -> display name" and had already drifted: maintenance_long is a
live lane with four task routes and a dedicated worker in the operator's
stack, and the roster did not know its name — so the System tab labelled it
`Worker (maintenance_long)`. Adding a lane table beside it would have made
three copies.
The ceiling honours cgroup limits rather than the host's. `os.cpu_count()`
reports the HOST's cores from inside a container, so a 4-core quota on a
32-core host would otherwise offer 32 slots — and the operator's own stack
sets `cpus: '4.0'` on ml-worker, so that is real configuration, not a
hypothetical. Memory reads cgroup v2 then v1, and recognises v1's
PAGE_SIZE-aligned LONG_MAX sentinel by magnitude rather than treating it as
petabytes.
Every uncertain case fails LOW. An unreadable limit yields UNKNOWN_CEILING,
never unlimited — not knowing how much memory there is must not read as
plenty. A box too small to hold one model beside the web process gets an ML
ceiling of 0 rather than a floor of 1: offering a slot that OOMs the
container the first time it is used is exactly what this exists to prevent.
ML_BYTES_PER_SLOT is 4 GiB and is UNMEASURED — flagged as such in the code,
with the method for replacing it with a real figure. It decides whether a
stranger's server survives enabling tagging, so it errs toward refusing a
slot that would have fitted.
Seeded one-of-each with ml at 0 and disabled (alembic 0103). ML off is step
6's requirement arriving early: enabling the lane is what triggers the SigLIP
download, and rule 164 permits a runtime fetch only for a feature that is
optional and clearly off. The seed values are literals rather than an import
of LANES — a migration is a statement about one moment, and importing the
live defaults would silently change what this revision does on a fresh
database in 2027.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
87 lines
3.4 KiB
Python
87 lines
3.4 KiB
Python
"""worker_lane — how many slots the operator wants each worker lane to have.
|
|
|
|
Milestone 422 step 1. One row per lane in `services/worker_lanes.LANES`.
|
|
|
|
## What is NOT in here
|
|
|
|
The lane's queues and its display name. Those are decided by `celery_app.py`'s
|
|
`task_routes` — an operator cannot move a backup off `maintenance_long` — so
|
|
storing them would be a row that can contradict the routing table, with
|
|
nothing to notice until a queue had no consumer. They live in
|
|
`services/worker_lanes.py`; this table holds only what an operator may change.
|
|
|
|
The DERIVED CEILING is also absent, and that is deliberate rather than an
|
|
omission. It is computed from the container's cgroup limits on every read, so
|
|
a row written on a 32GB host and later run in a 4GB container is bounded by
|
|
the 4GB — a stored ceiling would quietly authorise what the box can no longer
|
|
hold.
|
|
|
|
## The three numbers
|
|
|
|
slots <= slots_cap <= derived_ceiling
|
|
(live) (this row) (computed)
|
|
|
|
Operator's distinction, 2026-09-22: the derived value is *a cap on the cap*.
|
|
`slots_cap` is theirs and is always lowerable; it simply may not exceed what
|
|
the container can hold. The CHECK constraint below enforces the left half,
|
|
which is a fact about the row; the right half is enforced at write, because
|
|
it depends on a value no database column holds.
|
|
|
|
## enabled
|
|
|
|
Whether the lane consumes its queues at all. This is how ML ships off
|
|
(milestone 422 step 6): `enabled=false` with `slots=0`, so a fresh install
|
|
never loads a model or reaches HuggingFace, and turning tagging on in
|
|
Settings is what triggers the fetch.
|
|
|
|
Not a substitute for `slots=0`. A lane can be enabled with zero slots while
|
|
it is being resized, and the two answer different questions: `enabled` is
|
|
intent, `slots` is capacity.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
Boolean,
|
|
CheckConstraint,
|
|
DateTime,
|
|
Integer,
|
|
String,
|
|
func,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class WorkerLane(Base):
|
|
__tablename__ = "worker_lane"
|
|
__table_args__ = (
|
|
# Bare names — Base.metadata's naming convention prepends
|
|
# ck_worker_lane_. Pre-prefixing here doubles it, which is what
|
|
# alembic 0088 had to rename four constraints for (#3275).
|
|
CheckConstraint("slots >= 0", name="slots_non_negative"),
|
|
CheckConstraint("slots_cap >= 0", name="cap_non_negative"),
|
|
# The invariant that makes the cap mean anything. Enforced in the
|
|
# database rather than only in the service, because a row that
|
|
# violates it is not a rejected request — it is a lane that will be
|
|
# reconciled UP to a value the operator capped.
|
|
CheckConstraint("slots <= slots_cap", name="slots_within_cap"),
|
|
)
|
|
|
|
# The lane name from services/worker_lanes.LANES — never a container
|
|
# hostname. See models/service_seen.py for why: celery's worker names here
|
|
# are `celery@<container id>`, minted fresh on every deploy.
|
|
name: Mapped[str] = mapped_column(String(32), primary_key=True)
|
|
|
|
slots: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
slots_cap: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False)
|
|
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=func.now(),
|
|
onupdate=func.now(),
|
|
)
|