bvandeusenandClaude Opus 5
45bb7044f7
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 31s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m44s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"
There was a reason and it had expired, and underneath it there was plain
waste.
The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.
The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.
So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.
Consequences, stated rather than hidden:
- The live figures are up to one sweep old. measured_at travels with each
lane and the page says how old, because a stale number presented as
current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
staleness thresholds are in a relationship. 60s against a 90s stale
threshold left one missed tick between normal and all-yellow — the
shape of lesson #4355 — so the period is 30s, named once in
worker_lanes, and system_health asserts its headroom at import with a
test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
the direction asked for: "idle instances quiet down when not running".
Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
64 lines
2.4 KiB
Python
64 lines
2.4 KiB
Python
"""worker_lane_sample — where the sizing sweep leaves what it measured.
|
|
|
|
Operator, 2026-09-23, on the System tab: *"there is a repull every time this
|
|
page loads — is there a reason this info isn't being tracked in the
|
|
background and stored in some way?"*
|
|
|
|
`/api/system/workers` ran a full celery inspect on every call — four
|
|
broadcasts on an eleven-second budget — and the page polls it every fifteen
|
|
seconds. `size_worker_lanes` was already inspecting on a timer to decide pool
|
|
sizes, computing exactly these numbers and discarding them. This table is
|
|
where they land instead, and the endpoint becomes a plain read.
|
|
|
|
## Why a new table rather than columns on `worker_lane`
|
|
|
|
`worker_lane` holds the one number an operator sets. Putting a measurement
|
|
beside it is the mistake alembic 0105 undid: `slots` sat next to `slots_cap`,
|
|
and a measurement next to a preference reads as a second preference.
|
|
|
|
No backfill. A row appears when the sweep first runs (within its period), and
|
|
until then the lane reads as not-yet-measured, which is true.
|
|
|
|
Revision ID: 0107
|
|
Revises: 0106
|
|
Create Date: 2026-09-23
|
|
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0107"
|
|
down_revision: Union[str, None] = "0106"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.create_table(
|
|
"worker_lane_sample",
|
|
sa.Column("lane", sa.String(length=32), primary_key=True),
|
|
# Nullable=False with no server_default: the sweep writes every column
|
|
# on every upsert, so a row only ever exists complete.
|
|
sa.Column("present", sa.Boolean(), nullable=False),
|
|
sa.Column("replicas", sa.Integer(), nullable=False),
|
|
# Nullable on purpose — unknown, never zero. A worker that answered
|
|
# without reporting its pool, and a queue the broker did not answer
|
|
# for, must not be summed as empty.
|
|
sa.Column("pool", sa.Integer(), nullable=True),
|
|
sa.Column("active", sa.Integer(), nullable=False),
|
|
sa.Column("reserved", sa.Integer(), nullable=False),
|
|
sa.Column("queue_depth", sa.Integer(), nullable=True),
|
|
sa.Column(
|
|
"measured_at",
|
|
sa.DateTime(timezone=True),
|
|
nullable=False,
|
|
server_default=sa.func.now(),
|
|
),
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_table("worker_lane_sample")
|