feat: reconcile every running lane back to its stored slots (4293)

Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by its
supervisor comes back at its ENV concurrency, silently below whatever the
operator set, and nothing on step 2's write path would ever notice. Storing
the value made it survivable; this makes it survive.

A BEAT TASK, NOT A HOOK IN WEB — a deliberate deviation from the step as
written, for a reason already recorded in this codebase. Step 3 said "web
applies the stored values after it starts". It cannot: service_roster.py
documents that hypercorn runs --workers 4, so anything in before_serving
becomes four concurrent loops per container hammering the broker forever.

service_roster's own answer — refresh on demand from whichever request
arrives — was also rejected, because the two solve different problems. A
stale ROSTER only misleads someone looking at it, so recomputing when they
look is exactly right. A lane running at the wrong size is doing less work
than it was told to whether or not anyone is watching, and the case that
matters is a deploy at 3am followed by a backlog nobody is awake to see.

So: unattended, every 5 minutes, on the quick `maintenance` lane beside the
other recovery sweeps. Accepted cost — a dead scheduler stops reconciliation,
but a dead scheduler already stops every other sweep and the roster reports
it, so this adds no new blind spot.

A BUG I WROTE AND CAUGHT BEFORE COMMITTING. The first version called
set_lane_enabled_sync unconditionally, so a settled system re-sent
add_consumer for every queue on every tick — forever. Harmless per call
(add_consumer on an already-consumed queue does nothing), unbounded in
aggregate, and completely invisible. That is lesson #4183's failure mode
exactly, in the very function whose docstring cites it.

Worse, my test would not have caught it: it asserted only on grew/shrank.
LaneLiveState now carries `consuming` — which queues a lane is actually
serving, distinct from the queues it was configured with — so the reconcile
compares before acting. The test now asserts ALL FOUR control families are
silent on a settled tick, plus a new case for an already-disabled lane, which
is the other half of the same fixed point.

An absent lane is SKIPPED, not corrected. present=False means nothing
answered, not zero slots; correcting it would be a conclusion from an unswept
read (snippet #3969), and there would be nothing to send the message to. One
lane failing does not stop the others.

One inspect serves every lane: the two setters now take an optional
pre-fetched LaneLiveState, so a tick costs one broker round trip rather than
one per lane.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

tests/test_worker_control.py

@@ -37,13 +37,20 @@ class _Control:
         self.cancelled.append((queue, destination))
 
 
-def _stub_live(monkeypatch, lane_name, *, pools, present=True, reserved=0):
+def _stub_live(
+    monkeypatch, lane_name, *, pools, present=True, reserved=0, consuming=None,
+):
+    # `consuming` defaults to the lane's full queue set — i.e. an ENABLED
+    # lane. Tests for the disabled case pass an empty set explicitly.
+    if consuming is None:
+        consuming = set(LANES_BY_NAME[lane_name].queues)
     state = wc.LaneLiveState(
         present=present,
         replicas=len(pools),
         hostnames=sorted(pools),
         pools=dict(pools),
         reserved=reserved,
+        consuming=set(consuming),
     )
     monkeypatch.setattr(
         wc, "inspect_lanes_sync",
@@ -186,3 +193,131 @@ def test_an_unknown_queue_set_maps_to_no_lane():
     """A deployment slicing CELERY_QUEUES differently has no lane row to
     control. Honest rather than an error — the roster still reports it."""
     assert wc._lane_for_queues(("something", "else")) is None
+
+
+# --- the reconcile (step 3) --------------------------------------------------
+
+
+def test_a_settled_system_sends_no_control_messages(monkeypatch):
+    """THE property this sweep lives or dies on. It runs every 5 minutes
+    forever, so a converged tick must be silent — one broker round trip and
+    nothing else. An enforcer that re-issues a grow of zero churns forever and
+    buries a real correction in its own noise (lesson #4183).
+    """
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "worker", pools={"host-a": 4})
+
+    result = wc.reconcile_lanes_sync({"worker": (4, True)})
+
+    # EVERY control family, not just the pool ones. The first version of this
+    # test asserted only grew/shrank and would have passed while the reconcile
+    # re-sent add_consumer for all four queues on every tick — harmless per
+    # call, unbounded churn in aggregate, and invisible.
+    assert control.grew == []
+    assert control.shrank == []
+    assert control.added == []
+    assert control.cancelled == []
+    assert result["changed"] == []
+
+
+def test_a_worker_back_at_its_env_concurrency_is_corrected(monkeypatch):
+    """The failure this exists for: a restart drops the pool to CELERY_
+    CONCURRENCY, silently below what the operator set."""
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "worker", pools={"host-a": 2})
+
+    result = wc.reconcile_lanes_sync({"worker": (6, True)})
+
+    assert control.grew == [(4, ["host-a"])]
+    assert result["changed"] == ["worker"]
+
+
+def test_an_absent_lane_is_skipped_not_corrected(monkeypatch):
+    """`present=False` is 'nothing answered', not 'zero slots'. Correcting it
+    would be a conclusion drawn from an unswept read (snippet #3969) — and
+    there is nothing to send the message to anyway."""
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "worker", pools={}, present=False)
+
+    result = wc.reconcile_lanes_sync({"worker": (6, True)})
+
+    assert result["skipped"] == ["worker"]
+    assert result["changed"] == []
+    assert control.grew == []
+    assert control.shrank == []
+
+
+def test_one_lane_failing_does_not_stop_the_others(monkeypatch):
+    """A broker blip on one lane must not leave the rest un-reconciled for
+    another five minutes."""
+    control = _stub_control(monkeypatch)
+    present = wc.LaneLiveState(
+        present=True, replicas=1, hostnames=["host-a"], pools={"host-a": 1},
+    )
+    absent = wc.LaneLiveState()
+    monkeypatch.setattr(
+        wc, "inspect_lanes_sync",
+        lambda: {"worker": absent, "scheduler": present,
+                 "maintenance_long": absent, "ml": absent},
+    )
+
+    result = wc.reconcile_lanes_sync({
+        "worker": (4, True), "scheduler": (3, True),
+    })
+
+    assert result["skipped"] == ["worker"]
+    assert result["changed"] == ["scheduler"]
+    assert control.grew == [(2, ["host-a"])]
+
+
+def test_a_lane_disabled_in_settings_but_still_consuming_is_stopped(monkeypatch):
+    """The case that made `consuming` necessary: a lane turned off while its
+    worker was down comes back consuming, and must be stopped when it
+    returns."""
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "ml", pools={"host-a": 0}, consuming={"ml"})
+
+    result = wc.reconcile_lanes_sync({"ml": (0, False)})
+
+    assert control.cancelled == [("ml", ["host-a"])]
+    assert result["changed"] == ["ml"]
+
+
+def test_an_already_disabled_lane_is_not_cancelled_again(monkeypatch):
+    """The other half of the fixed point. `cancel_consumer` on a queue that is
+    not being consumed succeeds and does nothing, so an unconditional
+    reconcile would churn here forever with no symptom."""
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "ml", pools={"host-a": 0}, consuming=set())
+
+    result = wc.reconcile_lanes_sync({"ml": (0, False)})
+
+    assert control.cancelled == []
+    assert control.added == []
+    assert result["changed"] == []
+
+
+def test_a_lane_with_no_stored_row_is_left_alone(monkeypatch):
+    """A lane in LANES but not in `desired` is one whose row has not been
+    seeded. Inventing a target here would let the sweep enforce a number that
+    disagrees with the migration it is supposed to be upholding."""
+    control = _stub_control(monkeypatch)
+    _stub_live(monkeypatch, "worker", pools={"host-a": 2})
+
+    result = wc.reconcile_lanes_sync({})
+
+    assert result["changed"] == []
+    assert control.grew == []
+
+
+def test_the_reconcile_task_is_registered_and_scheduled():
+    """A task name only enters `celery.tasks` when its module is imported, and
+    a beat entry naming a task that is not registered fails at tick time
+    rather than at import — silently, every five minutes."""
+    import backend.app.tasks.maintenance  # noqa: F401
+    from backend.app.celery_app import celery
+
+    name = "backend.app.tasks.maintenance.reconcile_worker_lanes"
+    assert name in celery.tasks
+    scheduled = {e["task"] for e in celery.conf.beat_schedule.values()}
+    assert name in scheduled