fix: the roster's inspect budget was exactly the work it waited for (4295)

From the operator's first consolidated deploy, 2026-09-23. The app is serving
— showcase, thumbnails, a Patreon ingest tick, all five lanes in one
container — and this repeats in the log:

    WARNING service roster: celery inspect failed; roster not refreshed
    File "service_roster.py", line 138, in refresh_celery_roster
        grouped = await asyncio.wait_for(...)
    TimeoutError

The inspect calls were working. The budget was wrong.

`_inspect_celery_sync` makes TWO broadcasts — `active_queues()` and
`active()` — and a broadcast with no `destination` cannot know how many
replies to expect, so each waits out its full timeout rather than returning
on the last reply. The sync call costs ~2 x INSPECT_TIMEOUT_SECONDS.

The wrapper allowed `INSPECT_TIMEOUT_SECONDS * 2`. That reads like a safety
factor and is precisely the worst case with nothing left over — and this runs
on a web process that was serving ninety thumbnails a second at the time, so
the thread handing off through `asyncio.to_thread` need not even be scheduled
inside the budget. A budget equal to the work fails under any load at all.

Now derived: `INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS + slack`, with
the round-trip count named beside the calls it counts. Both tests assert the
RELATION rather than the numbers, and one reads the source to check the count
still matches the calls actually made — a third inspect call added later is
exactly how this comes back silently.

Consequence while it was broken: the roster stopped advancing and the System
tab's rows went stale, with a traceback per attempt. Never an outage —
`refresh_celery_roster` catches and returns, `/api/system/health` kept
answering 200 throughout, which the same log shows.

## Observed, not fixed here

`worker_control.inspect_lanes_sync` makes FOUR of these broadcasts
(active_queues, stats, active, reserved) at 2.0s each — roughly 8s — and
`lane_view` awaits it with no deadline at all. That is the Settings ->
Activity -> Worker lanes card, so that card likely takes ~8s to load, and the
composite healthcheck carries the same cost against its 15s timeout. Reported
to the operator rather than changed: they are mid-deploy, and the fix is to
cut round trips rather than raise a number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/services/service_roster.py

@@ -51,6 +51,29 @@ REFRESH_TTL_SECONDS = 20.0
 # page that exists to explain it.
 INSPECT_TIMEOUT_SECONDS = 2.0
 
+# How many broadcast round trips `_inspect_celery_sync` makes. Named, because
+# the wrapper's budget is derived from it and the two must not drift.
+#
+# `active_queues()` and `active()` are separate broadcasts, and a broadcast
+# with no `destination` cannot know how many replies to expect — so each one
+# waits out its full timeout rather than returning on the last reply. The sync
+# call therefore costs ~2 x INSPECT_TIMEOUT_SECONDS in the ordinary case, not
+# once.
+INSPECT_ROUND_TRIPS = 2
+
+# Slack for the thread handoff. `asyncio.to_thread` hands work to the default
+# executor, and on a loaded web process — the operator's showcase page pulling
+# ninety thumbnails a second — the thread may not even be scheduled inside the
+# budget, let alone finish.
+#
+# This exists because the wrapper used to allow `INSPECT_TIMEOUT_SECONDS * 2`,
+# which LOOKS like a safety factor and is exactly the worst case with nothing
+# left over. Observed on the operator's first consolidated deploy, 2026-09-23:
+# a TimeoutError traceback per refresh while the two inspect calls were
+# working perfectly. A budget equal to the work is a budget that fails under
+# any load at all.
+INSPECT_SLACK_SECONDS = 3.0
+
 # Queue set -> the name an operator recognises. Sorted-tuple keys, because the
 # order celery reports them in is not guaranteed.
 #
@@ -86,6 +109,10 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
     from ..celery_app import celery as celery_app
 
     insp = celery_app.control.inspect(timeout=INSPECT_TIMEOUT_SECONDS)
+    # TWO broadcasts, each waiting out its own timeout — see
+    # INSPECT_ROUND_TRIPS, which the caller's budget is derived from. Adding a
+    # third call here without updating that constant puts the wrapper back
+    # under the work it is waiting for.
     active_queues = insp.active_queues() or {}
     active_tasks = insp.active() or {}
 
@@ -137,7 +164,10 @@ async def refresh_celery_roster(session: AsyncSession) -> None:
     try:
         grouped = await asyncio.wait_for(
             asyncio.to_thread(_inspect_celery_sync),
-            timeout=INSPECT_TIMEOUT_SECONDS * 2,
+            timeout=(
+                INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS
+                + INSPECT_SLACK_SECONDS
+            ),
         )
     except Exception:
         log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True)