bvandeusenandClaude Opus 5 f23ab9f50e fix: the roster's inspect budget was exactly the work it waited for (4295) ...

From the operator's first consolidated deploy, 2026-09-23. The app is serving
— showcase, thumbnails, a Patreon ingest tick, all five lanes in one
container — and this repeats in the log:

    WARNING service roster: celery inspect failed; roster not refreshed
    File "service_roster.py", line 138, in refresh_celery_roster
        grouped = await asyncio.wait_for(...)
    TimeoutError

The inspect calls were working. The budget was wrong.

`_inspect_celery_sync` makes TWO broadcasts — `active_queues()` and
`active()` — and a broadcast with no `destination` cannot know how many
replies to expect, so each waits out its full timeout rather than returning
on the last reply. The sync call costs ~2 x INSPECT_TIMEOUT_SECONDS.

The wrapper allowed `INSPECT_TIMEOUT_SECONDS * 2`. That reads like a safety
factor and is precisely the worst case with nothing left over — and this runs
on a web process that was serving ninety thumbnails a second at the time, so
the thread handing off through `asyncio.to_thread` need not even be scheduled
inside the budget. A budget equal to the work fails under any load at all.

Now derived: `INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS + slack`, with
the round-trip count named beside the calls it counts. Both tests assert the
RELATION rather than the numbers, and one reads the source to check the count
still matches the calls actually made — a third inspect call added later is
exactly how this comes back silently.

Consequence while it was broken: the roster stopped advancing and the System
tab's rows went stale, with a traceback per attempt. Never an outage —
`refresh_celery_roster` catches and returns, `/api/system/health` kept
answering 200 throughout, which the same log shows.

## Observed, not fixed here

`worker_control.inspect_lanes_sync` makes FOUR of these broadcasts
(active_queues, stats, active, reserved) at 2.0s each — roughly 8s — and
`lane_view` awaits it with no deadline at all. That is the Settings ->
Activity -> Worker lanes card, so that card likely takes ~8s to load, and the
composite healthcheck carries the same cost against its 15s timeout. Reported
to the operator rather than changed: they are mid-deploy, and the fix is to
cut round trips rather than raise a number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

2026-09-23 10:32:09 -04:00

..

app

fix: the roster's inspect budget was exactly the work it waited for (4295)

2026-09-23 10:32:09 -04:00

__init__.py

feat: scaffold backend Python project (Quart + SQLAlchemy + Celery deps)

2026-05-14 07:31:39 -04:00