Agent: DRY pass + Status rate-metrics + real start/stop state machine #182

Merged

bvandeusen merged 3 commits from dev into main 2026-07-01 16:30:33 -04:00

Conversation 0 Commits 3 Files Changed 5

+363 -244

bvandeusen commented 2026-07-01 16:25:29 -04:00

Owner

Copy Link

Copy Source

Three agent-worker fixes stacked on dev, all CI-green (run 1840). Ships to :latest for deploy.

Commits

• 98b2ac9 — DRY pass on agent/fc_agent (behavior-preserving). Consolidated genuine duplication into single-source helpers: worker._fail/_release/_stopped/_abort_if_stopped/_timed/_ewma, client._submit/_post_quiet, detectors.Proposers._top, config._bool_env.
• 91ea06b — Status shows smoothed jobs/min + downloads/min. The buffer/on GPU/downloaders tiles were instantaneous samples of state that flips many times a second, so a 3 s poll only caught noise (read as "frozen"/"jumpy"). Replaced with two EWMA-smoothed derived-rate tiles (GPU throughput + fetch throughput, which together show pipeline balance); new monotonic downloaded counter. Buffer stays as the occupancy bar; pool sizes demoted to the sub-line.
• 6282e75 — real start/stop state machine. The Status pill hung on "stopping" forever: the backend had no lifecycle state, so the UI faked "stopping" from !running && active>0, which wedged when a consumer hung mid-release + stop() blocked the handler on curator releases. Now the backend owns stopped→starting→running→stopping→stopped and drives every transition; stop() returns instantly with a background monitor that lands on stopped bounded by a 20 s timeout (can never hang); starting→running only once curator actually answers; buttons gated by state; Start/Stop presses + every transition are logged.

Deploy note

Agent runs :latest (main), so this needs main + a re-pull to take effect on the desktop agent. No migrations; web/backend unaffected.

🤖 Generated with Claude Code

Three agent-worker fixes stacked on `dev`, all CI-green (run 1840). Ships to `:latest` for deploy. ## Commits - **`98b2ac9` — DRY pass on `agent/fc_agent`** (behavior-preserving). Consolidated genuine duplication into single-source helpers: `worker._fail/_release/_stopped/_abort_if_stopped/_timed/_ewma`, `client._submit/_post_quiet`, `detectors.Proposers._top`, `config._bool_env`. - **`91ea06b` — Status shows smoothed jobs/min + downloads/min.** The `buffer`/`on GPU`/`downloaders` tiles were instantaneous samples of state that flips many times a second, so a 3 s poll only caught noise (read as "frozen"/"jumpy"). Replaced with two EWMA-smoothed derived-rate tiles (GPU throughput + fetch throughput, which together show pipeline balance); new monotonic `downloaded` counter. Buffer stays as the occupancy bar; pool sizes demoted to the sub-line. - **`6282e75` — real start/stop state machine.** The Status pill hung on "stopping" forever: the backend had no lifecycle state, so the UI faked "stopping" from `!running && active>0`, which wedged when a consumer hung mid-release + `stop()` blocked the handler on curator releases. Now the backend owns `stopped→starting→running→stopping→stopped` and drives every transition; `stop()` returns instantly with a background monitor that lands on `stopped` bounded by a 20 s timeout (can never hang); `starting→running` only once curator actually answers; buttons gated by state; Start/Stop presses + every transition are logged. ## Deploy note Agent runs `:latest` (main), so this needs `main` + a re-pull to take effect on the desktop agent. No migrations; web/backend unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 3 commits 2026-07-01 16:25:30 -04:00

refactor(agent): DRY pass on the GPU agent worker package ... 98b2ac90dd

Consolidate genuine duplication in agent/fc_agent into single-source
helpers (behavior-preserving; DRY Pass process #594):

worker.py
- _fail(jid, image_id, exc, verb) — 4 terminal "fail this job" blocks
  (downloader HTTP-fault + decode, consumer non-transient + generic).
- _release(job_ids) (was _release_owned) — the one lease hand-back path;
  6 inline release([jid])+unhold sites now route through it.
- _stopped(stop_evt) + _abort_if_stopped(jid, stop_evt) — 4 stop-check
  -and-release blocks and every bare stop-check.
- _timed(stage) contextmanager — ~8 monotonic()/_record() timing pairs;
  records only on clean exit, matching the old skip-on-raise behavior.
- _ewma(prev, x, alpha) module fn — 3 EWMA updates in the autoscaler.

client.py
- _submit(path, payload) — submit / submit_embedding (retrying session).
- _post_quiet(path, payload) — heartbeat / fail / release fire-and-forget.

detectors.py
- Proposers._top(detector, image, cap) — merges components() and panels().

config.py
- _bool_env(name, default) — auto_start / auto_scale env parsing.

Left alone (recorded): the xyxy→norm-xywh conversion duplicated across
models.py/detectors.py (2 copies, independent wrapper modules — sharing
would couple them), and the _ensure_embedder/_ensure_proposers pair (same
lock shape, different concepts).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent): Status shows smoothed jobs/min + downloads/min (replace jumpy gauges) ... 91ea06be79

The buffer / on-GPU / downloader counts flip many times a second, so a 3s
status poll only ever samples noise — the tiles looked frozen (same value
twice) or random (wildly different), reading as "the Status section doesn't
update" when the backend was in fact live (operator-flagged 2026-07-01).

Replace the three instantaneous gauge tiles with two derived RATE tiles:
  - jobs / min      — GPU throughput, from the monotonic `processed` counter
  - downloads / min — fetch throughput, from a new monotonic `downloaded`
                      counter (bumped when a job is decoded into the buffer)
Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the
reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped
at 0 (agent restart resets the counters), and skip a backgrounded-tab gap.

The still-useful instantaneous state is demoted, not lost: buffer stays as
the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited
out` (transient) gets promoted to a tile.

backend: worker.status() gains `downloaded`; `_bump(downloaded=)`.
frontend: retiled Status + rate math in applyStatus; VERSION → .7.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(agent): real start/stop state machine — kill the stuck "stopping" pill ... 6282e753a9

The Status pill hung on "stopping" forever (operator-flagged 2026-07-01).
Root cause: the backend had no lifecycle state — status() only returned
running/stopped — so the UI FABRICATED "stopping" in JS as `!running &&
active>0`. That pill only cleared when the backend's `active` counter hit 0,
but stop() (a) blocked the HTTP handler on lease-release calls to curator and
(b) left `active>0` whenever a consumer wedged mid-submit/release to an
overloaded curator → "stopping" that never resolved.

Give the backend a real, truthful state it drives itself:
  stopped → starting → running → stopping → stopped
- start(): → starting; a downloader flips it to running on its FIRST
  successful lease (so "running" means curator is actually answering, not
  just "Start was clicked"). If curator's down it honestly stays "starting".
- stop(): → stopping; returns immediately (no handler block). A background
  monitor waits for the worker threads to actually exit, releases leases,
  then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can
  NEVER hold the UI in "stopping" again. In-flight work is handed back safely.
- Buttons follow the real state (Start only from stopped; both disabled
  through the transition), so you can't fight a transition.
- Log every Start/Stop button press (routes) and every transition (worker),
  so the Logs panel shows exactly what each button did.

Frontend now trusts s.state (drops the active>0 hack); VERSION → .8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit ed20df905b into main 2026-07-01 16:30:33 -04:00

bvandeusen referenced this pull request 2026-07-01 20:00:56 -04:00

Agent: server-side throughput rates + killable-on-stop ffmpeg #183

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#182