Agent: DRY pass + Status rate-metrics + real start/stop state machine #182

Merged

bvandeusen merged 3 commits from dev into main 2026-07-01 16:30:33 -04:00

Conversation 0 Commits 3 Files Changed 5

+363 -244

bvandeusen commented 2026-07-01 16:25:29 -04:00

Owner

Copy Link

Copy Source

Three agent-worker fixes stacked on dev, all CI-green (run 1840). Ships to :latest for deploy.

Commits

• 98b2ac9 — DRY pass on agent/fc_agent (behavior-preserving). Consolidated genuine duplication into single-source helpers: worker._fail/_release/_stopped/_abort_if_stopped/_timed/_ewma, client._submit/_post_quiet, detectors.Proposers._top, config._bool_env.
• 91ea06b — Status shows smoothed jobs/min + downloads/min. The buffer/on GPU/downloaders tiles were instantaneous samples of state that flips many times a second, so a 3 s poll only caught noise (read as "frozen"/"jumpy"). Replaced with two EWMA-smoothed derived-rate tiles (GPU throughput + fetch throughput, which together show pipeline balance); new monotonic downloaded counter. Buffer stays as the occupancy bar; pool sizes demoted to the sub-line.
• 6282e75 — real start/stop state machine. The Status pill hung on "stopping" forever: the backend had no lifecycle state, so the UI faked "stopping" from !running && active>0, which wedged when a consumer hung mid-release + stop() blocked the handler on curator releases. Now the backend owns stopped→starting→running→stopping→stopped and drives every transition; stop() returns instantly with a background monitor that lands on stopped bounded by a 20 s timeout (can never hang); starting→running only once curator actually answers; buttons gated by state; Start/Stop presses + every transition are logged.

Deploy note

Agent runs :latest (main), so this needs main + a re-pull to take effect on the desktop agent. No migrations; web/backend unaffected.

🤖 Generated with Claude Code

Three agent-worker fixes stacked on `dev`, all CI-green (run 1840). Ships to `:latest` for deploy. ## Commits - **`98b2ac9` — DRY pass on `agent/fc_agent`** (behavior-preserving). Consolidated genuine duplication into single-source helpers: `worker._fail/_release/_stopped/_abort_if_stopped/_timed/_ewma`, `client._submit/_post_quiet`, `detectors.Proposers._top`, `config._bool_env`. - **`91ea06b` — Status shows smoothed jobs/min + downloads/min.** The `buffer`/`on GPU`/`downloaders` tiles were instantaneous samples of state that flips many times a second, so a 3 s poll only caught noise (read as "frozen"/"jumpy"). Replaced with two EWMA-smoothed derived-rate tiles (GPU throughput + fetch throughput, which together show pipeline balance); new monotonic `downloaded` counter. Buffer stays as the occupancy bar; pool sizes demoted to the sub-line. - **`6282e75` — real start/stop state machine.** The Status pill hung on "stopping" forever: the backend had no lifecycle state, so the UI faked "stopping" from `!running && active>0`, which wedged when a consumer hung mid-release + `stop()` blocked the handler on curator releases. Now the backend owns `stopped→starting→running→stopping→stopped` and drives every transition; `stop()` returns instantly with a background monitor that lands on `stopped` bounded by a 20 s timeout (can never hang); `starting→running` only once curator actually answers; buttons gated by state; Start/Stop presses + every transition are logged. ## Deploy note Agent runs `:latest` (main), so this needs `main` + a re-pull to take effect on the desktop agent. No migrations; web/backend unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 3 commits 2026-07-01 16:25:30 -04:00

refactor(agent): DRY pass on the GPU agent worker package ... 98b2ac90dd

Consolidate genuine duplication in agent/fc_agent into single-source
helpers (behavior-preserving; DRY Pass process #594):

worker.py
- _fail(jid, image_id, exc, verb) — 4 terminal "fail this job" blocks
  (downloader HTTP-fault + decode, consumer non-transient + generic).
- _release(job_ids) (was _release_owned) — the one lease hand-back path;
  6 inline release([jid])+unhold sites now route through it.
- _stopped(stop_evt) + _abort_if_stopped(jid, stop_evt) — 4 stop-check
  -and-release blocks and every bare stop-check.
- _timed(stage) contextmanager — ~8 monotonic()/_record() timing pairs;
  records only on clean exit, matching the old skip-on-raise behavior.
- _ewma(prev, x, alpha) module fn — 3 EWMA updates in the autoscaler.

client.py
- _submit(path, payload) — submit / submit_embedding (retrying session).
- _post_quiet(path, payload) — heartbeat / fail / release fire-and-forget.

detectors.py
- Proposers._top(detector, image, cap) — merges components() and panels().

config.py
- _bool_env(name, default) — auto_start / auto_scale env parsing.

Left alone (recorded): the xyxy→norm-xywh conversion duplicated across
models.py/detectors.py (2 copies, independent wrapper modules — sharing
would couple them), and the _ensure_embedder/_ensure_proposers pair (same
lock shape, different concepts).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent): Status shows smoothed jobs/min + downloads/min (replace jumpy gauges) ... 91ea06be79

The buffer / on-GPU / downloader counts flip many times a second, so a 3s
status poll only ever samples noise — the tiles looked frozen (same value
twice) or random (wildly different), reading as "the Status section doesn't
update" when the backend was in fact live (operator-flagged 2026-07-01).

Replace the three instantaneous gauge tiles with two derived RATE tiles:
  - jobs / min      — GPU throughput, from the monotonic `processed` counter
  - downloads / min — fetch throughput, from a new monotonic `downloaded`
                      counter (bumped when a job is decoded into the buffer)
Together they also show pipeline balance (dl/min > j/min ⇒ GPU-bound; the
reverse ⇒ GPU starved). Both are EWMA-smoothed over the poll deltas, clamped
at 0 (agent restart resets the counters), and skip a backgrounded-tab gap.

The still-useful instantaneous state is demoted, not lost: buffer stays as
the occupancy bar; downloaders/consumers/on-GPU move to the sub-line. `waited
out` (transient) gets promoted to a tile.

backend: worker.status() gains `downloaded`; `_bump(downloaded=)`.
frontend: retiled Status + rate math in applyStatus; VERSION → .7.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(agent): real start/stop state machine — kill the stuck "stopping" pill ... 6282e753a9

The Status pill hung on "stopping" forever (operator-flagged 2026-07-01).
Root cause: the backend had no lifecycle state — status() only returned
running/stopped — so the UI FABRICATED "stopping" in JS as `!running &&
active>0`. That pill only cleared when the backend's `active` counter hit 0,
but stop() (a) blocked the HTTP handler on lease-release calls to curator and
(b) left `active>0` whenever a consumer wedged mid-submit/release to an
overloaded curator → "stopping" that never resolved.

Give the backend a real, truthful state it drives itself:
  stopped → starting → running → stopping → stopped
- start(): → starting; a downloader flips it to running on its FIRST
  successful lease (so "running" means curator is actually answering, not
  just "Start was clicked"). If curator's down it honestly stays "starting".
- stop(): → stopping; returns immediately (no handler block). A background
  monitor waits for the worker threads to actually exit, releases leases,
  then → stopped — bounded by STOPPING_TIMEOUT (20s) so a wedged submit can
  NEVER hold the UI in "stopping" again. In-flight work is handed back safely.
- Buttons follow the real state (Start only from stopped; both disabled
  through the transition), so you can't fight a transition.
- Log every Start/Stop button press (routes) and every transition (worker),
  so the Logs panel shows exactly what each button did.

Frontend now trusts s.state (drops the active>0 hack); VERSION → .8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit ed20df905b into main 2026-07-01 16:30:33 -04:00

bvandeusen referenced this pull request 2026-07-01 20:00:56 -04:00

Agent: server-side throughput rates + killable-on-stop ffmpeg #183

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#182