Was failing with ModuleNotFoundError for httpx when run via system
python — httpx is a project dep but isn't on the system interpreter's
path. Adding PEP 723 script metadata + uv-run shebang means the script
auto-resolves its deps in an ephemeral venv on every invocation, no
project-venv setup required.
Run with `uv run scripts/bench_ollama.py …` or directly via the shebang
`./scripts/bench_ollama.py …`. `python scripts/bench_ollama.py …` still
works only when httpx happens to be on the active interpreter.
Docstring updated to reflect the running options.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The curator scenario hardcoded think=true, which is qwen3-family-specific.
Non-qwen3 models silently ignore the field, so cross-family curator
comparisons were apples-to-oranges (qwen thinks, others don't).
New --think flag:
- auto (default): scenario-driven — chat=off, curator=on. Matches the
prior behaviour and the most common case.
- off: force disabled across all runs. Use for fair cross-family
comparison; aligns behaviour explicitly even though non-qwen models
would ignore think anyway.
- on: force enabled across all runs. Use to measure what think
contributes on the same model (paired runs: --think off then on).
Output markdown table now records the think mode used, so saved results
are self-documenting when you diff cross-server or cross-config.
Docstring + usage examples updated to reflect the qwen3 candidate set
the bench was originally tuned for.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Standalone tool to measure Ollama model performance under the two
workload shapes the chat+curator architecture would impose:
- chat scenario: short user message, short reply, no thinking. Mirrors
the no-tools chat companion's expected load.
- curator scenario: ~700-token journal transcript with an extraction
prompt, thinking enabled. Mirrors the curator's expected load.
Defaults to CPU-only inference (num_gpu=0). Streams responses; reports
TTFT, total wall time, tokens/sec (from Ollama's eval_count/eval_duration
so it excludes client-side stream overhead), and prompt token count.
First request per (model, num_gpu) is a warm-up to load the model into
memory; not counted in the measured runs.
Designed for cross-server comparison: --server points at any Ollama
instance, --out writes a markdown table. Comparing the two CPU servers
becomes a matter of running the same command on each and diffing the
output.
Lives outside the chat/curator architecture commitment — measurement
tool only. Tells us "is qwen2.5:32b on CPU fast enough for a 10-20 min
curator cadence?" without writing any of the architecture code yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen