Standalone tool to measure Ollama model performance under the two
workload shapes the chat+curator architecture would impose:
- chat scenario: short user message, short reply, no thinking. Mirrors
the no-tools chat companion's expected load.
- curator scenario: ~700-token journal transcript with an extraction
prompt, thinking enabled. Mirrors the curator's expected load.
Defaults to CPU-only inference (num_gpu=0). Streams responses; reports
TTFT, total wall time, tokens/sec (from Ollama's eval_count/eval_duration
so it excludes client-side stream overhead), and prompt token count.
First request per (model, num_gpu) is a warm-up to load the model into
memory; not counted in the measured runs.
Designed for cross-server comparison: --server points at any Ollama
instance, --out writes a markdown table. Comparing the two CPU servers
becomes a matter of running the same command on each and diffing the
output.
Lives outside the chat/curator architecture commitment — measurement
tool only. Tells us "is qwen2.5:32b on CPU fast enough for a 10-20 min
curator cadence?" without writing any of the architecture code yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen
d3d4294c30