fix(llm): default generate_completion num_ctx to Config.OLLAMA_NUM_CTX
Non-streaming generate_completion was the only LLM entry point that didn't default num_ctx — stream_chat and stream_chat_with_tools both fall back to Config.OLLAMA_NUM_CTX (16384). When a caller omitted the argument, Ollama silently used the model's default window (~4k on qwen3) and truncated the prompt. That footgun was masked by fallback paths in the research pipeline: _generate_outline's prompt carries ~12 sources × 2000 chars (~6k tokens) of source material plus a system prompt, so the prompt got chopped, the model never saw the sources, JSON parsing failed twice, and run_research_pipeline dropped into the single-note "monolith" fallback (research.py:251). The user reported chat 215 producing such a monolith note for a multi-source research topic. Two-layer fix: - Default num_ctx to Config.OLLAMA_NUM_CTX inside generate_completion, matching the streaming entry points. Any current or future caller that forgets the argument stops silently losing input. - Pin num_ctx=16384 explicitly in _generate_outline and _generate_executive_summary with comments pointing at the failure mode, so a refactor of the generate_completion default can't silently regress the research pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -317,9 +317,17 @@ async def generate_completion(
|
||||
num_ctx overrides the model's context window for this call only.
|
||||
"""
|
||||
last_exc: Exception | None = None
|
||||
options: dict = {"num_predict": max_tokens}
|
||||
if num_ctx is not None:
|
||||
options["num_ctx"] = num_ctx
|
||||
# Default num_ctx to Config.OLLAMA_NUM_CTX (matching stream_chat /
|
||||
# stream_chat_with_tools). Without this, Ollama silently uses the model's
|
||||
# default window (~4k on qwen3) and truncates anything longer. That is
|
||||
# how the research pipeline's outline step kept falling back to a single
|
||||
# monolith note: its 12-source prompt is ~6k tokens and was being chopped
|
||||
# before the model ever saw it. Non-streaming callers must not inherit
|
||||
# that footgun — if you truly want the model default, pass num_ctx=0.
|
||||
options: dict = {
|
||||
"num_predict": max_tokens,
|
||||
"num_ctx": num_ctx if num_ctx is not None else Config.OLLAMA_NUM_CTX,
|
||||
}
|
||||
for attempt in range(3):
|
||||
if attempt > 0:
|
||||
delay = 3.0 * attempt
|
||||
|
||||
@@ -63,7 +63,17 @@ async def _generate_outline(topic: str, sources: list[dict], model: str) -> list
|
||||
]
|
||||
for attempt in range(2):
|
||||
try:
|
||||
raw = await generate_completion(messages, model, max_tokens=400)
|
||||
# Pin num_ctx explicitly. The prompt carries up to 12 sources at
|
||||
# 2000 chars each (~6k tokens of source material alone) plus the
|
||||
# system prompt — well over Ollama's default model window on
|
||||
# qwen3. Without this, Ollama silently truncates the prompt, the
|
||||
# model can't see most of the sources, JSON parsing fails twice,
|
||||
# and the pipeline falls back to a single monolith note
|
||||
# (`research.py:251`). Do not remove even if `generate_completion`
|
||||
# appears to default this — see the comment there.
|
||||
raw = await generate_completion(
|
||||
messages, model, max_tokens=400, num_ctx=16384
|
||||
)
|
||||
raw = raw.strip()
|
||||
raw = re.sub(r"^```(?:json)?\s*", "", raw)
|
||||
raw = re.sub(r"\s*```$", "", raw)
|
||||
@@ -161,7 +171,13 @@ async def _generate_executive_summary(
|
||||
},
|
||||
]
|
||||
try:
|
||||
raw = await generate_completion(messages, model, max_tokens=600)
|
||||
# Pin num_ctx explicitly — see `_generate_outline` comment for the
|
||||
# rationale. This prompt carries N sections × 1500 chars of section
|
||||
# prose, which can easily exceed the default model window. Don't
|
||||
# trust the `generate_completion` default to stick.
|
||||
raw = await generate_completion(
|
||||
messages, model, max_tokens=600, num_ctx=16384
|
||||
)
|
||||
return raw.strip()
|
||||
except Exception:
|
||||
logger.warning("Executive summary generation failed for '%s'", topic, exc_info=True)
|
||||
|
||||
Reference in New Issue
Block a user