fix(llm): default generate_completion num_ctx to Config.OLLAMA_NUM_CTX

Non-streaming generate_completion was the only LLM entry point that
didn't default num_ctx — stream_chat and stream_chat_with_tools both
fall back to Config.OLLAMA_NUM_CTX (16384). When a caller omitted the
argument, Ollama silently used the model's default window (~4k on
qwen3) and truncated the prompt.

That footgun was masked by fallback paths in the research pipeline:
_generate_outline's prompt carries ~12 sources × 2000 chars (~6k
tokens) of source material plus a system prompt, so the prompt got
chopped, the model never saw the sources, JSON parsing failed twice,
and run_research_pipeline dropped into the single-note "monolith"
fallback (research.py:251). The user reported chat 215 producing such
a monolith note for a multi-source research topic.

Two-layer fix:
- Default num_ctx to Config.OLLAMA_NUM_CTX inside generate_completion,
  matching the streaming entry points. Any current or future caller
  that forgets the argument stops silently losing input.
- Pin num_ctx=16384 explicitly in _generate_outline and
  _generate_executive_summary with comments pointing at the failure
  mode, so a refactor of the generate_completion default can't
  silently regress the research pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-13 18:20:58 -04:00
parent 4e4dbb8783
commit 70cea78c2f
2 changed files with 29 additions and 5 deletions
+11 -3
View File
@@ -317,9 +317,17 @@ async def generate_completion(
num_ctx overrides the model's context window for this call only.
"""
last_exc: Exception | None = None
options: dict = {"num_predict": max_tokens}
if num_ctx is not None:
options["num_ctx"] = num_ctx
# Default num_ctx to Config.OLLAMA_NUM_CTX (matching stream_chat /
# stream_chat_with_tools). Without this, Ollama silently uses the model's
# default window (~4k on qwen3) and truncates anything longer. That is
# how the research pipeline's outline step kept falling back to a single
# monolith note: its 12-source prompt is ~6k tokens and was being chopped
# before the model ever saw it. Non-streaming callers must not inherit
# that footgun — if you truly want the model default, pass num_ctx=0.
options: dict = {
"num_predict": max_tokens,
"num_ctx": num_ctx if num_ctx is not None else Config.OLLAMA_NUM_CTX,
}
for attempt in range(3):
if attempt > 0:
delay = 3.0 * attempt
+18 -2
View File
@@ -63,7 +63,17 @@ async def _generate_outline(topic: str, sources: list[dict], model: str) -> list
]
for attempt in range(2):
try:
raw = await generate_completion(messages, model, max_tokens=400)
# Pin num_ctx explicitly. The prompt carries up to 12 sources at
# 2000 chars each (~6k tokens of source material alone) plus the
# system prompt — well over Ollama's default model window on
# qwen3. Without this, Ollama silently truncates the prompt, the
# model can't see most of the sources, JSON parsing fails twice,
# and the pipeline falls back to a single monolith note
# (`research.py:251`). Do not remove even if `generate_completion`
# appears to default this — see the comment there.
raw = await generate_completion(
messages, model, max_tokens=400, num_ctx=16384
)
raw = raw.strip()
raw = re.sub(r"^```(?:json)?\s*", "", raw)
raw = re.sub(r"\s*```$", "", raw)
@@ -161,7 +171,13 @@ async def _generate_executive_summary(
},
]
try:
raw = await generate_completion(messages, model, max_tokens=600)
# Pin num_ctx explicitly — see `_generate_outline` comment for the
# rationale. This prompt carries N sections × 1500 chars of section
# prose, which can easily exceed the default model window. Don't
# trust the `generate_completion` default to stick.
raw = await generate_completion(
messages, model, max_tokens=600, num_ctx=16384
)
return raw.strip()
except Exception:
logger.warning("Executive summary generation failed for '%s'", topic, exc_info=True)