fix(llm): default generate_completion num_ctx to Config.OLLAMA_NUM_CTX
Non-streaming generate_completion was the only LLM entry point that didn't default num_ctx — stream_chat and stream_chat_with_tools both fall back to Config.OLLAMA_NUM_CTX (16384). When a caller omitted the argument, Ollama silently used the model's default window (~4k on qwen3) and truncated the prompt. That footgun was masked by fallback paths in the research pipeline: _generate_outline's prompt carries ~12 sources × 2000 chars (~6k tokens) of source material plus a system prompt, so the prompt got chopped, the model never saw the sources, JSON parsing failed twice, and run_research_pipeline dropped into the single-note "monolith" fallback (research.py:251). The user reported chat 215 producing such a monolith note for a multi-source research topic. Two-layer fix: - Default num_ctx to Config.OLLAMA_NUM_CTX inside generate_completion, matching the streaming entry points. Any current or future caller that forgets the argument stops silently losing input. - Pin num_ctx=16384 explicitly in _generate_outline and _generate_executive_summary with comments pointing at the failure mode, so a refactor of the generate_completion default can't silently regress the research pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -317,9 +317,17 @@ async def generate_completion(
|
|||||||
num_ctx overrides the model's context window for this call only.
|
num_ctx overrides the model's context window for this call only.
|
||||||
"""
|
"""
|
||||||
last_exc: Exception | None = None
|
last_exc: Exception | None = None
|
||||||
options: dict = {"num_predict": max_tokens}
|
# Default num_ctx to Config.OLLAMA_NUM_CTX (matching stream_chat /
|
||||||
if num_ctx is not None:
|
# stream_chat_with_tools). Without this, Ollama silently uses the model's
|
||||||
options["num_ctx"] = num_ctx
|
# default window (~4k on qwen3) and truncates anything longer. That is
|
||||||
|
# how the research pipeline's outline step kept falling back to a single
|
||||||
|
# monolith note: its 12-source prompt is ~6k tokens and was being chopped
|
||||||
|
# before the model ever saw it. Non-streaming callers must not inherit
|
||||||
|
# that footgun — if you truly want the model default, pass num_ctx=0.
|
||||||
|
options: dict = {
|
||||||
|
"num_predict": max_tokens,
|
||||||
|
"num_ctx": num_ctx if num_ctx is not None else Config.OLLAMA_NUM_CTX,
|
||||||
|
}
|
||||||
for attempt in range(3):
|
for attempt in range(3):
|
||||||
if attempt > 0:
|
if attempt > 0:
|
||||||
delay = 3.0 * attempt
|
delay = 3.0 * attempt
|
||||||
|
|||||||
@@ -63,7 +63,17 @@ async def _generate_outline(topic: str, sources: list[dict], model: str) -> list
|
|||||||
]
|
]
|
||||||
for attempt in range(2):
|
for attempt in range(2):
|
||||||
try:
|
try:
|
||||||
raw = await generate_completion(messages, model, max_tokens=400)
|
# Pin num_ctx explicitly. The prompt carries up to 12 sources at
|
||||||
|
# 2000 chars each (~6k tokens of source material alone) plus the
|
||||||
|
# system prompt — well over Ollama's default model window on
|
||||||
|
# qwen3. Without this, Ollama silently truncates the prompt, the
|
||||||
|
# model can't see most of the sources, JSON parsing fails twice,
|
||||||
|
# and the pipeline falls back to a single monolith note
|
||||||
|
# (`research.py:251`). Do not remove even if `generate_completion`
|
||||||
|
# appears to default this — see the comment there.
|
||||||
|
raw = await generate_completion(
|
||||||
|
messages, model, max_tokens=400, num_ctx=16384
|
||||||
|
)
|
||||||
raw = raw.strip()
|
raw = raw.strip()
|
||||||
raw = re.sub(r"^```(?:json)?\s*", "", raw)
|
raw = re.sub(r"^```(?:json)?\s*", "", raw)
|
||||||
raw = re.sub(r"\s*```$", "", raw)
|
raw = re.sub(r"\s*```$", "", raw)
|
||||||
@@ -161,7 +171,13 @@ async def _generate_executive_summary(
|
|||||||
},
|
},
|
||||||
]
|
]
|
||||||
try:
|
try:
|
||||||
raw = await generate_completion(messages, model, max_tokens=600)
|
# Pin num_ctx explicitly — see `_generate_outline` comment for the
|
||||||
|
# rationale. This prompt carries N sections × 1500 chars of section
|
||||||
|
# prose, which can easily exceed the default model window. Don't
|
||||||
|
# trust the `generate_completion` default to stick.
|
||||||
|
raw = await generate_completion(
|
||||||
|
messages, model, max_tokens=600, num_ctx=16384
|
||||||
|
)
|
||||||
return raw.strip()
|
return raw.strip()
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Executive summary generation failed for '%s'", topic, exc_info=True)
|
logger.warning("Executive summary generation failed for '%s'", topic, exc_info=True)
|
||||||
|
|||||||
Reference in New Issue
Block a user