fix(chat,curator): unstick chat from silent generation crashes; curator only sees new messages
Two related reliability fixes. 1. routes/chat.py — guard run_generation against uncaught exceptions. run_generation is launched with asyncio.create_task(); any exception raised inside the coroutine is silently swallowed by the event loop, the buffer stays in GenerationState.RUNNING forever, and every subsequent POST /api/chat/conversations/<id>/messages returns 409 'Generation already in progress' — locking the user out of the chat with no log trail. Observed in dev 2026-05-22: assistant message 768 created at 20:36:59 with status=generating, stayed in that state for an hour+, and four follow-up message attempts returned 409 instantly. The generation task hung before any internal log line could fire, so the only diagnostic was the 409 responses themselves. Wrap run_generation in _run_generation_guarded() that catches exceptions, logs with full traceback, transitions the buffer to ERRORED, emits a final 'done' SSE event so any active stream client closes cleanly, and marks the assistant message status=error in the DB. After this, a stuck conversation recovers on its own the next time the user sends a message — no manual DB poke needed. 2. services/curator_scheduler.py — pass last_curator_run_at as 'since' to the curator so each sweep only sees messages added after the previous successful pass. Previously the scheduler called run_curator_for_conversation(conv_id) with no 'since' argument, so the curator defaulted to its 24h lookback window. Within an active journal session that meant every 15-min sweep re-extracted beats from messages already captured on prior sweeps — producing duplicate moments. _candidate_conversations() now returns (conv_id, last_curator_run_at) tuples; _sweep() threads the timestamp through. First-run case (last_curator_run_at IS NULL) falls back to the curator's default 24h window, which is what we want — process recent backlog on first contact, then only deltas after. Manual trigger path (POST /api/journal/curator/run/<conv_id>) is intentionally NOT changed; it still passes since=None so the 24h re-sweep behaviour is preserved for ad-hoc 'reprocess today' clicks from the UI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -178,19 +178,58 @@ async def send_message_route(conv_id: int):
|
||||
|
||||
model = await get_setting(uid, "default_model", Config.OLLAMA_MODEL) or Config.OLLAMA_MODEL
|
||||
|
||||
# Launch background generation task (context building happens inside the task)
|
||||
asyncio.create_task(run_generation(
|
||||
buf, history, model,
|
||||
uid, conv_id, conv.title, content,
|
||||
context_note_id=context_note_id,
|
||||
include_note_ids=include_note_ids,
|
||||
excluded_note_ids=excluded_note_ids,
|
||||
think=think,
|
||||
rag_project_id=effective_rag_project_id,
|
||||
workspace_project_id=workspace_project_id,
|
||||
user_timezone=user_timezone,
|
||||
voice_mode=(conv.conversation_type == "voice"),
|
||||
))
|
||||
# Launch background generation task (context building happens inside the task).
|
||||
#
|
||||
# Wrap in a top-level guard: `run_generation` is fire-and-forget via
|
||||
# asyncio.create_task. Without a wrapper, an unhandled exception inside
|
||||
# the coroutine is swallowed by the event loop, the buffer stays in
|
||||
# GenerationState.RUNNING forever, and every subsequent POST to this
|
||||
# conversation returns 409 — locking the user out of the chat surface
|
||||
# with no log trail. Observed in production 2026-05-22 where the
|
||||
# generation hung before any internal log line could fire.
|
||||
async def _run_generation_guarded():
|
||||
try:
|
||||
await run_generation(
|
||||
buf, history, model,
|
||||
uid, conv_id, conv.title, content,
|
||||
context_note_id=context_note_id,
|
||||
include_note_ids=include_note_ids,
|
||||
excluded_note_ids=excluded_note_ids,
|
||||
think=think,
|
||||
rag_project_id=effective_rag_project_id,
|
||||
workspace_project_id=workspace_project_id,
|
||||
user_timezone=user_timezone,
|
||||
voice_mode=(conv.conversation_type == "voice"),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"run_generation crashed for conv %d msg %d (model=%s); "
|
||||
"transitioning buffer to FAILED so the route can be used again",
|
||||
conv_id, assistant_msg.id, model,
|
||||
)
|
||||
try:
|
||||
buf.state = GenerationState.ERRORED
|
||||
buf.append_event(
|
||||
"done",
|
||||
{"done": True, "error": "Generation crashed; see server logs"},
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to mark buffer ERRORED for conv %d", conv_id)
|
||||
try:
|
||||
from fabledassistant.services.generation_task import _update_message
|
||||
await _update_message(
|
||||
assistant_msg.id,
|
||||
"",
|
||||
"error",
|
||||
tool_calls=None,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Failed to mark assistant message %d as error after crash",
|
||||
assistant_msg.id,
|
||||
)
|
||||
|
||||
asyncio.create_task(_run_generation_guarded())
|
||||
|
||||
return jsonify({
|
||||
"assistant_message_id": assistant_msg.id,
|
||||
|
||||
Reference in New Issue
Block a user