fix(chat,curator): unstick chat from silent generation crashes; curator only sees new messages

Two related reliability fixes.

1. routes/chat.py — guard run_generation against uncaught exceptions.

run_generation is launched with asyncio.create_task(); any exception
raised inside the coroutine is silently swallowed by the event loop,
the buffer stays in GenerationState.RUNNING forever, and every
subsequent POST /api/chat/conversations/<id>/messages returns 409
'Generation already in progress' — locking the user out of the chat
with no log trail.

Observed in dev 2026-05-22: assistant message 768 created at 20:36:59
with status=generating, stayed in that state for an hour+, and four
follow-up message attempts returned 409 instantly. The generation
task hung before any internal log line could fire, so the only
diagnostic was the 409 responses themselves.

Wrap run_generation in _run_generation_guarded() that catches
exceptions, logs with full traceback, transitions the buffer to
ERRORED, emits a final 'done' SSE event so any active stream
client closes cleanly, and marks the assistant message status=error
in the DB. After this, a stuck conversation recovers on its own
the next time the user sends a message — no manual DB poke needed.

2. services/curator_scheduler.py — pass last_curator_run_at as 'since'
to the curator so each sweep only sees messages added after the
previous successful pass.

Previously the scheduler called run_curator_for_conversation(conv_id)
with no 'since' argument, so the curator defaulted to its 24h
lookback window. Within an active journal session that meant every
15-min sweep re-extracted beats from messages already captured
on prior sweeps — producing duplicate moments.

_candidate_conversations() now returns (conv_id, last_curator_run_at)
tuples; _sweep() threads the timestamp through. First-run case
(last_curator_run_at IS NULL) falls back to the curator's default
24h window, which is what we want — process recent backlog on
first contact, then only deltas after.

Manual trigger path (POST /api/journal/curator/run/<conv_id>) is
intentionally NOT changed; it still passes since=None so the
24h re-sweep behaviour is preserved for ad-hoc 'reprocess today'
clicks from the UI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-22 17:55:52 -04:00
parent 49325816a3
commit fdb0f10848
2 changed files with 72 additions and 20 deletions
+52 -13
View File
@@ -178,19 +178,58 @@ async def send_message_route(conv_id: int):
model = await get_setting(uid, "default_model", Config.OLLAMA_MODEL) or Config.OLLAMA_MODEL
# Launch background generation task (context building happens inside the task)
asyncio.create_task(run_generation(
buf, history, model,
uid, conv_id, conv.title, content,
context_note_id=context_note_id,
include_note_ids=include_note_ids,
excluded_note_ids=excluded_note_ids,
think=think,
rag_project_id=effective_rag_project_id,
workspace_project_id=workspace_project_id,
user_timezone=user_timezone,
voice_mode=(conv.conversation_type == "voice"),
))
# Launch background generation task (context building happens inside the task).
#
# Wrap in a top-level guard: `run_generation` is fire-and-forget via
# asyncio.create_task. Without a wrapper, an unhandled exception inside
# the coroutine is swallowed by the event loop, the buffer stays in
# GenerationState.RUNNING forever, and every subsequent POST to this
# conversation returns 409 — locking the user out of the chat surface
# with no log trail. Observed in production 2026-05-22 where the
# generation hung before any internal log line could fire.
async def _run_generation_guarded():
try:
await run_generation(
buf, history, model,
uid, conv_id, conv.title, content,
context_note_id=context_note_id,
include_note_ids=include_note_ids,
excluded_note_ids=excluded_note_ids,
think=think,
rag_project_id=effective_rag_project_id,
workspace_project_id=workspace_project_id,
user_timezone=user_timezone,
voice_mode=(conv.conversation_type == "voice"),
)
except Exception:
logger.exception(
"run_generation crashed for conv %d msg %d (model=%s); "
"transitioning buffer to FAILED so the route can be used again",
conv_id, assistant_msg.id, model,
)
try:
buf.state = GenerationState.ERRORED
buf.append_event(
"done",
{"done": True, "error": "Generation crashed; see server logs"},
)
except Exception:
logger.exception("Failed to mark buffer ERRORED for conv %d", conv_id)
try:
from fabledassistant.services.generation_task import _update_message
await _update_message(
assistant_msg.id,
"",
"error",
tool_calls=None,
)
except Exception:
logger.exception(
"Failed to mark assistant message %d as error after crash",
assistant_msg.id,
)
asyncio.create_task(_run_generation_guarded())
return jsonify({
"assistant_message_id": assistant_msg.id,