Recurring app/db crashes with no clear cause in existing logs.
Adds three crash-class indicators with minimal overhead (~1 log
line/min, 0.1ms work per heartbeat).
services/diagnostics.py:
1. **Heartbeat** every 60s logs a snapshot:
- RSS memory (from /proc/self/status — no deps).
- asyncio task count.
- DB pool: size / checked_in / checked_out / overflow.
- Curator busy state (from is_curator_running()).
- Uptime.
A sudden silence in heartbeats bounds the crash time to within
60s. The last snapshot before silence usually rules in or out:
memory growth -> OOM, pool exhaustion -> connection leak, hung
curator -> stuck async task.
2. **Signal handler** for SIGTERM/SIGINT logs the signal name +
final snapshot before letting Hypercorn handle the actual
shutdown. Distinguishes 'orderly shutdown via signal X' from
'silent log gap then container exit code 137' (SIGKILL / OOM-kill
are uncatchable; their absence in our log IS the diagnostic).
3. **Asyncio exception hook** logs full tracebacks for unhandled
task exceptions with the task/coro name. Default behaviour
swallows these silently — exactly the pattern that locked us
out of chat at 409 for an hour back on 2026-05-22 before we
added the guard around run_generation.
app.py wires start_diagnostics() into before_serving and
stop_diagnostics() into after_serving. stop_diagnostics emits one
final snapshot so the silence that follows is intentional, not a
crash.
How to use the new logs to diagnose:
- App restarts with 'received SIGTERM' in the last lines:
Orderly shutdown (docker stop / swarm restart / manual). Look
upstream for who issued it.
- App restarts with no shutdown line, last heartbeat 30+s before:
Likely SIGKILL — OOM-kill or container resource limit. Check
'docker ps -a' for exit code 137, or 'dmesg | grep -i kill' on host.
- App restarts with no shutdown line, heartbeat showed climbing
RSS: Memory leak. Snapshot the last heartbeat's MB value vs
earlier — if it doubled over hours, OOM is the cause.
- App restarts, db_pool checked_out kept growing: Connection leak.
Look for code paths that open async_session() but never exit
the 'async with' block.
- App seemed alive but stopped responding to requests, heartbeats
continued: Curator hung holding _CURATOR_RUN_LOCK. Check
curator_busy=true across multiple heartbeats — if stuck >5min,
the Ollama call hung. Restart Ollama or the Scribe stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
