bvandeusen
eb02603092
Recurring app/db crashes with no clear cause in existing logs. Adds three crash-class indicators with minimal overhead (~1 log line/min, 0.1ms work per heartbeat). services/diagnostics.py: 1. **Heartbeat** every 60s logs a snapshot: - RSS memory (from /proc/self/status — no deps). - asyncio task count. - DB pool: size / checked_in / checked_out / overflow. - Curator busy state (from is_curator_running()). - Uptime. A sudden silence in heartbeats bounds the crash time to within 60s. The last snapshot before silence usually rules in or out: memory growth -> OOM, pool exhaustion -> connection leak, hung curator -> stuck async task. 2. **Signal handler** for SIGTERM/SIGINT logs the signal name + final snapshot before letting Hypercorn handle the actual shutdown. Distinguishes 'orderly shutdown via signal X' from 'silent log gap then container exit code 137' (SIGKILL / OOM-kill are uncatchable; their absence in our log IS the diagnostic). 3. **Asyncio exception hook** logs full tracebacks for unhandled task exceptions with the task/coro name. Default behaviour swallows these silently — exactly the pattern that locked us out of chat at 409 for an hour back on 2026-05-22 before we added the guard around run_generation. app.py wires start_diagnostics() into before_serving and stop_diagnostics() into after_serving. stop_diagnostics emits one final snapshot so the silence that follows is intentional, not a crash. How to use the new logs to diagnose: - App restarts with 'received SIGTERM' in the last lines: Orderly shutdown (docker stop / swarm restart / manual). Look upstream for who issued it. - App restarts with no shutdown line, last heartbeat 30+s before: Likely SIGKILL — OOM-kill or container resource limit. Check 'docker ps -a' for exit code 137, or 'dmesg | grep -i kill' on host. - App restarts with no shutdown line, heartbeat showed climbing RSS: Memory leak. Snapshot the last heartbeat's MB value vs earlier — if it doubled over hours, OOM is the cause. - App restarts, db_pool checked_out kept growing: Connection leak. Look for code paths that open async_session() but never exit the 'async with' block. - App seemed alive but stopped responding to requests, heartbeats continued: Curator hung holding _CURATOR_RUN_LOCK. Check curator_busy=true across multiple heartbeats — if stuck >5min, the Ollama call hung. Restart Ollama or the Scribe stack. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fabled Scribe
A self-hosted second brain and project management application with integrated LLM capabilities. Write, organise, and act on your notes and tasks with the help of a local AI assistant — all running on your own hardware.
Features
Notes and tasks with a Markdown editor, sub-tasks, milestones, and kanban project workspaces. AI chat with streaming responses, RAG over your notes, and tool use (web search, calendar, weather). A daily briefing that digests your tasks, RSS feeds, and weather on a schedule. Knowledge graph, per-user/group sharing, PWA with push notifications, an MCP server for external AI clients, and an Android companion app.
Quick Start
Prerequisites: Docker and Docker Compose. 8 GB+ RAM recommended for LLM inference.
Download docker-compose.quickstart.yml from this repo, then:
# Optional but recommended — set a secret key
export SECRET_KEY=your-random-secret-here
docker compose -f docker-compose.quickstart.yml up -d
Open http://localhost:5000. The first user to register becomes admin. Go to Settings → General to pull an LLM model — qwen3:8b or llama3.1:8b are good starting points.
GPU: Ollama runs CPU-only by default. See the comments in
docker-compose.quickstart.ymlto enable NVIDIA GPU passthrough.
Development: To build from source, see Development.
Documentation
| Doc | Contents |
|---|---|
| Architecture | Stack, design decisions, data models, key services |
| Configuration | Environment variables, Docker Compose, production setup, security |
| Features | Detailed feature breakdown and keyboard shortcuts |
| Development | Dev workflow, CI/CD, migrations, release process |
| API Keys & MCP | API key management and Fable MCP install guide |
| SSO / OAuth | OIDC setup for Authentik, Keycloak, and other providers |
| API Reference | All REST API endpoints |
| Android App | Flutter companion app architecture and feature status |
License
This project is privately maintained.