Mitigation for the nightly fabledscribe Postgres outage on the
vdnt-docker02 Swarm node (incidents 2026-05-15/16/17 around 03:50 UTC).
Confirmed kill chain (not the trigger): a brief host-level setns/exec
stall makes the Docker healthcheck exec fail with exit 1 → unhealthy →
SIGKILL → fast-shutdown can't finish on NFS in 10s → exit 137 → swarm
restart_policy.max_attempts: 5 burns out → DB stays dead.
Hardens the `db` service so a transient host blip can't escalate to
killing the database:
- stop_grace_period: 120s (gives PG room to fsync on shutdown)
- healthcheck: interval 30s / timeout 10s / retries 10 / start_period 180s
(only gates app startup order — not authoritative liveness)
- prod: restart_policy condition=on-failure, max_attempts=0, window=120s
- quickstart/dev: restart: unless-stopped
Host-side trigger (what stalls runc/exec at ~03:50 UTC) is still under
investigation — see project_pg_nightly_outage.md.
Note: the Portainer prod stack differs from docker-compose.prod.yml
here (NFS bind, traefik labels, no ollama). The same `db:` block needs
to be pasted into Portainer for the prod mitigation to apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- TRUST_PROXY_HEADERS=true: ensures rate limiting keys on the real client IP
(X-Forwarded-For / X-Real-IP) rather than the Traefik container IP
- SECURE_COOKIES=true: sets the Secure flag on session cookies since the app
is served behind TLS via Traefik
- Remove ports: "5000:5000" — Traefik routes traffic internally; publishing
the port directly would bypass Traefik middleware
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Docker Compose:
- Enable Ollama GPU passthrough (nvidia, count: all) in both dev and prod files
- Add OLLAMA_FLASH_ATTENTION=1 (faster attention on GPU in both files)
- Add OLLAMA_MAX_LOADED_MODELS=2 and OLLAMA_KEEP_ALIVE=30m to prod (was already in dev)
- Remove 8G memory limit from prod Ollama service (CPU-bound constraint, no longer valid)
llm.py:
- Increase num_ctx 16384 → 32768 in stream_chat and stream_chat_with_tools (GPU VRAM allows it)
- Increase num_predict cap 4096 → 8192 for tool-augmented responses
generation_task.py:
- Parallelize build_context, get_tools_for_user, and get_setting all from the start
- As soon as tools list is ready (fast DB call), launch classify_intent as an asyncio.Task
- Await build_context and classify_intent together via asyncio.gather
- Intent result is pre-computed before the generation loop; loop just reads pre_intent on round 0
- intent_ms timing now reflects wall-clock time from intent start to completion
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Registration auto-closes after first user; admin can toggle from /admin/users
- Admin user management view with user list and delete
- Password confirmation on registration form
- Password change in Settings (PUT /api/auth/password)
- Session cookie hardening: HttpOnly, SameSite=Lax, optional Secure flag
- Startup warning when SECRET_KEY is default
- Production deployment docs: reverse proxy, rate limiting, CSP headers
- Fix assist prompt to preserve markdown headings in target sections
- Simplify prod compose networking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bvandeusen