d4666bea7f9148ddbd34b01760faa9347ab875aa
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
91bafb641f |
refactor: Phase 8 — backend deletion (chat / voice / push / journal / curator)
Mega-commit. Strips all server-side LLM machinery now that Phase 7 has
removed the corresponding UI surfaces and the MCP HTTP endpoint is the
sole assistant interface.
Deleted (services/):
chat, generation_buffer, generation_log, generation_task, llm, tools/
(entire package), stt, tts, voice_config, voice_library, push,
journal_closeout, journal_pipeline, journal_prep, journal_scheduler,
journal_search, curator, curator_scheduler, consolidation,
tag_suggestions, research, weather, article_fetcher, pending_actions,
moments, assist, wikipedia.
Deleted (routes/):
chat, voice, push, journal, quick_capture, fable_mcp_dist.
Deleted (models/):
conversation, generation_tool_log, push_subscription,
pending_curator_action, moment, weather_cache.
Deleted (tests/):
test_generation_log, test_journal_*, test_consolidation, test_lookup_tool,
test_notes_consolidation_trigger, test_record_moment_guards,
test_research_pipeline, test_tools_*, test_tool_use_fixes,
test_voice_library, test_weather_service, test_calendar_tool_tz,
test_wikipedia.
Deleted (top-level):
fable-mcp/ (legacy standalone stdio package — wheel-build pipeline
also removed from Dockerfile).
app.py:
- blueprint registrations for the 6 deleted routes
- startup hook trimmed: no more Ollama warmup, KV-cache priming,
journal/curator schedulers, voice model loading
- shutdown hook simplified
- httpx import dropped (was for Ollama calls)
pyproject.toml:
- removed deps: pywebpush, feedparser, html2text, trafilatura
- removed [voice] extras entirely
- description updated for the MCP-first architecture
Dockerfile:
- removed faster-whisper / piper-tts install steps
- removed bundled piper voice download stage
- removed fable-mcp wheel build stage
Surviving-file edits:
- services/auth.py: drop Conversation table claim on first-user setup
- services/backup.py: drop conversation / push-subscription export+restore;
v1/v2 restore now silently skip pre-pivot conversation data
- services/notes.py: drop maybe_consolidate trigger on task done/cancelled;
drop _maybe_trigger_project_summary (LLM auto-summary)
- services/projects.py: drop generate_project_summary + backfill_project_summaries
(both LLM-driven)
- services/user_profile.py: drop append_observations / consolidate /
clear_learned_data (curator-tied) and build_profile_context
(was LLM system-prompt builder)
- services/notifications.py: stub out _fire_push_notif (was send_push_notification)
- services/event_scheduler.py: drop event-reminder push + chat-retention
cleanup job; keep CalDAV pull-sync + reminders job (in-app)
- services/diagnostics.py: _curator_busy() always False
- routes/notes.py: drop /assist, /assist/stream, /suggest-tags endpoints
- routes/tasks.py: drop /<id>/consolidate endpoint
- routes/settings.py: drop /models, KV-cache-prime-on-save, journal-schedule
timezone hook, and the SearXNG search-test endpoint; inline _is_private_url
(was in services/llm.py)
- routes/admin.py: drop /voice, /voice/reload endpoints
- routes/profile.py: drop /consolidate, /observations (GET, DELETE)
- models/__init__.py: drop the 6 dead model imports
Frontend cascade:
- stores/push.ts: deleted entirely (no callers after Phase 7)
- stores/settings.ts: drop checkVoiceStatus + voice-status state
- views/SettingsView.vue: drop Locations section + journalConfig state
(was tied to /api/journal/config); drop JournalConfig + journal/voice
api/client imports
- frontend/api/client.ts: orphaned voice/journal/profile-observation/
fable-mcp-dist exports are left as dead but harmless (call them and
they 404; type-check is clean).
Pre-existing v1 backups that contained conversations/messages still
restore — those tables are silently dropped from the import path.
Anyone pulling the new image with a populated database will need the
Phase 9 migration to drop the dead tables (coming next).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
8a8d6fc9f2 |
feat(diagnostics): persist crash state to /data so it survives container death
The previous diagnostic instrumentation only wrote to stdout — fine for
'tail the logs while debugging', useless for 'crash happened at 3am
and Docker rotated the logs by morning'. This commit makes the
diagnostic state durable across container restart, OOM-kill, and log
rotation by writing to the mounted /data volume.
Four artifacts in /data/diagnostics/:
- current.json — overwritten atomically every heartbeat. Holds the
last known good snapshot (rss, asyncio_tasks, db_pool, curator_busy,
uptime, pid). Post-crash, this file alone tells you what the app
was doing 0-60 seconds before it died. Atomic write (tmp+rename)
so a crash mid-write can't leave a half-written file.
- last_shutdown.json — written when SIGTERM/SIGINT is caught OR
after_serving fires cleanly. If this file's mtime is older than
current.json's, the previous run died WITHOUT calling shutdown
(== SIGKILL, OOM-kill, or container hard-stop).
- last_exception.json — written when the asyncio exception hook
fires. Includes task name, coro name, exception type and message
alongside the resource snapshot.
- diag.log + diag.log.1..5 — rotating file log (10 MB × 5 backups
= 50 MB cap) containing every heartbeat, signal, and exception.
Separate from the app's stdout logger so Docker log rotation
can't take it out.
- previous_run.json — written at startup IF the post-mortem detects
the previous run died abruptly. Includes the abrupt-death snapshot
preserved for retrospection, so a recurring crash pattern can be
diffed over time.
Post-mortem at startup:
- Reads current.json + last_shutdown.json mtimes.
- If current.json is newer (== no clean shutdown happened after the
last heartbeat), logs a WARNING: 'PREVIOUS RUN DIED ABRUPTLY. Last
heartbeat was Xs before this startup. Last-known state: {...}'
- The warning lands in BOTH stdout AND the persistent diag.log, so
the operator notices it even if they only check one place.
- Stashes the abrupt-death snapshot in previous_run.json for later.
How the operator uses this after a crash:
1. cat /data/diagnostics/current.json -- last known good state
2. cat /data/diagnostics/last_shutdown.json -- did it shut down cleanly?
3. cat /data/diagnostics/last_exception.json -- any unhandled exception?
4. tail -100 /data/diagnostics/diag.log -- the lead-up
If current is newer than last_shutdown and last_exception doesn't
exist: SIGKILL or OOM (uncatchable). Check docker exit code 137
and host dmesg for oom-killer lines.
If last_exception.json exists: a background task crashed. The
traceback in the file names the coro.
If current.json's rss_mb was climbing across heartbeats: memory
leak / OOM trajectory. Bound the cause to whatever was active.
If current.json's db_pool checked_out was climbing: connection leak.
Look for code paths opening async_session() without exiting
'async with'.
If curator_busy=true across multiple heartbeats: curator hung on
Ollama. Restart Ollama or the Scribe stack to release the lock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
eb02603092 |
feat: diagnostic instrumentation for crash investigation
Recurring app/db crashes with no clear cause in existing logs. Adds three crash-class indicators with minimal overhead (~1 log line/min, 0.1ms work per heartbeat). services/diagnostics.py: 1. **Heartbeat** every 60s logs a snapshot: - RSS memory (from /proc/self/status — no deps). - asyncio task count. - DB pool: size / checked_in / checked_out / overflow. - Curator busy state (from is_curator_running()). - Uptime. A sudden silence in heartbeats bounds the crash time to within 60s. The last snapshot before silence usually rules in or out: memory growth -> OOM, pool exhaustion -> connection leak, hung curator -> stuck async task. 2. **Signal handler** for SIGTERM/SIGINT logs the signal name + final snapshot before letting Hypercorn handle the actual shutdown. Distinguishes 'orderly shutdown via signal X' from 'silent log gap then container exit code 137' (SIGKILL / OOM-kill are uncatchable; their absence in our log IS the diagnostic). 3. **Asyncio exception hook** logs full tracebacks for unhandled task exceptions with the task/coro name. Default behaviour swallows these silently — exactly the pattern that locked us out of chat at 409 for an hour back on 2026-05-22 before we added the guard around run_generation. app.py wires start_diagnostics() into before_serving and stop_diagnostics() into after_serving. stop_diagnostics emits one final snapshot so the silence that follows is intentional, not a crash. How to use the new logs to diagnose: - App restarts with 'received SIGTERM' in the last lines: Orderly shutdown (docker stop / swarm restart / manual). Look upstream for who issued it. - App restarts with no shutdown line, last heartbeat 30+s before: Likely SIGKILL — OOM-kill or container resource limit. Check 'docker ps -a' for exit code 137, or 'dmesg | grep -i kill' on host. - App restarts with no shutdown line, heartbeat showed climbing RSS: Memory leak. Snapshot the last heartbeat's MB value vs earlier — if it doubled over hours, OOM is the cause. - App restarts, db_pool checked_out kept growing: Connection leak. Look for code paths that open async_session() but never exit the 'async with' block. - App seemed alive but stopped responding to requests, heartbeats continued: Curator hung holding _CURATOR_RUN_LOCK. Check curator_busy=true across multiple heartbeats — if stuck >5min, the Ollama call hung. Restart Ollama or the Scribe stack. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |