Empty FastMCP instance with the post-pivot instructions block. Tools
get registered in phases 2 and 3; ASGI mounting + bearer-auth comes
in task 1.4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous diagnostic instrumentation only wrote to stdout — fine for
'tail the logs while debugging', useless for 'crash happened at 3am
and Docker rotated the logs by morning'. This commit makes the
diagnostic state durable across container restart, OOM-kill, and log
rotation by writing to the mounted /data volume.
Four artifacts in /data/diagnostics/:
- current.json — overwritten atomically every heartbeat. Holds the
last known good snapshot (rss, asyncio_tasks, db_pool, curator_busy,
uptime, pid). Post-crash, this file alone tells you what the app
was doing 0-60 seconds before it died. Atomic write (tmp+rename)
so a crash mid-write can't leave a half-written file.
- last_shutdown.json — written when SIGTERM/SIGINT is caught OR
after_serving fires cleanly. If this file's mtime is older than
current.json's, the previous run died WITHOUT calling shutdown
(== SIGKILL, OOM-kill, or container hard-stop).
- last_exception.json — written when the asyncio exception hook
fires. Includes task name, coro name, exception type and message
alongside the resource snapshot.
- diag.log + diag.log.1..5 — rotating file log (10 MB × 5 backups
= 50 MB cap) containing every heartbeat, signal, and exception.
Separate from the app's stdout logger so Docker log rotation
can't take it out.
- previous_run.json — written at startup IF the post-mortem detects
the previous run died abruptly. Includes the abrupt-death snapshot
preserved for retrospection, so a recurring crash pattern can be
diffed over time.
Post-mortem at startup:
- Reads current.json + last_shutdown.json mtimes.
- If current.json is newer (== no clean shutdown happened after the
last heartbeat), logs a WARNING: 'PREVIOUS RUN DIED ABRUPTLY. Last
heartbeat was Xs before this startup. Last-known state: {...}'
- The warning lands in BOTH stdout AND the persistent diag.log, so
the operator notices it even if they only check one place.
- Stashes the abrupt-death snapshot in previous_run.json for later.
How the operator uses this after a crash:
1. cat /data/diagnostics/current.json -- last known good state
2. cat /data/diagnostics/last_shutdown.json -- did it shut down cleanly?
3. cat /data/diagnostics/last_exception.json -- any unhandled exception?
4. tail -100 /data/diagnostics/diag.log -- the lead-up
If current is newer than last_shutdown and last_exception doesn't
exist: SIGKILL or OOM (uncatchable). Check docker exit code 137
and host dmesg for oom-killer lines.
If last_exception.json exists: a background task crashed. The
traceback in the file names the coro.
If current.json's rss_mb was climbing across heartbeats: memory
leak / OOM trajectory. Bound the cause to whatever was active.
If current.json's db_pool checked_out was climbing: connection leak.
Look for code paths opening async_session() without exiting
'async with'.
If curator_busy=true across multiple heartbeats: curator hung on
Ollama. Restart Ollama or the Scribe stack to release the lock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Recurring app/db crashes with no clear cause in existing logs.
Adds three crash-class indicators with minimal overhead (~1 log
line/min, 0.1ms work per heartbeat).
services/diagnostics.py:
1. **Heartbeat** every 60s logs a snapshot:
- RSS memory (from /proc/self/status — no deps).
- asyncio task count.
- DB pool: size / checked_in / checked_out / overflow.
- Curator busy state (from is_curator_running()).
- Uptime.
A sudden silence in heartbeats bounds the crash time to within
60s. The last snapshot before silence usually rules in or out:
memory growth -> OOM, pool exhaustion -> connection leak, hung
curator -> stuck async task.
2. **Signal handler** for SIGTERM/SIGINT logs the signal name +
final snapshot before letting Hypercorn handle the actual
shutdown. Distinguishes 'orderly shutdown via signal X' from
'silent log gap then container exit code 137' (SIGKILL / OOM-kill
are uncatchable; their absence in our log IS the diagnostic).
3. **Asyncio exception hook** logs full tracebacks for unhandled
task exceptions with the task/coro name. Default behaviour
swallows these silently — exactly the pattern that locked us
out of chat at 409 for an hour back on 2026-05-22 before we
added the guard around run_generation.
app.py wires start_diagnostics() into before_serving and
stop_diagnostics() into after_serving. stop_diagnostics emits one
final snapshot so the silence that follows is intentional, not a
crash.
How to use the new logs to diagnose:
- App restarts with 'received SIGTERM' in the last lines:
Orderly shutdown (docker stop / swarm restart / manual). Look
upstream for who issued it.
- App restarts with no shutdown line, last heartbeat 30+s before:
Likely SIGKILL — OOM-kill or container resource limit. Check
'docker ps -a' for exit code 137, or 'dmesg | grep -i kill' on host.
- App restarts with no shutdown line, heartbeat showed climbing
RSS: Memory leak. Snapshot the last heartbeat's MB value vs
earlier — if it doubled over hours, OOM is the cause.
- App restarts, db_pool checked_out kept growing: Connection leak.
Look for code paths that open async_session() but never exit
the 'async with' block.
- App seemed alive but stopped responding to requests, heartbeats
continued: Curator hung holding _CURATOR_RUN_LOCK. Check
curator_busy=true across multiple heartbeats — if stuck >5min,
the Ollama call hung. Restart Ollama or the Scribe stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated changes per operator request 2026-05-24:
1. Settings UI rename matching the language we actually use:
- Chat Model -> Chat & Voice Model
- Worker Model -> Curator Model
Setting KEYS (default_model / background_model) unchanged on
purpose; renaming them requires a migration touching 50+ call
sites for purely UX-facing benefit.
2. Settings UI help text rewritten:
- Chat & Voice: documents that it handles chat AND small
conversational automations (titles, tags). Recommends
OLLAMA_NUM_PARALLEL=2+ on the Ollama server so background
automations get their own KV-cache slot and don't evict
the chat model's working state.
- Curator: notes the app enforces SERIAL execution regardless
of NUM_PARALLEL — only one curator pass runs at a time. This
matters most for 70b CPU models where a second instance
would waste system RAM.
3. Enforce serial curator execution globally:
- New module-level _CURATOR_RUN_LOCK in services/curator.py.
- run_curator_for_conversation now wraps its body in 'async
with _CURATOR_RUN_LOCK' — every entry point (scheduler sweep,
manual route trigger, future hooks) is serialized through it.
- is_curator_running() helper exposes the lock state.
- routes/journal.py manual trigger checks is_curator_running()
first and returns 409 {busy: true} immediately rather than
blocking the HTTP request for minutes waiting for a 70b CPU
pass to finish. The user can retry once the curator clears.
Why a 409 instead of queue: a curator pass on a 70b CPU model
can take 5+ minutes. Tying up an HTTP worker that long is bad;
making the user wait without feedback is worse. 409 surfaces
the busy state immediately and the user retries when they want.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three prompt fixes addressing real failure modes observed in dev
journal data (conv 312, May 23):
curator.py — JOURNAL_CALIBRATION:
1. Strengthen the one-call-per-beat rule. Previous wording said 'do
not collapse multiple beats' but didn't explicitly forbid the
reverse: multiple record_moment calls for the SAME beat with
different phrasings. Observed in moments 7+8, 9+10, 11+14, 12+15,
13+16 — same content captured twice within a single curator pass.
New rule: explicit 'EXACTLY ONE tool call per distinct beat', plus
a 'check whether you already recorded this beat this turn' step.
2. Rewrite the save_person/save_place guidance. Previous wording
over-emphasized 'better to skip than invent' to the point that
the curator ignored explicit user introductions like 'my father's
name is Dale and my mother's name is Lynn, we went to Olive Garden'
— no save_person for Dale or Lynn, no save_place for Olive Garden.
The conservative-skip rule should apply to AMBIGUOUS mentions
('a friend told me'), not to explicit introductions. New rule
spells this out with positive examples.
journal_prep.py — _PREP_SYSTEM_PROMPT:
Extend the no-invent guards. The existing rule covered weather
specifically; today's prep added new fabrications:
- 'tasks due today include X' when tasks_due_today is empty and X is
actually 64 days overdue
- 'at 1:00 PM' when no time exists in the data
- 'currently in progress' applied to tasks where status is 'todo'
Three new rules: (a) never invent a task's due status — frame by the
bucket it actually appears under; (b) never invent times of day —
tasks have dates, not times; (c) never paraphrase a task's status
to something the data doesn't say.
journal_pipeline.py — JOURNAL_CALIBRATION:
1. Promote the one-question rule from buried bullet to top of the
prompt, with stronger phrasing ('ONE question per reply, MAXIMUM
... if you find yourself writing a second question mark, delete
it'). Observed: 3 questions per reply in every conv 312 assistant
turn ('how was it? what'd you order? did she enjoy it?').
2. Add explicit no-fishing rule: don't ask the user to share pictures,
send details, fetch information for the model. Reacts to what they
actually said, not what they didn't. Observed: 'do you have any
pictures you can share?' on msg 789.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chat and background model roles effectively swapped during the
conversation+curator pivot, but call sites still used OLD routing.
This commit re-routes each call to the model whose new role fits.
Moved to background_model (worker — heavy, deliberate):
- services/journal_prep.py: daily prep generation.
- services/user_profile.py: observation consolidation.
Moved to default_model (chat — small, fast):
- services/chat.py save_response_as_note: note title generation.
- services/tag_suggestions.py: tag suggestions.
Already routed correctly (unchanged): curator, closeout, consolidation,
project summaries, history summarization.
SettingsView.vue: help text rewritten for both model fields to
describe new roles. Background Model UI label renamed to Worker
Model so the heavier role is visible from the picker. Warning copy
updated to recommend OLLAMA_MAX_LOADED_MODELS=2+ so chat and worker
can stay loaded simultaneously.
Schema names default_model and background_model unchanged on purpose
(renaming requires migration + touches ~50 call sites for UX-only gain).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTTP surface for the review queue. Three endpoints, all under
the existing /api/journal blueprint to keep the journal-related routes
together:
- GET /api/journal/pending — list current user's pending actions.
- POST /api/journal/pending/<id>/approve — replay the proposed tool
call via execute_tool(authority='user'). On success, marks
the row 'approved'; on replay error, leaves it pending so
the user can retry.
- POST /api/journal/pending/<id>/reject — marks 'rejected' with no
execution.
Each route is a thin wrapper around services/pending_actions and
delegates user-scoping to the service (which checks user_id on every
load — actions are private to the proposer).
api/client.ts:
- PendingCuratorAction interface mirroring the backend dict shape:
id, user_id, conv_id, action_type, target_type/id/label, payload,
current_snapshot, status, timestamps.
- listPendingActions / approvePendingAction / rejectPendingAction
helpers for the upcoming Needs Review panel.
C5 next: the panel itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The interceptor that closes the loop on the curator review queue.
With this commit, the curator can call update_note / update_milestone
/ update_project / update_profile / delete_note — those calls are
caught by execute_tool's authority='curator' path, snapshotted, and
written to pending_curator_actions for the user to approve or reject
later. Additive tools still run immediately.
services/tools/_registry.py:
- New _CURATOR_MUTATING_TOOLS frozenset: {update_note, update_milestone,
update_project, update_profile, delete_note}. update_event /
delete_event intentionally excluded — calendar events should always
be explicit user intent.
- execute_tool gains a keyword-only parameter, defaulting
to 'user'. Default behaviour is unchanged; existing callers keep
working without changes.
- When authority='curator' AND tool is in _CURATOR_MUTATING_TOOLS,
_queue_for_review captures a snapshot of the target via a per-tool
helper and writes a pending action. Returns {success:true,
pending:true, action_id:N, message:...} so the curator sees the
call as 'completed' for its bookkeeping.
- Per-tool snapshot helpers: _snapshot_note (covers update_note +
delete_note — uses the same fuzzy match update_note_tool uses, so
the snapshot reflects what'd actually be mutated), _snapshot_milestone,
_snapshot_project, _snapshot_profile. Snapshot capture is best-effort
— failure logs but still queues with empty snapshot so a curator
proposal never silently drops.
services/curator.py:
- Allowlist now includes the five mutating tools. They're safe to expose
because execute_tool intercepts them; the curator can propose without
being able to actually mutate.
- The execute_tool call now passes authority='curator'.
- System prompt explicitly authorizes the proposal pattern:
'update_note', 'update_milestone', 'update_project', 'update_profile',
'delete_note' are described as proposing tools that wait for user
approval. 'Don't try to update or delete anything' line removed.
services/pending_actions.py:
- approve() now passes authority='user' on the replay so the curator
interceptor doesn't re-route the replay back into pending and create
an infinite loop.
What's left in the queue:
- C4: API routes (list/approve/reject endpoints).
- C5: Frontend Needs Review panel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The backend foundation for curator-proposed mutations awaiting user
approval. No tools route to this yet — that's C3's job. This commit
just lands the schema and the service API everything else will use.
Migration 0051 — new table:
- id, user_id (CASCADE), conv_id (SET NULL — survives conv deletion).
- action_type (the tool name to replay), target_type/target_id/
target_label (display hints).
- payload (jsonb — the curator's proposed args, replayed verbatim
on approval).
- current_snapshot (jsonb — the target's state at proposal time, so
the review UI can render an honest diff even if other work modified
the entity between proposal and review).
- status ('pending' / 'approved' / 'rejected') + CHECK constraint.
- created_at / reviewed_at.
- Partial index ix_pending_curator_actions_user_pending narrowed to
status='pending' — the Needs Review panel hits this constantly,
history rows just accumulate.
Model: PendingCuratorAction with to_dict() for API serialization.
Service services/pending_actions.py:
- create_pending(...) — called from the curator interceptor (C3).
Accepts an already-fetched current_snapshot so each mutating tool
can capture target state in its own way (notes vs milestones vs
profile have different shapes).
- list_pending(user_id, limit=50) — what the Needs Review panel reads.
- approve(action_id, user_id) — replays via execute_tool and marks
approved on success. Stays pending on replay error so the user
can retry. NOTE: approve passes the request through execute_tool
unchanged for now; C3 will add authority='user' so the upcoming
curator interceptor doesn't re-intercept the replay and loop.
- reject(action_id, user_id) — marks rejected with no execution.
C3 next: wires the curator interceptor (authority='curator' on
execute_tool routes mutating tools to create_pending instead of
running them), adds the mutating tools back to the curator's
allowlist, and updates approve() to pass authority='user'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Layer 2 of the surfacing strategy (per 2026-05-23 design discussion).
The curator already has search_notes / search_journal / search_projects
in its allowlist for entity resolution; this commit just directs it
to use those searches more broadly — to surface relevant past work
that connects to today's beats.
Specifically, the system prompt now instructs the curator to:
- Search for projects/topics/people the user mentions, even when not
strictly needed for record_moment entity linking.
- Weave 1-2 short references to relevant past entries into the final
summary line, when they connect meaningfully to today's beats.
The summary feeds back into the chat model's system prompt on the
next turn (per Phase 3 of the architecture), so the chat model gains
contextual awareness of related past work without needing tools to
retrieve it itself.
Light explicit guardrails in the prompt: don't enumerate (avoid 'found
5 related notes'), don't invent references (only mention what was
actually retrieved), don't force a connection when nothing relevant
turns up.
This is the prompt-only Layer 2. Layer 1 (always-on RAG injection
into chat context) was already in place. Layer 3 (dedicated 'you
might want to revisit' surface in the right rail) is deliberately
deferred until 1+2 are observed in practice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related tightenings to the curator's behavior, both driven by user
questions about scope (2026-05-23):
1. **Tighten the prompt to extract beats only from User: lines.**
The transcript shows each message prefixed with role (User: / Assistant:).
The previous prompt instructed the model to capture beats but didn't
explicitly forbid using Assistant: content as a source. A small or
medium model could read 'It sounds like you had coffee with Sarah'
from an Assistant: line and turn it into a moment, even though that's
the assistant paraphrasing the user — not a user statement.
New prompt explicitly: Only User: lines are journal entries. Assistant:
lines are context for disambiguation only. Never create a record from
content that appears only in Assistant: text.
2. **Additive-only tool allowlist for the curator.**
The curator previously had access to the full journal tool set —
including update_*, delete_*, create_event, set_rag_scope, etc. The
architecture removed tools from the chat for exactly the reason that
confidently-wrong tool calls corrupt user data; the curator faces
the same risk async. Filtering the tool list at curator-time keeps
the boundary tight even if the system prompt fails to dissuade the
model from hallucinated tool names.
New _CURATOR_ALLOWED_TOOLS frozenset includes:
- Additive primary work: record_moment, create_note (handles both
notes and tasks via status), log_work (appends to existing task
timeline — additive on its own row), save_person, save_place,
create_project, create_milestone.
- Read-only helpers needed for entity resolution: search_notes,
search_projects, search_journal, list_tasks, list_projects,
list_milestones, read_note, get_project, get_profile.
Explicitly excluded: every update_*, every delete_*, create_event
(calendar events need explicit user intent, not curator inference),
set_rag_scope, lookup/research_topic/search_images (different
surface entirely).
Two-layer enforcement: the system prompt lists what's available and
forbids the rest, AND the actual tools list passed to Ollama is
filtered to the allowlist. So even if the model hallucinates a
forbidden tool name, the call can't fire — execute_tool returns
'Unknown tool: <name>'.
Bonus cleanup: _format_transcript now skips system and tool-role
messages. They were noise for the curator's task (system prompts
are instructions, tool results are JSON from prior calls). The
narrowed transcript matches the contract the prompt enforces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two-in-one cleanup motivated by the chat hang in dev 2026-05-22.
The crash root cause from the guarded-task traceback:
UnboundLocalError: cannot access local variable 'get_setting'
where it is not associated with a value
File generation_task.py:257, in run_generation
think = (await get_setting(user_id, 'think_enabled', 'false'))...
generation_task.py imports get_setting at module top, but a later
'if voice_mode: from ... import get_setting' block scopes it as a
function-local. When voice_mode=False the local import never runs,
but Python had already flagged get_setting as local for the entire
body — the think_enabled read at line 257 hit UnboundLocalError.
The line itself was dead-weight anyway. With the conversation+curator
architecture: chat ships tools=[] (think on a no-tools pass is pure
latency cost; nothing for the model to reason ABOUT in tool-call
terms), and the curator hardcodes think=False already. The user
setting was a holdover from before the architecture pivot. Removing
it entirely is cleaner than fixing the scoping bug to preserve a
toggle nobody should be using:
- generation_task.py: think hardcoded False. Removed the get_setting
call (which fixes the UnboundLocalError as a side effect).
- SettingsView.vue: dropped the Enable model thinking checkbox, the
thinkEnabled / savingThinkEnabled refs, the saveThinkEnabled
function, and the think_enabled load step.
- Migration 0050: DELETE FROM settings WHERE key='think_enabled'
to clean up any stored rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related reliability fixes.
1. routes/chat.py — guard run_generation against uncaught exceptions.
run_generation is launched with asyncio.create_task(); any exception
raised inside the coroutine is silently swallowed by the event loop,
the buffer stays in GenerationState.RUNNING forever, and every
subsequent POST /api/chat/conversations/<id>/messages returns 409
'Generation already in progress' — locking the user out of the chat
with no log trail.
Observed in dev 2026-05-22: assistant message 768 created at 20:36:59
with status=generating, stayed in that state for an hour+, and four
follow-up message attempts returned 409 instantly. The generation
task hung before any internal log line could fire, so the only
diagnostic was the 409 responses themselves.
Wrap run_generation in _run_generation_guarded() that catches
exceptions, logs with full traceback, transitions the buffer to
ERRORED, emits a final 'done' SSE event so any active stream
client closes cleanly, and marks the assistant message status=error
in the DB. After this, a stuck conversation recovers on its own
the next time the user sends a message — no manual DB poke needed.
2. services/curator_scheduler.py — pass last_curator_run_at as 'since'
to the curator so each sweep only sees messages added after the
previous successful pass.
Previously the scheduler called run_curator_for_conversation(conv_id)
with no 'since' argument, so the curator defaulted to its 24h
lookback window. Within an active journal session that meant every
15-min sweep re-extracted beats from messages already captured
on prior sweeps — producing duplicate moments.
_candidate_conversations() now returns (conv_id, last_curator_run_at)
tuples; _sweep() threads the timestamp through. First-run case
(last_curator_run_at IS NULL) falls back to the curator's default
24h window, which is what we want — process recent backlog on
first contact, then only deltas after.
Manual trigger path (POST /api/journal/curator/run/<conv_id>) is
intentionally NOT changed; it still passes since=None so the
24h re-sweep behaviour is preserved for ad-hoc 'reprocess today'
clicks from the UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two architectural bugs in the conversation+curator rollout that
explain the no-response chat in dev:
1. Journal system prompt still instructed tool calls.
JOURNAL_CALIBRATION instructed the model to CALL record_moment,
search_notes, save_person, etc. — but the chat surface ships tools=[]
per the new architecture. The model received contradictory orders
('use these tools' + 'you have no tools') and produced either empty
output or tool-call-shaped text that gets stripped to empty content,
surfacing as status=error or stuck status=generating messages.
Replaced with a chat-only calibration: ~25 lines focused on tone,
length, anti-coaching, and the load-bearing rule 'never claim to
have done anything for the user' (the curator handles capture
silently and separately). JOURNAL_PERSONA also rewritten to drop
the 'use tools to act on their behalf' line.
2. Pre-warm warmed Config.OLLAMA_MODEL ahead of user's real choice.
_pull_model(Config.OLLAMA_MODEL, warm=True) at boot pushed the
system default (qwen3:latest) into VRAM before _warm_user_models()
ran for each user's actual default_model setting. On a single-GPU
setup the second warm could swap the first out — so the user's
chat model wasn't necessarily resident when their first message
landed. Now we just pull the supporting models without warming
them; only user-configured chat models get warm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The architecture loop closes. Curator extracts beats and writes a
≤240-char summary; the next chat turn loads that summary into the
journal system prompt so the chat model — which has no tools and
cannot retrieve anything itself — gains awareness of recent topics
captured by the curator.
Migration 0049:
- conversations.curator_summary (text, nullable). Last-write-wins; no
history of prior summaries.
models/conversation.py:
- New curator_summary column on Conversation.
services/curator_scheduler.py:
- _stamp_last_run() takes an optional summary kwarg; persists it when
non-empty (clobbering the previous summary). Empty summary keeps
the existing one rather than overwriting useful context with "".
- _sweep() passes result.summary through.
routes/journal.py:
- Manual /api/journal/curator/run/<conv_id> writes curator_summary
alongside last_curator_run_at on success.
services/journal_pipeline.py:
- build_journal_system_prompt() gains an optional `conv_id` param.
When provided, appends a "CURATOR NOTES" block at the end of the
system prompt with the conversation's stored summary. Positioned
after ambient context so the chat model treats it as current
awareness rather than background.
services/llm.py:
- Threads conv_id through to build_journal_system_prompt.
This is the last commit of the conversation+curator architecture
arc (Fable #172):
- Phase 1a (a7002a8): chat=tools[], curator service backend
- Phase 1b (a73dd17): right-rail captures panel + manual trigger
- Phase 2 (83f1676): auto-scheduler every 15 min
- Phase 3 (this): curator summary → chat context feedback loop
Operator can now device-test the architecture end-to-end: have a
journal conversation (model can't lie about tool calls because it
has none), wait for the scheduler or hit "Process captures", see
moments appear in the right rail, then continue the conversation
and notice the chat model staying topic-aware via the summary block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The curator now runs automatically every 15 minutes against any
journal conversation that has user messages newer than its last
curator run. Manual triggers from Phase 1b still work and now also
stamp the timestamp so the scheduler doesn't double-process.
Migration 0048:
- conversations.last_curator_run_at (timestamptz, nullable).
- Partial index ix_conversations_journal_last_curator on the column
filtered to conversation_type='journal'. The scheduler's candidate
query is "journal AND (NULL OR stale)" so an index narrowed to
journal rows is the right shape — index size stays small even on
instances with many non-journal conversations.
models/conversation.py:
- New `last_curator_run_at` column on Conversation. DateTime imported.
services/curator_scheduler.py (new):
- IntervalTrigger every 15 min via BackgroundScheduler (same pattern
as journal_scheduler.py).
- _candidate_conversations(): SELECT journal conversations where the
newest user message is newer than last_curator_run_at (or NULL).
Capped at 20 per sweep so a backlog after downtime doesn't stall
the scheduler.
- _sweep() processes candidates sequentially under an asyncio.Lock
so overlapping ticks can't double-fire on the same conversation.
Failed runs leave the timestamp alone — natural retry on next sweep.
- start_/stop_curator_scheduler() wired into app.py boot/shutdown.
routes/journal.py:
- Manual /api/journal/curator/run/<conv_id> stamps last_curator_run_at
on success. Errors don't stamp so the scheduler retries.
What's still pending:
- Phase 3: feedback loop (curator summary into chat context). Currently
the curator's summary lives in the run result but doesn't reach the
chat model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend half of the conversation+curator architecture (Fable #172).
Decouples the journal chat surface from tool calling: the chat model
now sees `tools=[]` and just talks, while a separate curator pass
extracts beats and fires the tool calls.
services/generation_task.py:
- When conversation_type == "journal", pass `tools=[]` to Ollama
regardless of what the journal tool set would normally provide.
The chat model literally cannot fire record_moment / create_task /
etc., so it cannot lie about firing them — the primary failure
mode this architecture removes.
services/curator.py (new):
- `run_curator_for_conversation(conv_id, since=None)` loads recent
messages, builds a curator-specific system prompt (extract beats,
emit tool calls, optionally a one-line summary), and iterates the
Ollama tool-call loop using the user's background_model so the
chat model's KV cache survives.
- Same tool registry as a normal journal conversation
(record_moment, search_notes, update_task, create_task,
save_person, save_place, etc.). The curator chooses naturally
among them; no need for a separate curator-specific filter.
- Returns CuratorRunResult with per-call status + a summary line.
- Caps at 4 tool-call rounds — bounded task (extract beats from a
fixed transcript), shouldn't need more.
- Errors land in result.error rather than raising; the manual
trigger surface (and later the scheduler) want a structured
result, not exceptions.
routes/journal.py:
- New POST /api/journal/curator/run/<conv_id> for manual triggers.
Validates conv ownership before running. Returns the
CuratorRunResult dict so the UI can show what was captured.
What's not in this commit (deferred to later phases):
- The scheduler that auto-runs the curator (phase 2 — adds the
`conversations.last_curator_run_at` column + APScheduler job).
- Curator → chat feedback loop (phase 3 — summary gets injected
into subsequent chat system prompts).
- Right-rail captures panel in JournalView (phase 1b — pure frontend
work, separate commit for clean review).
- Research surface separation (phase 4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Building on the kokoro→piper swap (B1), this adds the admin-side
voice management story so additional voices can be installed without
rebuilding the image. The bundled two voices stay as immediate defaults;
everything else is opt-in via a one-click install from the catalog.
Backend (services/voice_library.py):
- fetch_catalog() pulls voices.json from the piper-voices HF repo with
a 24h in-memory TTL. Manual refresh available via ?refresh=1 on the
library endpoint.
- shape_catalog_for_ui() projects the raw HF dict (~250 voices, lots of
nesting) into UI-friendly cards: id, name, language, country, quality,
size, install state. Sorted by language_code then name for stable
display. Install state distinguishes bundled (read-only) from user
(admin-installed, can be removed).
- install_voice() downloads .onnx + .onnx.json into /data/voices with
atomic .tmp → rename so a failed partial download can't leave a
corrupt model around. Idempotent — re-installing an already-present
voice is a no-op.
- uninstall_voice() removes /data voices; bundled /opt voices raise
PermissionError (403 at the route layer).
- Strict voice-id regex prevents path traversal in install/uninstall.
Routes (admin-only, since these write to shared /data and affect all
users on the instance):
- GET /api/voice/voices/library
- POST /api/voice/voices/install
- DELETE /api/voice/voices/<voice_id>
Frontend:
- New "Voice Library" section in Settings → Voice, visible only to
admin users. Collapsed by default; expand to load the catalog
on-demand (doesn't hammer HF for non-admins).
- Free-text filter across id, language code, language name, country,
and dataset name. Refresh button forces a catalog re-fetch.
- Per-voice row shows id, language/country/quality/speaker count, size,
and either an Install button, a Remove button (user voices), or a
"bundled" badge (read-only voices in /opt/piper-voices).
- Installs and uninstalls refresh both the library list AND the active
voice picker so the new voice is immediately selectable.
- VoiceLibraryEntry exported from api/client.ts; new client helpers
getVoiceLibrary/installVoice/uninstallVoice.
Tests:
- Pure-transformation unit tests for shape_catalog_for_ui,
_resolve_file_urls, and the voice-id regex (path-traversal coverage).
- DB/network paths (fetch_catalog, install_voice) need a real
environment — left to CI integration tests or device verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kokoro has been stale upstream since April 2025 (`requires_python<3.13`),
which broke the Python 3.14 build. Piper is the active replacement:
maintained by OHF/Home Assistant, depends only on onnxruntime +
pathvalidate (no torch, no spacy, no transformers), and has cp314
support today.
Dockerfile:
- Add `pip install piper-tts` after the STT install.
- Bundle two default voices (en_US-amy-medium, en_US-ryan-medium) into
/opt/piper-voices at build. Additional voices can be downloaded into
/data/voices via the admin UI (separate commit).
- Image add over the STT-only baseline: ~150 MB.
services/tts.py — full rewrite:
- New voice-discovery layer scans /opt/piper-voices + /data/voices for
.onnx + .onnx.json pairs. /data wins over /opt for the same id so
admin-downloaded voices can override bundled defaults.
- Single PiperVoice kept warm; switches via _switch_voice() when the
user changes their voice_tts_voice setting.
- list_voices() returns metadata read from .onnx.json sidecars (label
derived from filename, language, quality, sample_rate).
- synthesise() uses piper's SynthesisConfig; converts kokoro-shaped
`speed` multiplier to piper's `length_scale` (1.0 / speed).
- `voice_blend` parameter accepted but ignored — piper has no blend
equivalent; first entry's voice is used if anything is passed.
- Dropped: HuggingFace commit-hash tracking (~80 lines), the daily
check_for_kokoro_updates task, voice-tensor blending math.
routes/voice.py:
- tts_backend reports "piper" in /api/voice/status.
- /api/voice/voices no longer requires tts_available() — even with
the active voice failed to load, the catalog still lets the user
pick a different one.
- Synthesise request body dropped the voice_blend field; speed and
voice still supported.
alembic 0047_reset_voice_tts_settings:
- Deletes any stored voice_tts_voice (kokoro IDs that don't map to
piper) and voice_tts_blend (no piper equivalent) rows. Both
re-default cleanly on next read.
frontend:
- VoiceBlendEntry type removed from api/client.ts.
- synthesiseSpeech() signature dropped the voiceBlend parameter.
- SettingsView.vue Voice Blend section removed entirely (slider,
preview, slot management). voice_tts_blend save path removed.
- Default voice id changed from "af_heart" to "en_US-amy-medium".
- VoiceEntry gains optional language/quality/sample_rate fields
from the richer piper sidecar metadata.
Voice paths remain lazily guarded — `VOICE_ENABLED=false` (default)
starts the app cleanly regardless of which TTS deps are present.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an empirical surface for evaluating model swaps. One row per
assistant turn captures: model, think_enabled, tools_available,
tools_attempted, tools_succeeded, tools_failed (with error details
as JSONB). Without this, judging whether a new model "actually fires
record_moment when it should" relies on anecdote across user-reported
sessions. With it, the data is queryable directly.
Pieces:
- Migration 0046: generation_tool_log table with user_created and
per-conversation indexes.
- Model: SQLAlchemy GenerationToolLog with to_dict() for plain-dict
consumption outside session scope.
- Service: log_tool_outcomes() normalizes the in-app tool-call shape
(function/result/status) into the split buckets and persists. It
catches its own exceptions — telemetry failure must NEVER affect
the user-facing generation flow. recent_logs() helper for read.
- Integration in run_generation: called once per turn right after
log_generation, fire-and-forget.
- Tests: pure-normalization unit tests using a stub session — no DB
needed in CI. Cover the success/error split, the empty-tool-calls
case, the exception-swallowing contract, and the success=False
edge case where status incorrectly says "success".
No UI for the telemetry yet — internal infrastructure (the operator
is the consumer, not the journal user), which the FabledRulebook
"no UI no ship" explicitly excepts. Query via psql or extend the
Fable MCP later if direct shell access gets tiresome.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chat generation pipeline previously forced think=True unconditionally
to match qwen3's combined think+tools template, locking the system into
that model family. Bench data (2026-05-21, qwen3:30b-a3b/qwen3:32b on
CPU) showed thinking adds 1-2 minutes per turn for unclear quality
benefit — qwen3:30b-a3b even produced more rambling with think on.
This decouples think from the model family by reading a per-user
`think_enabled` setting (default `false`). Non-qwen3 models can now run
through the same pipeline without the silent-generation failure mode
that content-gated thinking would have caused — they just don't think.
qwen3 users who still want thinking can opt in via the Settings UI.
Settings UI:
- New "Enable model thinking" checkbox in General → Assistant section.
- Help text explains the default-off rationale and when to opt in.
- Persists via the existing settings API; no schema migration needed
(Setting is key/value text).
Telemetry to confirm whether this regresses tool-call reliability on
qwen3 (the current model) is in a follow-up commit (generation_tool_log).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prep prose (services/journal_prep.py):
- Emit explicit "WEATHER: none available — do NOT mention weather"
absent-marker so a small model can't invent partly-cloudy/temperature
prose when both configured locations have empty addresses.
- Replace negative-only system rule with positive-anchored guidance
forbidding weather/temp/precip mentions unless a numeric WEATHER
section is present; also bans echoing parenthetical labels verbatim.
- Reword overdue header to "(past their due date, still open — backlog,
not today's work)" and render lines as "was due <date>, N day(s)
overdue" with correct singular/plural. Supersedes the wording noted
in Fable task #159.
- Deterministic fabricated-weather reconciler: low-false-positive regex
detects fabricated weather phrasing; on trip with an empty section,
regenerate once with a corrective. Persistent fabrication logs ERROR
rather than mangling prose.
Journal route (routes/journal.py):
- Override message_count with len(messages) in _day_payload. The chat
path already does this; the journal path was hitting the
Conversation.to_dict() fallback to 0 because messages aren't
eager-loaded on that instance.
Tests:
- tests/test_journal_message_count.py — pins the model-level trap and
the override contract (3 cases).
- tests/test_journal_prep_hardening.py — 11 cases covering the
fabricated-weather reconciler and absent-marker rendering.
- tests/test_journal_prep_filtering.py — updated one stale assertion.
Tracks Fable task #171.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both export paths emit pin_kind and pin_label per note_version row.
Restore reads them via .get() so backups predating the schema still
import cleanly (defaults to None → rolling).
BackgroundScheduler with a single CronTrigger fires scan_all_users_for
_auto_pins via asyncio.run_coroutine_threadsafe (mirrors the journal-
scheduler pattern). Wired into app startup/shutdown alongside the other
schedulers.
_promote_stable_versions_for_note is the pure-function core: walks
versions chronologically and pins any with a >= AUTO_PIN_STABILITY_DAYS
(2-day) gap to the next version (or to now, for the latest). Auto-
generated label describes the stability window.
_scan_one_note loads versions for one note, runs the promotion, commits
mutations to the attached rows, then calls prune_auto_pins to cap the
auto bucket. scan_user_for_auto_pins fans out across the user's notes;
scan_all_users_for_auto_pins is the top-level entrypoint for the cron.
Per-note and per-user errors are caught and logged.
Auto-pinned versions live in their own bucket with MAX_AUTO_PINS=25 cap.
The scan job calls this after each note's promotions complete; the
oldest auto-pinned rows are dropped past the cap. Manual pins and
rolling rows are untouched.
pin_version sets pin_kind='manual' and pin_label on the target row.
Accepts already-pinned rows (promotes auto→manual, updates label).
Labels are capped at PIN_LABEL_MAX_LEN=500 chars; longer values raise
ValueError before any DB access.
unpin_version clears both fields, downgrading the row to rolling. Does
NOT delete — if the row is past the rolling FIFO depth, the next
autosave's prune will drop it.
The DELETE inside create_version now filters pin_kind IS NULL so pinned
rows (auto or manual) aren't counted toward MAX_VERSIONS=50 and aren't
candidates for deletion. Pinned versions live indefinitely regardless
of how heavy rolling autosave traffic gets on the same note.
Spec: docs/superpowers/specs/2026-05-13-note-version-pinning-design.md
- pin_kind: NULL=rolling, 'auto'=stability-scan, 'manual'=user-declared.
- pin_label: NULL for rolling; auto-generated for 'auto'; user-supplied
string for 'manual' (may be NULL).
No backfill — every existing row stays rolling. The daily auto-pin scan
will catch up on the first run after deploy.
New endpoint manually triggers a consolidation pass for a single task.
Bypasses the auto_consolidate_tasks setting since the user is asking
explicitly. Returns the task with the freshly-written body and
consolidated_at timestamp.
Also un-aliases description and body in the create/update task routes
(was: description folded into body as legacy fallback). With separate
fields under the task-as-durable-record design, both flow through as
distinct kwargs to create_note / update_note.
log_work description now mentions that logs feed the task's auto-summary,
nudging the LLM toward specific log content (commands, decisions, failures)
rather than vague entries.
create_note description gains a runbook-shape clause: code blocks, numbered
procedures, and explicit 'save this as a note/runbook' signals should
spawn standalone notes. Task-specific work-in-progress routes to log_work
instead.
create_note tool:
- New 'description' parameter accepted and forwarded to the service.
- When status is set (creating a task), 'body' is dropped before the
service call. Task bodies are owned by the consolidation pipeline.
update_note tool:
- New 'description' parameter; routed through update_fields.
- When the resolved target has is_task=True and 'body' is in the
arguments, the call errors with a message nudging toward log_work or
description. Knowledge notes are unaffected.
HTTP routes (POST/PATCH/PUT /api/notes) accept body freely — the
restriction is only at the LLM tool layer.
log_work tool now invokes maybe_consolidate(reason='log_added') after a
successful create_log. The gate inside the consolidation service handles
threshold + setting checks.
update_note service snapshots old_status before mutation and fires
maybe_consolidate(reason='task_closed') when the status transitions into
'done' or 'cancelled'. Re-saving an already-terminal status doesn't
retrigger — only transitions count.
consolidate_task reads the task title, description (read-only context),
and chronological work logs; builds a prompt via _build_consolidation_prompt;
calls generate_completion with the user's background_model setting; on a
non-empty result, writes back to Note.body, stamps consolidated_at, and
re-runs the embedding pipeline.
Errors are caught and logged. LLM failures leave body untouched so the
next trigger retries cleanly. Per-task asyncio lock prevents simultaneous
passes for the same task.
New services/consolidation.py module with maybe_consolidate() — the
debounced trigger gate. Two reasons:
- log_added: gated by DEFAULT_LOG_THRESHOLD (3) counted since the task's
consolidated_at timestamp.
- task_closed: bypasses the count gate; fires whenever status flips to
done/cancelled.
Both reasons gated by the auto_consolidate_tasks user setting (default
on). Per-task asyncio.Lock prevents two simultaneous passes for the same
task. consolidate_task is a stub here — full implementation in the next
commit.
create_note service accepts a new description kwarg and forwards it to the
Note constructor. PUT/PATCH/POST routes include description in the field
whitelist. update_note already passed **fields through setattr, so the new
column is reachable without touching that signature.
Spec: docs/superpowers/specs/2026-05-13-task-as-durable-record-design.md
- description: user-stated goal / initial context for tasks (NULL for
knowledge notes).
- consolidated_at: timestamp of the most recent auto-summary pass (NULL
until first consolidation).
- Migration 0044 backfills description from body for existing rows where
status IS NOT NULL (i.e. tasks). Body left in place; first consolidation
pass will overwrite it.
CI surfaced three issues:
- 'famous supply project' didn't substring-match 'Famous-Supply Work topics'
because the trailing filler word 'project' blocked the substring tier.
Strip {project, projects} from the query before the substring check.
- SequenceMatcher fallback against `combined` (title + description +
summary) diluted ratios to ~0.5 for plausible matches. Use title
directly; the 0.70 tier already handles description/summary mentions.
- Test patches used patch.object on a consumer module where
list_projects is imported locally — patch the source module instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen