Structural fix for the "end before start" bug class observed on prod
2026-04-29. Bad data became inexpressible at the schema level instead
of getting trapped in defensive read-path filters.
The hotfix that landed earlier today (94b169f) is reverted by the
preceding revert commit; this commit supersedes it cleanly with a
proper data-model change.
## Schema (migration 0043)
- Add `duration_minutes INTEGER NULLABLE` column on `events`.
- CHECK constraint: ``duration_minutes IS NULL OR duration_minutes >= 0``.
- Backfill from existing `end_dt`:
- end_dt valid (end > start) → duration_minutes = total minutes
- end_dt == start → duration_minutes = 0 (zero-duration point)
- end_dt NULL or end_dt < start → duration_minutes = NULL
(the corrupt prod row collapses cleanly to a point event)
- Drop the `end_dt` column. The wire format is preserved — `to_dict()`
emits `end_dt` as a derived `start_dt + duration_minutes`. Existing
API consumers (Flutter app, web frontend, CalDAV sync) keep
receiving the same response shape; they just no longer have a way
to PUT a stored `end_dt` that disagrees with `start_dt`.
## Service layer
- `Event.end_dt` becomes a `@property`. Setting it would require a
setter we deliberately don't define — writes always go through
`duration_minutes`.
- `_normalize_duration` is the single source-of-truth for input
reduction. Accepts (start, end_dt, duration_minutes), returns the
canonical `duration_minutes`, raises `ValueError` for negative
durations, end-before-start, or end/duration disagreement.
- `create_event` and `update_event` accept either `end_dt` or
`duration_minutes` for ergonomic compat; both convert via
`_normalize_duration`. Update validates the post-update state when
the patch includes either.
- `list_events` filter is simpler now: a coarse SQL prefilter
(`start_dt <= date_to`) plus Python-side refinement using the
derived `end_dt`. Avoids Postgres-specific interval arithmetic in
the WHERE clause; refinement runs over a per-user result set so
there's no scan-cost concern at personal scale.
- Recurring-event expansion uses `event.duration_minutes` directly
instead of computing `end - start`. No more negative-timedelta
hazard.
## CalDAV sync (incoming + outgoing)
- `caldav_sync.py` (pull) and `calendar_sync.py` (Radicale upsert)
both convert iCal `DTEND` → `duration_minutes` on the way in.
Outbound iCal still emits `DTEND` as `start_dt + duration_minutes`
via the model's derived property. iCal interop is unchanged.
## Behavioral upgrade for `update_event`
Pure end_dt model: moving start past the existing end_dt would either
silently corrupt or hard-reject. Duration model: the duration is
preserved by default, so moving start slides the effective end
forward — which is what users mean when they "move" an event.
Explicit clear is still possible via `end_dt=None`.
## Tests
`tests/test_events_service.py`:
- 6 new `_normalize_duration` unit tests (sugar conversion, zero
duration valid as point event, end-before-start rejected, negative
duration rejected, inconsistent end+duration rejected, none → None)
- New behavioral test: `update_event` preserves duration when only
start_dt changes (sliding semantics)
- New: clearing `end_dt=None` on update collapses to point event
- New: list_events surfaces a point event in the upcoming window
- New: list_events excludes a timed event whose effective end has
already passed
- Existing mock-event helper updated to use `duration_minutes`
instead of stored `end_dt`.
44 event-related tests pass; ruff clean.
## Out of scope (separate task)
Fable #161 — `find_events_by_query` returning multiple matches and
silently picking matches[0]. The exact root cause of how event id=2
got mutated in the first place; orthogonal to the storage model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes the entire RSS feature surface — feeds, items, embeddings, reactions,
discussion-note flow, briefing news context, settings, env-vars, and DB
tables. Keeps the URL-generic article-reader (the read_article LLM tool)
under a clean module so the LLM can still fetch arbitrary article content
from URLs the user provides.
Backend:
- New services/article_fetcher.py — single source of trafilatura URL→text
- New services/tools/article.py — read_article tool (was nested under tools/rss)
- Delete services/rss.py, rss_classifier.py, rss_filtering.py, article_context.py
- Delete services/tools/rss.py
- Delete models/rss_feed.py (RssFeed, RssItem), models/rss_item_embedding.py
- services/embeddings.py: drop upsert/semantic_search/backfill RSS helpers
- services/llm.py: remove _build_briefing_article_context, briefing-conv branch,
ARTICLE_DISCUSS_SEED skip-RAG branch; drop get_rss_items / add_rss_feed from
the actions list
- services/generation_task.py: drop _maybe_save_article_discussion_note + caller
- routes/chat.py: drop /api/chat/from-article/<id> endpoint
- routes/journal.py: re-import via web.py refactor (article_fetcher path)
- services/tools/__init__.py: register `article`, drop `rss`
- services/tools/_registry.py: drop the requires=='rss' check
- app.py: drop backfill_rss_item_embeddings + backfill_rss_article_content tasks
- config.py: prose-only edit (no env var change — RSS env vars were never first-class)
Frontend:
- stores/settings.ts: drop rssEnabled
- SettingsView.vue: drop the RSS-classification mention
- api/client.ts: drop openArticleInChat (the from-article endpoint is gone)
Tests:
- Delete tests/test_rss_service.py, test_news_api.py, test_article_reading.py
Migration:
- 0042_drop_rss: DROP TABLE rss_item_embeddings, rss_item_reactions, rss_items,
rss_feeds; DELETE settings rows for rss_enabled / briefing_*_topics
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Also rename services/tz.user_briefing_date → user_day_date with a backwards
compat alias (briefing modules using the old name will be deleted in the
upcoming briefing tear-down stage). Update services/chat.py to_dict to use
day_date.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both the /news discuss button and the briefing discuss button now call a
shared seed_article_discussion() helper that stages the synthetic
read_article tool exchange and the conversational seed prompt — behavior
stays byte-identical across entry points. /news also auto-starts
generation so the chat screen lands on an in-flight stream.
First assistant reply in a seeded article conversation is persisted as a
Note (tags: article-summary + article topics) and backlinked via
rss_items.discussion_note_id, so the knowledge base stops being amnesiac
about articles the user has engaged with.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Discuss button on news cards was producing one-shot replies because
the model got the whole trafilatura blob dropped into history with a
canned "summarize and discuss this article" prompt — no length guard, no
prep, no invitation to converse. Large articles got silently truncated by
Ollama; small articles got a tepid reply.
This reworks discuss_article around a three-layer cache:
context_prepared → content_full → fresh trafilatura fetch
First click on a small article fetches once, writes through to both
caches, and passes the body straight into the synthetic read_article
tool-result. First click on a large article additionally runs a parallel
map step (services/article_context.py) that chunks the body on paragraph
boundaries, summarizes each ~8k chunk to ~300 words of dense factual
prose via the background model, and concatenates the summaries under
section headers — all pinned to num_ctx=16384 so the map step doesn't
itself fall victim to silent truncation. Repeat clicks on either path
skip straight to the chat turn.
The canned summary prompt is replaced with a conversational seed that
invites the user into an actual discussion rather than a one-shot
synopsis, matching the goal of "have a conversation about an article,
not just read it."
discuss_topic is intentionally left untouched — it's the multi-article
aggregation path and needs a separate rework. Follow-up task will decide
whether to retire it or rework it on the cached-context approach.
Closes task #106.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Project.to_dict() now includes user_id and auto_summary
- Status validation unified to (active/completed/archived) on both
create and update project routes; update route previously had none
- Milestone routes: replace get_project (ownership-only) with
get_project_for_user so shared viewers/editors can access milestones
- Add get_milestone_in_project() to milestones service for project-
scoped lookup without user_id filter; all milestone routes use it
- Milestone PATCH now validates status as 'active'|'done'; fix tool
enum which was wrongly ['active','completed','cancelled']
- Write mutation routes (POST/PATCH/DELETE milestones) now check
can_write_project() and return 403 for read-only shared users
- update_project tool now exposes title and color fields so projects
can be renamed or recolored via chat
- create_project tool now exposes color field
- GET /api/projects?include_summary=true embeds summaries in one
backend pass; ProjectListView switches to this, eliminating N+1
per-project fetches
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- RRULE expansion: list_events now expands recurring events into
individual occurrences within the query window using python-dateutil
- CalDAV pull sync: new caldav_sync.py + POST /api/events/sync route;
imports remote events into the internal store by caldav_uid
- Past event search: search_events accepts include_past=true to search
historical events; exposed in the LLM tool definition
- Internal reminders: migration 0037 adds reminder_minutes +
reminder_sent_at columns; event_scheduler.py checks every 5 min and
fires push notifications; CalDAV sync job runs hourly
- reminder_minutes now stored and returned in create/update routes + tools
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SQLAlchemy reserves 'metadata' as a class attribute on declarative models.
Renamed to 'entity_meta' with explicit column name 'metadata' so the DB
column is unchanged but the Python attribute no longer conflicts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Embed RSS items at fetch time (nomic-embed-text); backfill at startup
- Semantic news search injected into chat system prompt ("Recent News You've Seen")
when items match query above 0.55 cosine threshold (independent of note RAG)
- "Discuss in chat" button on news cards — creates a seeded conversation with
the article title + full content, navigates directly to the new chat
- Briefing compilation now passes 500-char article excerpts (not just headlines)
to the LLM and uses 8192 num_ctx to accommodate the larger prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the freeform briefing-profile note with a DB-backed user_profiles
table. Users can edit job/industry/expertise/response preferences/interests/
work schedule via a new Settings → Profile tab. The LLM appends nightly
observations; at 14+ entries they are auto-consolidated into a learned_summary.
Profile context is injected into both briefing and chat system prompts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TaskStatus enum was missing 'cancelled' — the LLM tried to use it and
hit TaskStatus("cancelled") raising ValueError → 500. Added the value,
a migration to extend the task_status Postgres enum, and proper 400
validation guards on both create and update task routes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
msg_metadata maps to the 'metadata' DB column ('metadata' is reserved
by SQLAlchemy Declarative API). to_dict() exposes the key as 'metadata'
for frontend compatibility.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add session_version to users table. Sessions now carry session_version
alongside user_id; @login_required rejects any session where the version
doesn't match the DB value.
- migration 0024: session_version INTEGER NOT NULL DEFAULT 1
- models/user.py: session_version Mapped[int] column
- auth.py: version mismatch → session.clear() + 401
- services/auth.py: change_password and reset_password_with_token both
increment session_version, evicting all other live sessions
- routes/auth.py: login/register/oauth_callback store session_version;
update_password rehydrates current session with new version so the
user who changed their password stays logged in
Existing sessions will require one re-login after the migration runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NoteEditorView: two-column sidebar layout (project/milestone/tags/assist
always visible), removed assist toggle button, InlineAssistPanel removed.
Writing assist: whole_doc mode rewrites entire document; DiffView.vue
replaces editor during review showing full-document diff. Scope dropdown
in sidebar switches between whole-document and section modes.
Persistent drafts: migration 0022 adds note_drafts (UNIQUE per note+user)
and note_versions (max 20, auto-pruned) tables. Draft saved after generation
completes, restored on editor mount, cleared on accept/reject. Version
snapshot created automatically whenever note body changes on save.
HistoryPanel.vue: version list + DiffView modal, restore button writes
body back to editor.
Config: OLLAMA_NUM_CTX default raised to 65536; assist num_predict now
tracks Config.OLLAMA_NUM_CTX instead of a hardcoded 4096.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Images found via SearXNG are fetched server-side, stored on disk, and
served from /api/images/<id> — the user's browser never contacts the
original image host. Original URLs are preserved for citation.
New files:
- alembic/versions/0016_add_image_cache.py — image_cache table
- src/fabledassistant/models/image_cache.py — SQLAlchemy model
- src/fabledassistant/services/images.py — fetch/store/serve logic
- src/fabledassistant/routes/images.py — GET /api/images/<id>
Modified:
- config.py: IMAGE_CACHE_DIR (/data/images), IMAGE_MAX_BYTES (5 MB)
- research.py: _search_searxng_images() — SearXNG categories=images
- tools.py: _IMAGE_TOOLS def + search_images branch in execute_tool
- intent.py: search_images routing rule (explicit visual language only)
- app.py: register images_bp
- docker-compose.yml: image_cache named volume mounted at /data/images
- ToolCallCard.vue: "image_search" label
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without pool_pre_ping, stale pooled connections (left over after a Postgres
restart or network blip) cause immediate query failures. SQLAlchemy then
propagates the error rather than transparently reconnecting, which crashes
the Quart request handler and triggers a Swarm restart loop.
- pool_pre_ping=True: issues a lightweight SELECT 1 before each checkout;
discards and replaces stale connections silently
- pool_recycle=1800: recycles connections every 30 minutes to prevent
long-idle connections from going stale at the TCP/firewall level
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New NoteEmbedding model + migration 0014 stores float embeddings (JSONB)
- services/embeddings.py: get_embedding, upsert_note_embedding,
semantic_search_notes (cosine similarity), backfill_note_embeddings
- build_context() now tries semantic search first, falls back to keyword search;
accepts cached_note_ids to reuse last-turn notes and stabilise the system
prompt prefix for Ollama's KV cache
- generation_buffer.py: per-conversation note ID cache (get/set/clear)
- generation_task.py: passes cached IDs into build_context, updates cache
after each turn, and invalidates it after create_note/update_note/create_task
- app.py: pulls nomic-embed-text at startup and launches a background backfill
to embed all existing notes (30 s delay so Ollama has time to load the model)
- routes/notes.py + services/tools.py: fire-and-forget embedding update on
every note create or update via the API or LLM tool calls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ollama tool/function calling integration allows the LLM to create tasks,
create notes, and search existing notes on behalf of the user during chat.
Multi-round tool loop (max 5 rounds) lets the model execute tools then
produce a natural language response. Tool results are persisted in a new
JSONB column on messages and rendered as compact cards with linked titles.
- Migration 0013: add tool_calls JSONB column to messages
- New services/tools.py: tool definitions + execute_tool dispatcher
- llm.py: ChatChunk dataclass, stream_chat_with_tools(), date in system prompt
- generation_task.py: multi-round tool call loop with SSE tool_call events
- Frontend: ToolCallRecord type, streamingToolCalls in store, ToolCallCard
component, rendering in ChatMessage and ChatView streaming bubble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Users who forget their password can now request a reset link via email.
Tokens are SHA-256 hashed before storage, expire after 1 hour, and
previous unused tokens are invalidated on new requests. The forgot-password
endpoint always returns success to prevent email enumeration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 5.4 — Application Logging:
- AppLog model + migration 0010 for unified audit/usage/error logging
- Usage logging middleware in app.py (after_request for /api/* requests)
- Error logging in 500 handler with traceback capture
- Audit logging for auth events (register, login, login_failed, logout,
password_change) and admin actions (backup, restore, user_delete,
registration_toggle, smtp_config, smtp_test)
- Admin log viewer (LogsView.vue) with stats, category/search/date
filters, paginated table with expandable detail rows
- Admin logs API endpoints in admin.py (GET /logs, GET /logs/stats)
- Configurable retention via LOG_RETENTION_DAYS with hourly cleanup
Phase 5.5 — SMTP Email Notifications:
- aiosmtplib dependency for async email sending
- Email service (services/email.py) with STARTTLS/implicit TLS support
- Notification service (services/notifications.py) for security alerts
and task due date reminders with per-user preferences
- Admin SMTP config endpoints (GET/PUT /api/admin/smtp, POST test)
- SMTP config in Config class with env var + Docker secret support
- Settings UI: notification preferences for all users, SMTP config
section for admin with test email
Other changes:
- stream_chat() now accepts optional options dict (for num_predict)
- Increase assist MAX_BODY_CHARS from 3000 to 8000
- get_user_by_username() added to auth service
- apiStreamPost buffer processing refactored for robustness
- AppHeader: admin Logs nav link
- Router: /admin/logs route
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrite get_all_tags() with SQL unnest instead of loading all notes
- Consolidate convert_note_to_task/convert_task_to_note to single-session ops
- Add search_notes_for_context() with single OR-keyword query for build_context()
- Drop selectinload from list_conversations(), use correlated subquery for message_count
- Add set_settings_batch() for single-transaction multi-setting upserts
- Extract get_installed_models() shared helper into services/llm.py
- Delete services/tasks.py pass-through wrapper; routes/tasks.py imports from services.notes
- Add B-tree indexes on notes.title and conversations.updated_at (migration 0007)
- Add logging to services/notes.py, services/chat.py, services/settings.py
- Safe Conversation.to_dict() when messages relationship is not loaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Settings infrastructure: key-value settings table, GET/PUT API, Pinia store
- Configurable assistant name (default "Fable") in settings and LLM system prompt
- Model catalog with 18 models in 3 categories (General Purpose, Coding,
Uncensored / Creative Writing) with download/select/remove functionality
- Move Ollama status indicator from chat views to global nav bar
- Chat bubble layout: user messages right-aligned, assistant left-aligned
- Floating dark input bar with auto-focus and circular send button
- Fix HTML entity rendering (' apostrophe issue in marked/DOMPurify pipeline)
- Fix new chat button navigation (fetchConversation before router.push)
- Recent chats section on home page with "New Chat" button
- Update summary.md with Phase 4.5 changes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 4: Full chat system with SSE streaming, note-aware context, and
conversation persistence.
Backend:
- Migration 0005: conversations + messages tables with FKs and indexes
- Conversation/Message SQLAlchemy models with relationships
- LLM service: ensure_model (auto-pull on startup), stream_chat (NDJSON),
generate_completion, fetch_url_content (HTML stripping), build_context
(keyword extraction, related note search, URL content injection)
- Chat service: conversation CRUD, save_response_as_note,
summarize_conversation_as_note
- Chat routes blueprint: 9 endpoints including SSE streaming for messages,
save/summarize as note, Ollama model listing
- Auto-pull llama3.1 model on app startup (non-blocking)
Frontend:
- apiStreamPost: SSE client using fetch + ReadableStream
- Chat Pinia store with streaming state management
- ChatView: dedicated /chat page with conversation sidebar + message thread
- ChatPanel: slide-out panel with contextNoteId from current route
- ChatMessage: markdown-rendered message bubble with "Save as Note" action
- Updated AppHeader with Chat nav link + panel toggle button
- Updated App.vue to mount ChatPanel with route-derived context
- Added /chat and /chat/:id routes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bvandeusen