Compare commits

...

15 Commits

Author SHA1 Message Date
bvandeusen dcbe018297 Merge pull request 'Release v26.04.15.1 — Dynamic voice silence threshold + filled mic halo' (#35) from dev into main 2026-04-15 04:38:27 +00:00
bvandeusen 36fb71699b feat(voice): dynamic silence threshold + filled red mic halo
The web silence detector previously ran RMS over getByteFrequencyData
bytes (which are already dB-scaled) and re-log'd the result, producing
numbers that didn't line up with real dBFS. Combined with a static
-40 dB threshold that sat right on top of room ambient, silence
detection rarely fired and the user had to click stop manually.

- Rewrite useSilenceDetector to use getFloatTimeDomainData for honest
  linear RMS → dBFS.
- Silence threshold is now dynamic: track session peak dBFS and treat
  "silent" as 15 dB below peak. Auto-calibrates per mic/room.
- Grace period (1500 ms) at start so the user can begin speaking
  before checks arm; static -45 dB fallback until peak clears -25 dB
  so dead-silent sessions don't spin forever.
- Silence duration bumped 1500 → 2000 ms for breathing room.

Visual: detached red radial-gradient disc sits behind the mic in a
wrapper; the button no longer scales, so the white mic icon stays
legible on top while the halo grows from 1x to ~2.6x with live
amplitude. Much more obvious "you are live" signal than the prior
subtle box-shadow pulse.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 00:26:52 -04:00
bvandeusen 027fbec606 Merge pull request 'Release v26.04.14.3 — Live amplitude mic pulse' (#34) from dev into main 2026-04-15 02:54:42 +00:00
bvandeusen 730dbfaf7b feat(voice): pulse mic button with live amplitude
The mic button now scales and glows proportional to the silence
detector's RMS amplitude instead of sitting static. Gives obvious
feedback that audio is actually being picked up — silence still
breathes via a 0.1 floor, loud input caps at ~1.18x scale so it
doesn't punch through the input bar.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 22:46:37 -04:00
bvandeusen 267a975455 Merge pull request 'Release v26.04.14.2 — Discuss failure mode fixes' (#33) from dev into main 2026-04-15 01:45:36 +00:00
bvandeusen 7bd1548f71 fix(discuss): hard-fail empty articles and skip RAG on seed turn
Discuss flow was hallucinating unrelated content when article
extraction returned empty or RAG pulled in orphan notes that looked
more relevant than the generic seed prompt.

- seed_article_discussion raises EmptyArticleError on empty body;
  briefing and /news routes return 422 instead of staging an empty
  synthetic tool result.
- build_context skips RAG auto-injection when user_message matches
  ARTICLE_DISCUSS_SEED so the article IS the context on turn one;
  follow-up turns keep RAG on.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 18:13:17 -04:00
bvandeusen 9f3b3450fa Merge pull request 'Release v26.04.14.1' (#32) from dev into main 2026-04-14 12:02:13 +00:00
bvandeusen ba90ad8132 feat(article-discuss): unify /news + briefing entry points, persist summaries to RAG
Both the /news discuss button and the briefing discuss button now call a
shared seed_article_discussion() helper that stages the synthetic
read_article tool exchange and the conversational seed prompt — behavior
stays byte-identical across entry points. /news also auto-starts
generation so the chat screen lands on an in-flight stream.

First assistant reply in a seeded article conversation is persisted as a
Note (tags: article-summary + article topics) and backlinked via
rss_items.discussion_note_id, so the knowledge base stops being amnesiac
about articles the user has engaged with.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 07:54:24 -04:00
bvandeusen 9157740069 ci: gate typecheck/lint/test on ref so main merge commits skip work
Forgejo Actions doesn't consistently honor `on.push.branches` as a
filter — the merge commit landing on main after a release PR was still
triggering the full CI workflow on a SHA that had already passed on
dev, burning ~1 minute of runner time for no reason.

The `build` job already had a ref guard, so no duplicate images were
being pushed, but typecheck/lint/test ran unconditionally. Add the
same guard (`refs/heads/dev` or `refs/tags/v*`) to those three so main
pushes trigger the workflow but every job skips immediately with zero
runner time.

Also tightened the build job's tag filter from `refs/tags/` to
`refs/tags/v` for consistency with the new guards and to avoid ever
building from a non-version tag.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 23:40:41 -04:00
bvandeusen fd885c1bc8 Merge pull request 'Release v26.04.13.3' (#31) from dev into main 2026-04-14 03:36:09 +00:00
bvandeusen 8205590f8d feat(briefing): cache + map-reduce article context for rich discuss chats
The Discuss button on news cards was producing one-shot replies because
the model got the whole trafilatura blob dropped into history with a
canned "summarize and discuss this article" prompt — no length guard, no
prep, no invitation to converse. Large articles got silently truncated by
Ollama; small articles got a tepid reply.

This reworks discuss_article around a three-layer cache:

  context_prepared  →  content_full  →  fresh trafilatura fetch

First click on a small article fetches once, writes through to both
caches, and passes the body straight into the synthetic read_article
tool-result. First click on a large article additionally runs a parallel
map step (services/article_context.py) that chunks the body on paragraph
boundaries, summarizes each ~8k chunk to ~300 words of dense factual
prose via the background model, and concatenates the summaries under
section headers — all pinned to num_ctx=16384 so the map step doesn't
itself fall victim to silent truncation. Repeat clicks on either path
skip straight to the chat turn.

The canned summary prompt is replaced with a conversational seed that
invites the user into an actual discussion rather than a one-shot
synopsis, matching the goal of "have a conversation about an article,
not just read it."

discuss_topic is intentionally left untouched — it's the multi-article
aggregation path and needs a separate rework. Follow-up task will decide
whether to retire it or rework it on the cached-context approach.

Closes task #106.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 20:52:00 -04:00
bvandeusen 939b910372 Merge pull request 'Release v26.04.13.2' (#30) from dev into main 2026-04-13 23:00:33 +00:00
bvandeusen 70cea78c2f fix(llm): default generate_completion num_ctx to Config.OLLAMA_NUM_CTX
Non-streaming generate_completion was the only LLM entry point that
didn't default num_ctx — stream_chat and stream_chat_with_tools both
fall back to Config.OLLAMA_NUM_CTX (16384). When a caller omitted the
argument, Ollama silently used the model's default window (~4k on
qwen3) and truncated the prompt.

That footgun was masked by fallback paths in the research pipeline:
_generate_outline's prompt carries ~12 sources × 2000 chars (~6k
tokens) of source material plus a system prompt, so the prompt got
chopped, the model never saw the sources, JSON parsing failed twice,
and run_research_pipeline dropped into the single-note "monolith"
fallback (research.py:251). The user reported chat 215 producing such
a monolith note for a multi-source research topic.

Two-layer fix:
- Default num_ctx to Config.OLLAMA_NUM_CTX inside generate_completion,
  matching the streaming entry points. Any current or future caller
  that forgets the argument stops silently losing input.
- Pin num_ctx=16384 explicitly in _generate_outline and
  _generate_executive_summary with comments pointing at the failure
  mode, so a refactor of the generate_completion default can't
  silently regress the research pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 18:20:58 -04:00
bvandeusen 4e4dbb8783 fix(chat): feed title model raw turns instead of post-build_context messages
_generate_title was receiving the full messages list from build_context,
which prepends RAG snippets, RSS excerpts, URL content, and briefing
article dumps INTO the user-role message string. The role=="user" filter
inside _generate_title then handed that composite blob (capped at 300
chars) to gemma3:4b as "the user's message", so the background model
was titling conversations based on article excerpts instead of what the
user actually typed — producing wildly wrong titles like "Briefing
Profile Preferences & Schedule" for a plain calendar query. See #109.

Pass the raw history + user_content + assistant reply instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 17:15:39 -04:00
bvandeusen e4e1d1da49 fix(tz): interpret calendar and briefing dates in user's local timezone
Two related bugs where the server defaulted naive datetimes to UTC instead
of the configured user timezone, causing all-day events to land on the
previous day and briefings to "disappear" at UTC midnight.

- New services/tz.py helpers: get_user_tz, user_today, user_briefing_date
  (the briefing day flips at 4am local to align with the compilation slot,
  so the 00:00-04:00 local window still shows yesterday's briefing until
  the new one is generated).
- calendar create/list/update tools now parse naive datetimes in the
  user's TZ before converting to UTC for storage, and tool descriptions
  tell the model to pass plain local dates.
- briefing_conversations.get_or_create_today_conversation and the
  reset-today route use user_briefing_date so the in-progress briefing
  doesn't get replaced at 19:00 NY / UTC midnight.
- _run_profile_closeout targets user-local "yesterday" for consistency.

Regression tests added for the TZ helpers and the calendar tool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:35:27 -04:00
22 changed files with 1174 additions and 132 deletions
+23 -9
View File
@@ -1,8 +1,12 @@
# CI runs first; build only proceeds if all checks pass.
#
# Push to any branch: typecheck + lint + test
# Push to dev: gates + build :dev + :<sha>
# Tag v* (release): gates + build :latest + :<sha> + :<version>
# Push to dev: typecheck + lint + test + build :dev + :<sha>
# Tag v* (release): typecheck + lint + test + build :latest + :<sha> + :<version>
#
# main pushes are NOT gated here: a merge to main only happens after
# dev has already passed CI, and the release tag is the sole trigger
# for a production image. Re-running CI on the merge commit just burns
# runner time without changing the outcome.
#
# To cut a release:
# Create a release via the Forgejo UI on main with a v* tag name.
@@ -11,6 +15,13 @@
# PRs aren't triggered on purpose — this is a solo dev→main flow, so
# gating on branch push is already enough.
#
# NOTE on the `if:` guards below: Forgejo Actions does not consistently
# honor `on.push.branches` as a filter — merge commits landing on main
# still trigger the workflow, producing redundant runs on the same SHA
# that was already gated on dev. Every job therefore repeats the ref
# check so main pushes trigger the workflow but every job skips
# immediately (no runner time, no duplicate work).
#
# Required secrets (repo → Settings → Secrets → Actions):
# REGISTRY_USER — your Forgejo username
# REGISTRY_TOKEN — Forgejo PAT with write:packages scope
@@ -18,7 +29,7 @@ name: CI & Build
on:
push:
branches: [dev, main]
branches: [dev]
tags: ["v*"]
paths:
- "src/**"
@@ -51,6 +62,8 @@ env:
jobs:
typecheck:
name: TypeScript typecheck
# Skip on main merge-commit pushes — see workflow header comment.
if: github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/v')
runs-on: ci-runner
steps:
- uses: actions/checkout@v6
@@ -77,6 +90,7 @@ jobs:
lint:
name: Python lint
if: github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/v')
runs-on: ci-runner
steps:
- uses: actions/checkout@v6
@@ -88,6 +102,7 @@ jobs:
test:
name: Python tests
if: github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/v')
runs-on: ci-runner
steps:
- uses: actions/checkout@v6
@@ -118,11 +133,10 @@ jobs:
name: Build & push image
needs: [typecheck, lint, test]
# Build on dev branch pushes and version tag pushes only.
# main branch pushes run the gates for safety but do not build —
# the release tag (v*) is the sole trigger for a production image.
if: |
github.event_name == 'push' &&
(github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/'))
# Mirrors the ref guard on the gate jobs above — main merge-commit
# pushes skip here too, so no production image is ever built from a
# raw main push (only from the v* tag the release creates).
if: github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/v')
runs-on: ci-runner
permissions:
contents: read
@@ -0,0 +1,34 @@
"""Add content_full and context_prepared caches to rss_items.
Revision ID: 0038
Revises: 0037
Create Date: 2026-04-13
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0038"
down_revision = "0037"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("rss_items", sa.Column("content_full", sa.Text(), nullable=True))
op.add_column(
"rss_items",
sa.Column("context_prepared", sa.Text(), nullable=True),
)
op.add_column(
"rss_items",
sa.Column("content_fetched_at", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_column("rss_items", "content_fetched_at")
op.drop_column("rss_items", "context_prepared")
op.drop_column("rss_items", "content_full")
@@ -0,0 +1,38 @@
"""Link rss_items to their discussion-summary note.
Revision ID: 0039
Revises: 0038
Create Date: 2026-04-14
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0039"
down_revision = "0038"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column(
"rss_items",
sa.Column("discussion_note_id", sa.BigInteger(), nullable=True),
)
op.create_foreign_key(
"fk_rss_items_discussion_note_id",
"rss_items",
"notes",
["discussion_note_id"],
["id"],
ondelete="SET NULL",
)
def downgrade() -> None:
op.drop_constraint(
"fk_rss_items_discussion_note_id", "rss_items", type_="foreignkey"
)
op.drop_column("rss_items", "discussion_note_id")
+3 -1
View File
@@ -426,7 +426,9 @@ export async function deleteRssReaction(rssItemId: number): Promise<void> {
return apiDelete(`/api/briefing/rss-reactions/${rssItemId}`);
}
export async function openArticleInChat(itemId: number): Promise<{ conversation_id: number }> {
export async function openArticleInChat(
itemId: number,
): Promise<{ conversation_id: number; assistant_message_id?: number; status?: string }> {
return apiPost(`/api/chat/from-article/${itemId}`, {});
}
+69 -12
View File
@@ -114,6 +114,20 @@ const transcribingVoice = ref(false)
const recorder = useVoiceRecorder()
const silenceDetector = useSilenceDetector()
// Live mic halo. A solid red disc sits behind the mic button and scales
// with `silenceDetector.amplitude` (0..1 RMS, already amplified for
// visibility). The button itself stays put so the mic icon is always
// legible on top. 0.2 baseline breathes on silence; loud speech drives
// the disc out to ~2.6× the button size with a softer radial fade.
const micGlowStyle = computed(() => {
if (!recorder.recording.value) return { display: 'none' }
const pulse = 0.2 + Math.min(silenceDetector.amplitude.value, 1) * 0.8
return {
transform: `translate(-50%, -50%) scale(${1 + pulse * 1.6})`,
opacity: String(0.45 + pulse * 0.4),
}
})
async function toggleVoice() {
if (transcribingVoice.value) return
if (recorder.recording.value) {
@@ -230,23 +244,25 @@ defineExpose({ focus, prefill })
class="input-textarea"
></textarea>
<!-- PTT mic -->
<button
v-if="voiceEnabled"
class="btn-icon btn-mic"
:class="{ 'mic-recording': recorder.recording.value, 'mic-transcribing': transcribingVoice }"
@click.prevent="toggleVoice"
:disabled="transcribingVoice || !store.chatReady"
:title="recorder.recording.value ? 'Click to stop (or wait for silence)' : 'Click to speak'"
:aria-label="recorder.recording.value ? 'Stop recording' : 'Start recording'"
>
<!-- PTT mic (with live amplitude halo behind) -->
<div v-if="voiceEnabled" class="mic-wrapper">
<div class="mic-glow" :style="micGlowStyle" aria-hidden="true"></div>
<button
class="btn-icon btn-mic"
:class="{ 'mic-recording': recorder.recording.value, 'mic-transcribing': transcribingVoice }"
@click.prevent="toggleVoice"
:disabled="transcribingVoice || !store.chatReady"
:title="recorder.recording.value ? 'Click to stop (or wait for silence)' : 'Click to speak'"
:aria-label="recorder.recording.value ? 'Stop recording' : 'Start recording'"
>
<svg v-if="!transcribingVoice" width="17" height="17" viewBox="0 0 24 24" fill="currentColor">
<path d="M12 14c1.66 0 3-1.34 3-3V5c0-1.66-1.34-3-3-3S9 3.34 9 5v6c0 1.66 1.34 3 3 3zm-1-9c0-.55.45-1 1-1s1 .45 1 1v6c0 .55-.45 1-1 1s-1-.45-1-1V5zm6 6c0 2.76-2.24 5-5 5s-5-2.24-5-5H5c0 3.53 2.61 6.43 6 6.92V21h2v-3.08c3.39-.49 6-3.39 6-6.92h-2z"/>
</svg>
<svg v-else width="17" height="17" viewBox="0 0 24 24" fill="currentColor">
<circle cx="12" cy="12" r="8" opacity="0.35"/><circle cx="12" cy="12" r="4"/>
</svg>
</button>
</button>
</div>
<!-- Abort (streaming) or Send -->
<button
@@ -343,9 +359,50 @@ defineExpose({ focus, prefill })
.btn-icon:hover { opacity: 1; }
.btn-icon:disabled { opacity: 0.3; cursor: default; }
.btn-mic.mic-recording { opacity: 1; color: #ef4444; }
.btn-mic.mic-recording {
opacity: 1;
/* White icon sits on top of the red halo so it stays legible at any
pulse size. */
color: #fff;
position: relative;
z-index: 1;
}
.btn-mic.mic-transcribing { opacity: 0.5; }
/* Mic wrapper + live amplitude halo. The glow is a filled red disc
absolutely positioned behind the button, scaled/faded by the
computed `micGlowStyle` so the user gets unmistakable feedback
that their voice is being picked up while the mic icon itself
stays put and readable on top. */
.mic-wrapper {
position: relative;
display: inline-flex;
align-items: center;
justify-content: center;
flex-shrink: 0;
}
.mic-glow {
position: absolute;
top: 50%;
left: 50%;
width: 28px;
height: 28px;
border-radius: 50%;
background: radial-gradient(
circle,
rgba(239, 68, 68, 0.9) 0%,
rgba(239, 68, 68, 0.6) 55%,
rgba(239, 68, 68, 0) 100%
);
transform: translate(-50%, -50%) scale(1);
transform-origin: center;
pointer-events: none;
/* Smooth between amplitude samples (~100ms) so the pulse feels continuous
rather than stepping. */
transition: transform 0.12s ease-out, opacity 0.12s ease-out;
z-index: 0;
}
.note-picker-wrapper { position: relative; }
.note-picker-dropdown {
position: absolute;
+66 -14
View File
@@ -1,15 +1,54 @@
import { ref, readonly } from 'vue'
export interface SilenceDetectorOptions {
thresholdDb?: number // default -40
silenceDurationMs?: number // default 1500
minRecordingMs?: number // default 500
/**
* Absolute fallback silence threshold in dBFS, used during the first
* moments before any real speech has been observed. Once the session peak
* clears `dynamicArmDb` we switch to the dynamic threshold instead.
*/
fallbackThresholdDb?: number // default -45
/** How many dB below the session peak counts as "silent" once armed. */
dropFromPeakDb?: number // default 15
/**
* Session peak must reach this level (dBFS) before dynamic thresholding
* kicks in. Until then the fallback threshold is used so we don't lock
* onto a noise-floor peak.
*/
dynamicArmDb?: number // default -25
/** How long to wait at the start of recording before running silence checks. */
graceMs?: number // default 1500
silenceDurationMs?: number // default 2000
minRecordingMs?: number // default 500
}
/**
* Mic silence detector + live amplitude signal.
*
* Uses `getFloatTimeDomainData()` for honest linear RMS in [-1, 1] space,
* then derives dBFS for the silence threshold. The previous implementation
* ran RMS over `getByteFrequencyData` bytes — those bytes are already a
* dB-scaled quantity, so taking their RMS and re-log'ing it produced
* numbers that didn't line up with real dBFS and made any static threshold
* unpredictable.
*
* Silence threshold is dynamic: we track the session peak dBFS and treat
* "silent" as "current level is at least [dropFromPeakDb] below peak."
* This auto-calibrates to whatever mic and room the user is on. Until the
* peak climbs above [dynamicArmDb] we fall back to a conservative static
* threshold so a quiet room doesn't spin forever. A grace period at the
* start of the recording gives the user time to begin speaking before
* silence checks arm.
*
* `amplitude` is the raw linear RMS scaled ×5 and clamped to 1 so quiet
* speech still visibly moves UI that binds to it.
*/
export function useSilenceDetector(options: SilenceDetectorOptions = {}) {
const {
thresholdDb = -40,
silenceDurationMs = 1500,
fallbackThresholdDb = -45,
dropFromPeakDb = 15,
dynamicArmDb = -25,
graceMs = 1500,
silenceDurationMs = 2000,
minRecordingMs = 500,
} = options
@@ -18,31 +57,43 @@ export function useSilenceDetector(options: SilenceDetectorOptions = {}) {
let intervalId: ReturnType<typeof setInterval> | null = null
let silenceMs = 0
let startedAt = 0
let peakDb = -100
function start(stream: MediaStream, onSilence: () => void): void {
stop()
audioCtx = new AudioContext()
// Some browsers start AudioContext in "suspended" state — resume so
// getByteFrequencyData returns real values instead of all zeros.
// the analyser returns real values instead of all zeros.
audioCtx.resume().catch(() => {})
const source = audioCtx.createMediaStreamSource(stream)
const analyser = audioCtx.createAnalyser()
analyser.fftSize = 256
analyser.fftSize = 1024
source.connect(analyser)
const data = new Uint8Array(analyser.frequencyBinCount)
const samples = new Float32Array(analyser.fftSize)
silenceMs = 0
startedAt = Date.now()
peakDb = -100
intervalId = setInterval(() => {
analyser.getByteFrequencyData(data)
const rms = Math.sqrt(data.reduce((s, v) => s + v * v, 0) / data.length) / 255
amplitude.value = rms
analyser.getFloatTimeDomainData(samples)
let sumSq = 0
for (let i = 0; i < samples.length; i++) sumSq += samples[i] * samples[i]
const rms = Math.sqrt(sumSq / samples.length)
amplitude.value = Math.min(rms * 5, 1)
const db = rms > 0 ? 20 * Math.log10(rms) : -100
if (db < thresholdDb) {
const db = rms > 1e-7 ? 20 * Math.log10(rms) : -100
if (db > peakDb) peakDb = db
const elapsed = Date.now() - startedAt
if (elapsed < graceMs) return
const threshold =
peakDb > dynamicArmDb ? peakDb - dropFromPeakDb : fallbackThresholdDb
if (db < threshold) {
silenceMs += 100
if (silenceMs >= silenceDurationMs && Date.now() - startedAt >= minRecordingMs) {
if (silenceMs >= silenceDurationMs && elapsed >= minRecordingMs) {
stop()
onSilence()
}
@@ -63,6 +114,7 @@ export function useSilenceDetector(options: SilenceDetectorOptions = {}) {
}
amplitude.value = 0
silenceMs = 0
peakDb = -100
}
return { amplitude: readonly(amplitude), start, stop }
+19 -1
View File
@@ -1,6 +1,6 @@
from datetime import datetime, timezone
from sqlalchemy import ARRAY, DateTime, ForeignKey, Index, Integer, Text, UniqueConstraint
from sqlalchemy import ARRAY, BigInteger, DateTime, ForeignKey, Index, Integer, Text, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column, relationship
from fabledassistant.models import Base
@@ -46,6 +46,17 @@ class RssItem(Base):
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
# Truncated to 2000 chars to keep DB size reasonable
content: Mapped[str] = mapped_column(Text, default="")
# Full trafilatura-extracted article body, populated lazily on first
# discuss-click / enrichment pass. Nullable — most items never get this
# cached. Expires naturally with the item (90-day retention).
content_full: Mapped[str | None] = mapped_column(Text, nullable=True)
# Map-reduced conversation-ready context derived from content_full. See
# services/article_context.py — populated on first discuss click so
# repeat clicks skip both the fetch and the LLM map step.
context_prepared: Mapped[str | None] = mapped_column(Text, nullable=True)
content_fetched_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
fetched_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
)
@@ -55,6 +66,13 @@ class RssItem(Base):
classified_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
# Note persisting the first-click discussion summary. Set by the article
# discussion pipeline once the seeded chat completes its first assistant
# reply; links back into RAG so re-discussing the same article lands the
# prior summary in context.
discussion_note_id: Mapped[int | None] = mapped_column(
BigInteger, ForeignKey("notes.id", ondelete="SET NULL"), nullable=True
)
feed: Mapped["RssFeed"] = relationship(back_populates="items")
+20 -23
View File
@@ -330,10 +330,13 @@ async def reset_today_briefing():
driven from the MCP when iterating on prompts. Pair with
``POST /api/briefing/trigger`` to immediately regenerate.
"""
from datetime import date as _date
from sqlalchemy import delete as _delete
today = _date.today()
from fabledassistant.services.tz import user_briefing_date
# User-local briefing day (flips at 4am local), not ``date.today()`` —
# see services/tz.py for rationale.
today = await user_briefing_date(g.user.id)
async with async_session() as session:
result = await session.execute(
select(Conversation).where(
@@ -529,25 +532,21 @@ async def discuss_article(item_id: int):
if get_buffer(conv_id) is not None:
return jsonify({"error": "Generation already in progress"}), 409
from fabledassistant.services.rss import _fetch_full_article
article_content = await _fetch_full_article(item.url) or item.content or ""
# Shared helper handles the three-layer cache (context_prepared →
# content_full → fresh fetch), writes the synthetic read_article tool
# exchange and the conversational seed user prompt into the conversation.
# The /news from-article route calls the same helper so behavior stays
# byte-identical across entry points.
from fabledassistant.services.article_context import (
EmptyArticleError,
seed_article_discussion,
)
# Store synthetic assistant message with read_article tool result
synthetic_tool_calls = [{
"function": "read_article",
"arguments": {"url": item.url},
"result": {
"success": True,
"type": "article_content",
"url": item.url,
"content": article_content,
"truncated": False,
},
}]
await add_message(conv_id, "assistant", "", status="complete", tool_calls=synthetic_tool_calls)
# Store user message
await add_message(conv_id, "user", "Please summarize and discuss this article.")
model = await get_setting(uid, "default_model", "") or ""
try:
discuss_prompt = await seed_article_discussion(conv_id, item, model)
except EmptyArticleError as e:
return jsonify({"error": str(e)}), 422
# Reload conversation with fresh messages to build history
conv = await get_conversation(uid, conv_id)
@@ -569,15 +568,13 @@ async def discuss_article(item_id: int):
else:
history.append(msg_dict)
model = await get_setting(uid, "default_model", "") or ""
assistant_msg = await add_message(conv_id, "assistant", "", status="generating")
buf = create_buffer(conv_id, assistant_msg.id)
asyncio.create_task(run_generation(
buf, history, model,
uid, conv_id, conv.title or "",
"Please summarize and discuss this article.",
discuss_prompt,
))
return jsonify({"assistant_message_id": assistant_msg.id, "status": "generating"}), 202
+56 -25
View File
@@ -510,47 +510,78 @@ async def delete_model_route():
@chat_bp.route("/from-article/<int:item_id>", methods=["POST"])
@login_required
async def create_conversation_from_article(item_id: int):
"""Create a chat conversation seeded with an RSS article's content."""
"""Create a chat conversation seeded for article discussion and auto-run.
Mirrors the briefing ``discuss_article`` route: creates a fresh
conversation, stages the shared synthetic read_article exchange + seed
prompt, then kicks off generation so the client lands on an in-flight
stream. The Flutter and web chat screens reconnect to the running buffer
on mount.
"""
from sqlalchemy import select as _select
from fabledassistant.models import async_session as _async_session
from fabledassistant.models.conversation import Message
from fabledassistant.models.rss_feed import RssItem, RssFeed
from fabledassistant.services.article_context import (
EmptyArticleError,
seed_article_discussion,
)
uid = get_current_user_id()
async with _async_session() as session:
result = await session.execute(
_select(RssItem, RssFeed.title.label("feed_title"))
_select(RssItem)
.join(RssFeed, RssItem.feed_id == RssFeed.id)
.where(RssItem.id == item_id, RssFeed.user_id == uid)
)
row = result.first()
item = result.scalars().first()
if row is None:
if item is None:
return jsonify({"error": "Article not found"}), 404
item, feed_title = row
conv_title = (item.title or "Article discussion")[:80]
conv = await create_conversation(uid, title=conv_title, conversation_type="chat")
from fabledassistant.services.rss import _fetch_full_article
source = feed_title or "News"
content_body = (await _fetch_full_article(item.url) if item.url else None) or (item.content or "").strip()
seeded_text = f"**{source}**\n\n**{item.title}**"
if content_body:
seeded_text += f"\n\n{content_body}"
if item.url:
seeded_text += f"\n\nSource: {item.url}"
model = await get_setting(uid, "default_model", "") or Config.OLLAMA_MODEL
try:
discuss_prompt = await seed_article_discussion(conv.id, item, model)
except EmptyArticleError as e:
# Roll back the empty conversation so the user doesn't end up with a
# phantom entry in their chat list.
try:
await delete_conversation(uid, conv.id)
except Exception:
logger.warning("Failed to clean up empty article conversation %s", conv.id)
return jsonify({"error": str(e)}), 422
async with _async_session() as session:
msg = Message(
conversation_id=conv.id,
role="assistant",
content=seeded_text,
msg_metadata={"rss_item_ids": [item_id]},
)
session.add(msg)
await session.commit()
# Reload conversation so we see the two messages the helper just added.
conv = await get_conversation(uid, conv.id)
assert conv is not None
return jsonify({"conversation_id": conv.id}), 201
history: list[dict] = []
for msg in conv.messages:
if msg.role == "system":
continue
msg_dict: dict = {"role": msg.role, "content": msg.content or ""}
if msg.tool_calls:
msg_dict["tool_calls"] = [
{"function": {"name": tc["function"], "arguments": tc["arguments"]}}
for tc in msg.tool_calls
]
history.append(msg_dict)
for tc in msg.tool_calls:
history.append({"role": "tool", "content": json.dumps(tc.get("result", {}))})
else:
history.append(msg_dict)
assistant_msg = await add_message(conv.id, "assistant", "", status="generating")
buf = create_buffer(conv.id, assistant_msg.id)
asyncio.create_task(run_generation(
buf, history, model, uid, conv.id, conv.title or "", discuss_prompt,
))
return jsonify({
"conversation_id": conv.id,
"assistant_message_id": assistant_msg.id,
"status": "generating",
}), 202
@@ -0,0 +1,270 @@
"""Prepare article bodies as conversation-ready context.
Used by the briefing ``discuss-article`` flow and the ``/news`` discuss button.
A raw trafilatura extraction is often too large to drop whole into a chat
history without eating the context window, so this module runs a map-reduce
step over oversized articles and returns a compact, structured context that
still preserves the article's meaning across sections.
Small articles pass through unchanged — map-reduce only fires when the raw
body exceeds CHAR_BUDGET. The output is cached on ``rss_items.context_prepared``
by the caller, so repeat discuss-clicks on the same article skip this work
entirely.
The module also owns ``seed_article_discussion``, the shared routine that
stages a synthetic ``read_article`` tool exchange plus a conversational seed
prompt into a conversation. Both the briefing and ``/news`` entry points call
it so the two flows stay byte-identical — the only thing that differs between
them is whether the conversation already existed or was freshly created.
"""
from __future__ import annotations
import asyncio
import logging
import re
from fabledassistant.models import async_session
from fabledassistant.models.rss_feed import RssItem
from fabledassistant.services.chat import add_message
from fabledassistant.services.llm import generate_completion
logger = logging.getLogger(__name__)
# ~12k tokens at 4 chars/token. Comfortably under OLLAMA_NUM_CTX=16384
# with room left for system prompt, chat history, and the assistant reply.
CHAR_BUDGET = 48_000
# Chunk size for the map step on oversized articles. Overlap preserves
# context across paragraph boundaries that happen to land mid-sentence.
CHUNK_CHARS = 8_000
CHUNK_OVERLAP = 400
_PARA_SPLIT = re.compile(r"\n\s*\n")
def _chunk_by_paragraph(body: str) -> list[str]:
"""Split ``body`` into chunks of up to CHUNK_CHARS, respecting paragraphs.
Paragraphs longer than CHUNK_CHARS are split mid-paragraph as a last
resort. Adjacent chunks share CHUNK_OVERLAP chars of trailing text so
a sentence straddling the boundary stays readable on both sides.
"""
paragraphs = [p.strip() for p in _PARA_SPLIT.split(body) if p.strip()]
chunks: list[str] = []
current: list[str] = []
current_len = 0
for para in paragraphs:
para_len = len(para)
if para_len > CHUNK_CHARS:
if current:
chunks.append("\n\n".join(current))
current, current_len = [], 0
for i in range(0, para_len, CHUNK_CHARS - CHUNK_OVERLAP):
chunks.append(para[i : i + CHUNK_CHARS])
continue
if current_len + para_len + 2 > CHUNK_CHARS and current:
chunks.append("\n\n".join(current))
tail = current[-1][-CHUNK_OVERLAP:] if current else ""
current = [tail, para] if tail else [para]
current_len = len(tail) + para_len + (2 if tail else 0)
else:
current.append(para)
current_len += para_len + 2
if current:
chunks.append("\n\n".join(current))
return chunks
async def _summarize_chunk(title: str, chunk: str, index: int, total: int, model: str) -> str:
"""Map-step summary of one article chunk.
Aims for ~300 words of dense, factual prose — not bullet points — so the
downstream chat model can quote from it naturally.
"""
messages = [
{
"role": "system",
"content": (
"You are summarizing one section of a larger article so a downstream "
"conversation model can discuss the full article without having to read "
"every word.\n\n"
"Requirements:\n"
"- 250350 words of dense factual prose\n"
"- Preserve specific claims, numbers, names, and quotes\n"
"- Do NOT editorialize or add analysis\n"
"- Do NOT use bullet points or headings\n"
"- Do NOT say 'this section' or 'this article' — write content, not meta"
),
},
{
"role": "user",
"content": (
f"Article: {title}\n"
f"Section {index + 1} of {total}:\n\n{chunk}"
),
},
]
try:
# Pin num_ctx — same rationale as services/research.py:66. A large
# chunk plus system prompt can push well past the default window;
# silent truncation here would drop the tail of the chunk without
# any error, producing a misleading summary.
raw = await generate_completion(
messages, model, max_tokens=600, num_ctx=16384
)
return raw.strip()
except Exception:
logger.warning(
"Article chunk summary failed for section %d/%d of '%s'",
index + 1, total, title, exc_info=True,
)
# Fall back to the raw chunk truncated to ~1500 chars so the overall
# pipeline still delivers something rather than dropping the section.
return chunk[:1500]
async def prepare_article_context(
title: str,
url: str,
body: str,
model: str,
) -> str:
"""Return a conversation-ready context block for ``body``.
- Small article (≤ CHAR_BUDGET): returns ``body`` unchanged.
- Oversized article: runs a parallel map step over paragraph-aware
chunks and concatenates the summaries under section headers.
The returned string is what should go into the ``read_article`` synthetic
tool-result in chat history. Callers are responsible for caching it to
``rss_items.context_prepared``.
"""
body = body or ""
if len(body) <= CHAR_BUDGET:
return body
chunks = _chunk_by_paragraph(body)
logger.info(
"Article '%s' is %d chars, map-reducing into %d chunks",
title, len(body), len(chunks),
)
summaries = await asyncio.gather(
*[
_summarize_chunk(title, chunk, i, len(chunks), model)
for i, chunk in enumerate(chunks)
]
)
header = (
f"(This article was longer than the chat window could hold verbatim, "
f"so the full text was split into {len(chunks)} sections and each was "
"summarized below. Each section preserves specific claims, numbers, "
"and quotes from the original.)\n\n"
)
parts = [
f"## Section {i + 1}\n\n{summary}"
for i, summary in enumerate(summaries)
]
return header + "\n\n".join(parts)
# Conversational seed prompt for article discussions. Kept here so both the
# briefing and /news entry points use the exact same wording. See
# feedback_discuss_prompt_style memory: numbered checklists produce
# assignment-completion responses; this conversational seed opens a dialogue.
ARTICLE_DISCUSS_SEED = (
"I want to talk about this article. Start with a substantive summary "
"of what it's arguing and the key evidence it uses, then tell me what "
"stood out to you or seems worth pushing back on. I'll ask follow-ups "
"from there."
)
class EmptyArticleError(Exception):
"""Raised when an article has no extractable body text.
Callers (the briefing and /news discuss routes) map this to a 422 so the
user sees a clear error instead of a hallucinated summary built from an
empty synthetic tool result.
"""
async def seed_article_discussion(
conv_id: int,
item: RssItem,
model: str,
) -> str:
"""Stage the synthetic read_article tool exchange + conversational seed.
Used by both the briefing ``discuss_article`` route and the ``/news``
``from-article`` conversation creator. Handles the three-layer cache
(``context_prepared`` → ``content_full`` → fresh fetch) and inserts two
messages into ``conv_id``:
1. An assistant message with a synthetic ``read_article`` tool_call whose
``result.content`` carries the prepared article context. The message
also carries ``msg_metadata={"rss_item_id": ...}`` so the post-generation
hook in ``generation_task.py`` can locate it and persist the first
reply as a discussion-summary Note.
2. A user message with the shared conversational seed prompt.
Returns the seed prompt string so callers can pass it to ``run_generation``
as ``user_content``.
"""
# Avoid circulars: rss helper imports article_context indirectly nowhere,
# but keep this local for symmetry with the route-level imports it
# replaces.
from fabledassistant.services.rss import get_or_fetch_full_article
if item.context_prepared:
article_content = item.context_prepared
else:
raw_body = await get_or_fetch_full_article(item) or item.content or ""
if not raw_body.strip():
# Hard-fail rather than stage an empty synthetic tool result.
# An empty `content` field silently tells the model "the article
# has nothing in it" and it confabulates from RAG/history. Better
# to surface a clean error to the user.
logger.warning(
"Article discussion aborted: empty body for rss_item %s (%s)",
item.id, item.url,
)
raise EmptyArticleError(
"Couldn't extract any readable text from this article."
)
article_content = await prepare_article_context(
item.title or "", item.url, raw_body, model,
)
if not article_content.strip():
raise EmptyArticleError(
"Couldn't extract any readable text from this article."
)
async with async_session() as session:
fresh = await session.get(RssItem, item.id)
if fresh is not None:
fresh.context_prepared = article_content
await session.commit()
synthetic_tool_calls = [{
"function": "read_article",
"arguments": {"url": item.url},
"result": {
"success": True,
"type": "article_content",
"url": item.url,
"content": article_content,
"truncated": False,
},
}]
await add_message(
conv_id,
"assistant",
"",
status="complete",
tool_calls=synthetic_tool_calls,
msg_metadata={"rss_item_id": item.id, "article_seed": True},
)
await add_message(conv_id, "user", ARTICLE_DISCUSS_SEED)
return ARTICLE_DISCUSS_SEED
@@ -1,12 +1,13 @@
"""Create and manage briefing conversations."""
import logging
from datetime import date, datetime, timezone
from datetime import datetime, timezone
from sqlalchemy import select
from fabledassistant.models import async_session
from fabledassistant.models.conversation import Conversation, Message
from fabledassistant.services.tz import user_briefing_date
logger = logging.getLogger(__name__)
@@ -15,7 +16,10 @@ async def get_or_create_today_conversation(user_id: int, model: str) -> Conversa
"""
Return today's briefing conversation, creating it if it doesn't exist.
"""
today = date.today()
# "Today" is the user-local briefing day (flips at 4am local), not
# ``date.today()`` — in a UTC container the latter rolls over at
# 19:00 NY local and makes the in-progress briefing disappear.
today = await user_briefing_date(user_id)
async with async_session() as session:
result = await session.execute(
select(Conversation).where(
@@ -445,9 +445,12 @@ async def _run_profile_closeout(user_id: int, model: str) -> None:
"""
from fabledassistant.services.user_profile import append_observations
from fabledassistant.services.llm import generate_completion
from fabledassistant.services.tz import user_today
from fabledassistant.models.conversation import Conversation, Message
yesterday = (date.today() - timedelta(days=1))
# User-local "yesterday" so closeout always targets the day that just
# ended in the user's timezone, regardless of container TZ.
yesterday = (await user_today(user_id)) - timedelta(days=1)
async with async_session() as session:
result = await session.execute(
select(Conversation).where(
+3
View File
@@ -187,6 +187,7 @@ async def add_message(
context_note_id: int | None = None,
status: str | None = None,
tool_calls: list | None = None,
msg_metadata: dict | None = None,
) -> Message:
async with async_session() as session:
kwargs: dict = dict(
@@ -199,6 +200,8 @@ async def add_message(
kwargs["status"] = status
if tool_calls is not None:
kwargs["tool_calls"] = tool_calls
if msg_metadata is not None:
kwargs["msg_metadata"] = msg_metadata
msg = Message(**kwargs)
session.add(msg)
# Touch conversation updated_at
+106 -2
View File
@@ -36,6 +36,85 @@ _TOOL_CALL_MARKER = re.compile(r"^\s*\[TOOL_CALLS\]\s*", re.IGNORECASE)
DB_FLUSH_INTERVAL = 5.0 # seconds between partial DB flushes
async def _maybe_save_article_discussion_note(
user_id: int, conv_id: int, reply_content: str,
) -> None:
"""Persist a seeded article-discussion's first reply as a Note.
Fires after ``run_generation`` completes. Looks for a synthetic
read_article seed message on the conversation; if found AND the linked
``rss_items`` row has no ``discussion_note_id`` yet, saves ``reply_content``
as a Note, tags it, and writes the backlink. Subsequent discuss clicks on
the same article are a no-op (already linked).
Failures are logged and swallowed — the chat UI should never break because
Note persistence hit a snag.
"""
try:
if not reply_content or not reply_content.strip():
return
from sqlalchemy import select as _select
from fabledassistant.models.conversation import Message as _Message
from fabledassistant.models.rss_feed import RssItem as _RssItem
from fabledassistant.services.notes import create_note
async with async_session() as session:
result = await session.execute(
_select(_Message)
.where(_Message.conversation_id == conv_id)
.order_by(_Message.id.asc())
)
messages = result.scalars().all()
seed_meta = None
for m in messages:
meta = m.msg_metadata or {}
if meta.get("article_seed") and meta.get("rss_item_id"):
seed_meta = meta
break
if seed_meta is None:
return
item_id = int(seed_meta["rss_item_id"])
item = await session.get(_RssItem, item_id)
if item is None or item.discussion_note_id is not None:
return
article_title = (item.title or "Untitled article").strip()
article_url = item.url
article_topics = list(item.topics or [])
note_title = f"Article: {article_title}"[:200]
body_parts = [f"**Source:** {article_url}"] if article_url else []
body_parts.append(reply_content.strip())
note_body = "\n\n".join(body_parts)
tags = ["article-summary"] + [t for t in article_topics if t]
note = await create_note(
user_id=user_id,
title=note_title,
body=note_body,
tags=tags,
entity_meta={
"source": "article_discussion",
"rss_item_id": item_id,
"url": article_url,
"conversation_id": conv_id,
},
)
async with async_session() as session:
fresh = await session.get(_RssItem, item_id)
if fresh is not None and fresh.discussion_note_id is None:
fresh.discussion_note_id = note.id
await session.commit()
logger.info(
"Saved article-discussion summary as note %d for rss_item %d (conv %d)",
note.id, item_id, conv_id,
)
except Exception:
logger.warning(
"Failed to persist article-discussion note for conv %d",
conv_id, exc_info=True,
)
# ---------------------------------------------------------------------------
# Thinking decision
# ---------------------------------------------------------------------------
@@ -526,10 +605,35 @@ async def run_generation(
msg_count = len(non_system)
should_gen_title = not conv_title or (msg_count > 0 and msg_count % 10 == 0)
# Persist article-discussion seed conversations as a Note on their
# first assistant reply. This makes "Discuss" summaries part of RAG
# so the knowledge base stops being amnesiac about articles the user
# has already engaged with. The hook detects a seeded conversation by
# finding a synthetic read_article assistant message whose
# msg_metadata carries ``article_seed: True`` and whose rss_items row
# has no discussion_note_id yet. Fire-and-forget so the done event
# lands immediately.
asyncio.create_task(_maybe_save_article_discussion_note(
user_id, conv_id, buf.content_so_far,
))
if should_gen_title:
title_messages = messages + [
{"role": "assistant", "content": buf.content_so_far}
# Feed the title model the *raw* conversation turns only — never
# the post-build_context ``messages`` list. ``build_context``
# prepends RAG snippets, RSS excerpts, URL content, and briefing
# article dumps INTO the user message string itself, so filtering
# by role="user" downstream still surfaces that noise as the
# "user's message". That pollution caused wildly-wrong titles
# (bug #109) — the small background model was staring at article
# excerpts instead of what the user actually typed. Pass the
# original history + the raw user_content + the assistant reply.
title_messages: list[dict] = [
{"role": m["role"], "content": m.get("content") or ""}
for m in history
if m.get("role") in ("user", "assistant")
]
title_messages.append({"role": "user", "content": user_content})
title_messages.append({"role": "assistant", "content": buf.content_so_far})
async def _bg_title() -> None:
try:
+31 -14
View File
@@ -317,9 +317,17 @@ async def generate_completion(
num_ctx overrides the model's context window for this call only.
"""
last_exc: Exception | None = None
options: dict = {"num_predict": max_tokens}
if num_ctx is not None:
options["num_ctx"] = num_ctx
# Default num_ctx to Config.OLLAMA_NUM_CTX (matching stream_chat /
# stream_chat_with_tools). Without this, Ollama silently uses the model's
# default window (~4k on qwen3) and truncates anything longer. That is
# how the research pipeline's outline step kept falling back to a single
# monolith note: its 12-source prompt is ~6k tokens and was being chopped
# before the model ever saw it. Non-streaming callers must not inherit
# that footgun — if you truly want the model default, pass num_ctx=0.
options: dict = {
"num_predict": max_tokens,
"num_ctx": num_ctx if num_ctx is not None else Config.OLLAMA_NUM_CTX,
}
for attempt in range(3):
if attempt > 0:
delay = 3.0 * attempt
@@ -716,18 +724,27 @@ async def build_context(
orphan_only = rag_project_id is None
effective_project_id = rag_project_id if (rag_project_id is not None and rag_project_id != -1) else None
try:
from fabledassistant.services.embeddings import semantic_search_notes
for score, note in await semantic_search_notes(
user_id, user_message, exclude_ids=search_exclude or None, limit=8,
project_id=effective_project_id,
orphan_only=orphan_only,
):
found_scored.append((score, note))
except Exception:
logger.warning("Semantic note search failed, falling back to keyword search", exc_info=True)
# Skip RAG auto-injection on the first turn of a seeded article discussion.
# The article body is already the sole context the user wants — pulling in
# unrelated orphan notes tricks the model into summarizing those instead.
# Follow-up turns keep RAG on because by then the user's own messages drive
# the query rather than the generic seed prompt.
from fabledassistant.services.article_context import ARTICLE_DISCUSS_SEED
_skip_rag_for_article_seed = user_message.strip() == ARTICLE_DISCUSS_SEED
if not found_scored:
if not _skip_rag_for_article_seed:
try:
from fabledassistant.services.embeddings import semantic_search_notes
for score, note in await semantic_search_notes(
user_id, user_message, exclude_ids=search_exclude or None, limit=8,
project_id=effective_project_id,
orphan_only=orphan_only,
):
found_scored.append((score, note))
except Exception:
logger.warning("Semantic note search failed, falling back to keyword search", exc_info=True)
if not found_scored and not _skip_rag_for_article_seed:
keywords = _extract_keywords(user_message)
if keywords:
try:
+18 -2
View File
@@ -63,7 +63,17 @@ async def _generate_outline(topic: str, sources: list[dict], model: str) -> list
]
for attempt in range(2):
try:
raw = await generate_completion(messages, model, max_tokens=400)
# Pin num_ctx explicitly. The prompt carries up to 12 sources at
# 2000 chars each (~6k tokens of source material alone) plus the
# system prompt — well over Ollama's default model window on
# qwen3. Without this, Ollama silently truncates the prompt, the
# model can't see most of the sources, JSON parsing fails twice,
# and the pipeline falls back to a single monolith note
# (`research.py:251`). Do not remove even if `generate_completion`
# appears to default this — see the comment there.
raw = await generate_completion(
messages, model, max_tokens=400, num_ctx=16384
)
raw = raw.strip()
raw = re.sub(r"^```(?:json)?\s*", "", raw)
raw = re.sub(r"\s*```$", "", raw)
@@ -161,7 +171,13 @@ async def _generate_executive_summary(
},
]
try:
raw = await generate_completion(messages, model, max_tokens=600)
# Pin num_ctx explicitly — see `_generate_outline` comment for the
# rationale. This prompt carries N sections × 1500 chars of section
# prose, which can easily exceed the default model window. Don't
# trust the `generate_completion` default to stick.
raw = await generate_completion(
messages, model, max_tokens=600, num_ctx=16384
)
return raw.strip()
except Exception:
logger.warning("Executive summary generation failed for '%s'", topic, exc_info=True)
+33
View File
@@ -34,6 +34,34 @@ def _html_to_text(html: str) -> str:
return html
async def get_or_fetch_full_article(item: RssItem) -> str | None:
"""Return the full article body, fetching+caching on miss.
Checks ``item.content_full`` first — populated either by the enrichment
pass at feed-ingest time or by a previous discuss-click. On miss, fetches
via ``_fetch_full_article`` and writes through. Returns ``None`` only if
the fetch itself fails; ``item.content_full == ""`` is still a cache hit.
Callers must pass an RssItem attached to an open session if they want
the write-through to persist — otherwise the fetched text is returned
but the cache stays empty and the next click will re-fetch.
"""
if item.content_full is not None:
return item.content_full
if not item.url:
return None
text = await _fetch_full_article(item.url)
if text is None:
return None
async with async_session() as session:
fresh = await session.get(RssItem, item.id)
if fresh is not None:
fresh.content_full = text
fresh.content_fetched_at = datetime.now(timezone.utc)
await session.commit()
return text
async def _fetch_full_article(url: str) -> str | None:
"""Fetch a URL and extract its main article text via trafilatura.
@@ -209,6 +237,11 @@ async def fetch_and_cache_feed(feed_id: int, url: str) -> int:
item = await session.get(RssItem, item_id)
if item:
item.content = full_text
# Populate the discuss-click cache too so the
# first click skips straight to the map-reduce
# step without re-fetching.
item.content_full = full_text
item.content_fetched_at = datetime.now(timezone.utc)
await session.commit()
await upsert_rss_item_embedding(
item_id, feed_user_id, item.title or "", item.content
+51 -26
View File
@@ -14,15 +14,37 @@ from fabledassistant.services.events import (
)
from fabledassistant.services.tools._helpers import resolve_project
from fabledassistant.services.tools._registry import tool
from fabledassistant.services.tz import get_user_tz
async def _parse_datetime_in_user_tz(
user_id: int, value: str
) -> tuple[datetime, bool]:
"""Parse a date/datetime string from the model into a UTC-aware datetime.
Naive inputs are interpreted in the **user's local timezone** and then
converted to UTC for storage. Never default to UTC for naive inputs —
that's how all-day events landed on the wrong day for non-UTC users.
Returns ``(utc_datetime, was_date_only)``.
"""
was_date_only = "T" not in value and " " not in value
if was_date_only:
value = f"{value}T00:00:00"
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
if dt.tzinfo is None:
user_tz = await get_user_tz(user_id)
dt = dt.replace(tzinfo=user_tz)
return dt.astimezone(timezone.utc), was_date_only
@tool(
name="create_event",
description="Create a calendar event for the user. Use this when the user asks to schedule, add, or create a meeting, appointment, or event.",
description="Create a calendar event for the user. Use this when the user asks to schedule, add, or create a meeting, appointment, or event. Pass dates and datetimes in the user's local time — a bare date like '2026-09-30' is interpreted as that day in the user's configured timezone.",
parameters={
"title": {"type": "string", "description": "A descriptive event title"},
"start": {"type": "string", "description": "Start date (YYYY-MM-DD) or datetime (ISO 8601 with timezone offset)"},
"end": {"type": "string", "description": "Optional end date or datetime"},
"start": {"type": "string", "description": "Start date (YYYY-MM-DD) or datetime (YYYY-MM-DDTHH:MM) in the user's local time. Include a timezone offset only if the user explicitly mentions one."},
"end": {"type": "string", "description": "Optional end date or datetime in the user's local time"},
"duration": {"type": "integer", "description": "Optional duration in minutes (default 60, ignored if end is set or all_day is true)"},
"description": {"type": "string", "description": "Optional event description"},
"location": {"type": "string", "description": "Optional event location"},
@@ -40,23 +62,21 @@ async def create_event_tool(*, user_id, arguments, **_ctx):
start_str = arguments["start"]
end_str = arguments.get("end")
all_day = arguments.get("all_day", False)
if "T" not in start_str and " " not in start_str:
all_day = True
start_str = f"{start_str}T00:00:00"
try:
start_dt = datetime.fromisoformat(start_str)
if start_dt.tzinfo is None:
start_dt = start_dt.replace(tzinfo=timezone.utc)
# Naive dates/datetimes are interpreted in the user's local
# timezone, not UTC. Storing UTC for naive inputs caused all-day
# events to land on the previous day for negative-offset users.
start_dt, start_was_date_only = await _parse_datetime_in_user_tz(
user_id, start_str
)
except (ValueError, TypeError):
return {"success": False, "error": f"Invalid start datetime: {start_str!r}"}
if start_was_date_only:
all_day = True
end_dt = None
if end_str:
if "T" not in end_str and " " not in end_str:
end_str = f"{end_str}T00:00:00"
try:
end_dt = datetime.fromisoformat(end_str)
if end_dt.tzinfo is None:
end_dt = end_dt.replace(tzinfo=timezone.utc)
end_dt, _ = await _parse_datetime_in_user_tz(user_id, end_str)
except (ValueError, TypeError):
return {"success": False, "error": f"Invalid end datetime: {end_str!r}"}
project_id = None
@@ -86,25 +106,30 @@ async def create_event_tool(*, user_id, arguments, **_ctx):
@tool(
name="list_events",
description="List calendar events in a date range. Use this when the user asks what events or meetings they have. Always use full-day UTC ranges: date_from at T00:00:00Z and date_to at T23:59:59Z for the days of interest so events stored in UTC are not missed.",
description="List calendar events in a date range. Use this when the user asks what events or meetings they have. Pass plain local dates (YYYY-MM-DD) — the server interprets them in the user's timezone and expands to a full local day.",
parameters={
"date_from": {"type": "string", "description": "Start of range in ISO 8601 UTC format (e.g. 2025-01-15T00:00:00Z). Use T00:00:00Z for the start of the day."},
"date_to": {"type": "string", "description": "End of range in ISO 8601 UTC format (e.g. 2025-01-22T23:59:59Z). Use T23:59:59Z for the end of the day."},
"date_from": {"type": "string", "description": "Start of range as a local date (YYYY-MM-DD) or local datetime. Interpreted in the user's timezone."},
"date_to": {"type": "string", "description": "End of range as a local date (YYYY-MM-DD) or local datetime. A bare date is expanded to 23:59:59 local."},
},
required=["date_from", "date_to"],
read_only=True,
briefing=True,
)
async def list_events_tool(*, user_id, arguments, **_ctx):
# Bare local dates are expanded to a full local day in the user's TZ:
# date_from → 00:00 local, date_to → 23:59:59 local, both converted to
# UTC before the DB query. Previously the tool description told the
# model to pass UTC ranges, which missed events for non-UTC users.
try:
date_from = datetime.fromisoformat(arguments["date_from"].replace("Z", "+00:00"))
date_to = datetime.fromisoformat(arguments["date_to"].replace("Z", "+00:00"))
date_from, _ = await _parse_datetime_in_user_tz(
user_id, arguments["date_from"]
)
date_to_str = arguments["date_to"]
if "T" not in date_to_str and " " not in date_to_str:
date_to_str = f"{date_to_str}T23:59:59"
date_to, _ = await _parse_datetime_in_user_tz(user_id, date_to_str)
except (ValueError, TypeError, KeyError) as exc:
return {"success": False, "error": f"Invalid date range: {exc}"}
if date_from.tzinfo is None:
date_from = date_from.replace(tzinfo=timezone.utc)
if date_to.tzinfo is None:
date_to = date_to.replace(tzinfo=timezone.utc)
events = await events_list_events(user_id=user_id, date_from=date_from, date_to=date_to)
return {
"success": True,
@@ -177,9 +202,9 @@ async def update_event_tool(*, user_id, arguments, **_ctx):
val = arguments.get(key)
if val:
try:
dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
# Naive datetimes are user-local, not UTC — see
# ``_parse_datetime_in_user_tz`` docstring.
dt, _ = await _parse_datetime_in_user_tz(user_id, val)
fields[dt_field] = dt
except (ValueError, TypeError):
return {"success": False, "error": f"Invalid datetime for {key}: {val!r}"}
+47
View File
@@ -0,0 +1,47 @@
"""User-timezone helpers.
All datetimes in the DB are stored as UTC. The helpers here bridge between
that UTC storage and the user's configured local timezone (IANA string in
the ``user_timezone`` setting). Use these anywhere the model or UI talks
in terms of "today", "tomorrow", or a bare calendar date — never
``date.today()`` or ``datetime.now(timezone.utc)`` inside a per-user flow.
"""
from __future__ import annotations
from datetime import date, datetime, timedelta
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from fabledassistant.services.settings import get_setting
# Briefing day boundary — kept in sync with the compilation slot in
# ``briefing_scheduler.SLOTS``. The briefing day flips at this local hour
# (not midnight) so the 00:0004:00 local window still shows yesterday's
# briefing until the 4am compilation generates the new one.
BRIEFING_DAY_START_HOUR = 4
async def get_user_tz(user_id: int) -> ZoneInfo:
"""Return the user's IANA ``ZoneInfo``, falling back to UTC."""
tz_str = await get_setting(user_id, "user_timezone") or "UTC"
try:
return ZoneInfo(tz_str)
except (ZoneInfoNotFoundError, KeyError):
return ZoneInfo("UTC")
async def user_today(user_id: int) -> date:
"""Return today's calendar date in the user's local timezone."""
tz = await get_user_tz(user_id)
return datetime.now(tz).date()
async def user_briefing_date(user_id: int) -> date:
"""Return the current "briefing day" in the user's local timezone.
The briefing day flips at ``BRIEFING_DAY_START_HOUR`` (4am local),
aligned with the compilation slot that generates the day's briefing.
Between 00:00 and 04:00 local this still returns *yesterday*, so the
UI keeps showing the in-progress briefing until the new one is built.
"""
tz = await get_user_tz(user_id)
return (datetime.now(tz) - timedelta(hours=BRIEFING_DAY_START_HOUR)).date()
+80
View File
@@ -134,3 +134,83 @@ def test_history_builder_no_tool_calls_unchanged():
assert len(history) == 2
assert history[0] == {"role": "user", "content": "Hello"}
assert history[1] == {"role": "assistant", "content": "Hi there!"}
# ---------------------------------------------------------------------------
# prepare_article_context tests
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_prepare_article_context_small_passthrough():
"""Articles under CHAR_BUDGET pass through unchanged with zero LLM calls."""
from fabledassistant.services import article_context
body = "A short article.\n\nWith two paragraphs."
with patch(
"fabledassistant.services.article_context.generate_completion",
new_callable=AsyncMock,
) as mock_gen:
out = await article_context.prepare_article_context(
"Title", "https://example.com", body, "test-model",
)
assert out == body
mock_gen.assert_not_called()
@pytest.mark.asyncio
async def test_prepare_article_context_large_runs_map_reduce():
"""Articles over CHAR_BUDGET are chunked and map-reduced via the background model."""
from fabledassistant.services import article_context
# CHAR_BUDGET is 48_000 — build a body well over that with paragraph breaks
# so the chunker has natural splits to work with.
paragraph = ("Lorem ipsum dolor sit amet, consectetur adipiscing elit. " * 40).strip()
body = "\n\n".join([paragraph] * 30) # ~70k+ chars across 30 paragraphs
assert len(body) > article_context.CHAR_BUDGET
with patch(
"fabledassistant.services.article_context.generate_completion",
new_callable=AsyncMock,
return_value="Summary of this section with specific claims preserved.",
) as mock_gen:
out = await article_context.prepare_article_context(
"Long Article", "https://example.com/long", body, "test-model",
)
# At least one LLM call fired (the map step ran)
assert mock_gen.await_count >= 1
# Output carries the oversized-article header and section markers
assert "longer than the chat window" in out
assert "## Section 1" in out
# Map output is much smaller than the raw body
assert len(out) < len(body)
def test_chunk_by_paragraph_respects_boundaries():
"""Chunker splits on paragraph breaks, not mid-sentence."""
from fabledassistant.services.article_context import _chunk_by_paragraph, CHUNK_CHARS
paragraphs = [f"Paragraph {i}. " + ("x" * 1000) for i in range(20)]
body = "\n\n".join(paragraphs)
chunks = _chunk_by_paragraph(body)
# Each chunk stays under the budget
for chunk in chunks:
assert len(chunk) <= CHUNK_CHARS
# Total content is preserved (modulo overlap duplication, so ≥ original)
assert sum(len(c) for c in chunks) >= len(body) * 0.9
def test_chunk_by_paragraph_handles_oversized_paragraph():
"""A single paragraph larger than CHUNK_CHARS gets split mid-paragraph."""
from fabledassistant.services.article_context import _chunk_by_paragraph, CHUNK_CHARS
body = "x" * (CHUNK_CHARS * 3) # one huge paragraph, no breaks
chunks = _chunk_by_paragraph(body)
assert len(chunks) >= 3
for chunk in chunks:
assert len(chunk) <= CHUNK_CHARS
+134
View File
@@ -0,0 +1,134 @@
"""Regression tests: calendar tool respects the user's configured timezone."""
from datetime import timezone
from unittest.mock import AsyncMock, patch
import pytest
@pytest.mark.asyncio
async def test_create_event_all_day_bare_date_interprets_local_midnight():
"""Bare date like '2026-09-30' for a NY user = 2026-09-30 00:00 NY,
stored as 2026-09-30 04:00 UTC — NOT 2026-09-30 00:00 UTC (which would
be 2026-09-29 19:00 NY and land on the wrong day)."""
from fabledassistant.services.tools.calendar import create_event_tool
captured = {}
async def fake_create_event(**kwargs):
captured.update(kwargs)
event = AsyncMock()
event.to_dict.return_value = {"id": 1, **{k: v for k, v in kwargs.items() if not callable(v)}}
return event
with patch(
"fabledassistant.services.tools.calendar.get_user_tz",
AsyncMock(return_value=__import__("zoneinfo").ZoneInfo("America/New_York")),
), patch(
"fabledassistant.services.tools.calendar.events_create_event",
side_effect=fake_create_event,
):
result = await create_event_tool(
user_id=1,
arguments={"title": "Birthday", "start": "2026-09-30"},
)
assert result["success"] is True
start_dt = captured["start_dt"]
assert start_dt.tzinfo is not None
# Converted to UTC: 00:00 NY (EDT, UTC-4) == 04:00 UTC on the same day
utc = start_dt.astimezone(timezone.utc)
assert (utc.year, utc.month, utc.day) == (2026, 9, 30)
assert utc.hour == 4 # EDT offset; would be 5 in EST
assert captured["all_day"] is True
@pytest.mark.asyncio
async def test_create_event_naive_datetime_is_user_local():
"""'2026-03-15T14:30' for a NY user means 2:30pm NY, not 2:30pm UTC."""
from fabledassistant.services.tools.calendar import create_event_tool
captured = {}
async def fake_create_event(**kwargs):
captured.update(kwargs)
event = AsyncMock()
event.to_dict.return_value = {"id": 1}
return event
with patch(
"fabledassistant.services.tools.calendar.get_user_tz",
AsyncMock(return_value=__import__("zoneinfo").ZoneInfo("America/New_York")),
), patch(
"fabledassistant.services.tools.calendar.events_create_event",
side_effect=fake_create_event,
):
await create_event_tool(
user_id=1,
arguments={"title": "Meeting", "start": "2026-03-15T14:30"},
)
utc = captured["start_dt"].astimezone(timezone.utc)
# 14:30 NY on 2026-03-15 (after DST start) = 18:30 UTC
assert (utc.year, utc.month, utc.day, utc.hour, utc.minute) == (2026, 3, 15, 18, 30)
@pytest.mark.asyncio
async def test_create_event_explicit_offset_is_respected():
"""Model-supplied timezone offsets must not be overridden by user TZ."""
from fabledassistant.services.tools.calendar import create_event_tool
captured = {}
async def fake_create_event(**kwargs):
captured.update(kwargs)
event = AsyncMock()
event.to_dict.return_value = {"id": 1}
return event
with patch(
"fabledassistant.services.tools.calendar.get_user_tz",
AsyncMock(return_value=__import__("zoneinfo").ZoneInfo("America/New_York")),
), patch(
"fabledassistant.services.tools.calendar.events_create_event",
side_effect=fake_create_event,
):
await create_event_tool(
user_id=1,
arguments={"title": "Meeting", "start": "2026-03-15T14:30+00:00"},
)
utc = captured["start_dt"].astimezone(timezone.utc)
assert (utc.hour, utc.minute) == (14, 30)
@pytest.mark.asyncio
async def test_list_events_bare_date_range_covers_local_day():
"""date_from/date_to as bare dates should cover the full LOCAL day."""
from fabledassistant.services.tools.calendar import list_events_tool
captured = {}
async def fake_list_events(*, user_id, date_from, date_to):
captured["date_from"] = date_from
captured["date_to"] = date_to
return []
with patch(
"fabledassistant.services.tools.calendar.get_user_tz",
AsyncMock(return_value=__import__("zoneinfo").ZoneInfo("America/New_York")),
), patch(
"fabledassistant.services.tools.calendar.events_list_events",
side_effect=fake_list_events,
):
await list_events_tool(
user_id=1,
arguments={"date_from": "2026-09-30", "date_to": "2026-09-30"},
)
df = captured["date_from"].astimezone(timezone.utc)
dt = captured["date_to"].astimezone(timezone.utc)
# 2026-09-30 00:00 NY (EDT) = 04:00 UTC same day
assert (df.year, df.month, df.day, df.hour) == (2026, 9, 30, 4)
# 2026-09-30 23:59:59 NY (EDT) = 03:59:59 UTC next day
assert (dt.year, dt.month, dt.day, dt.hour) == (2026, 10, 1, 3)
+63
View File
@@ -0,0 +1,63 @@
"""Tests for user-timezone helpers (services/tz.py)."""
from datetime import datetime
from unittest.mock import AsyncMock, patch
from zoneinfo import ZoneInfo
import pytest
@pytest.mark.asyncio
async def test_get_user_tz_returns_configured_zone():
from fabledassistant.services import tz as tz_mod
with patch.object(tz_mod, "get_setting", AsyncMock(return_value="America/New_York")):
zone = await tz_mod.get_user_tz(1)
assert zone == ZoneInfo("America/New_York")
@pytest.mark.asyncio
async def test_get_user_tz_falls_back_to_utc_on_bad_input():
from fabledassistant.services import tz as tz_mod
with patch.object(tz_mod, "get_setting", AsyncMock(return_value="Not/AZone")):
zone = await tz_mod.get_user_tz(1)
assert zone == ZoneInfo("UTC")
@pytest.mark.asyncio
async def test_user_briefing_date_before_4am_returns_yesterday():
"""00:0003:59 local still shows yesterday's briefing day."""
from fabledassistant.services import tz as tz_mod
ny = ZoneInfo("America/New_York")
# 02:00 NY on 2026-04-13 → briefing day = 2026-04-12
fake_now = datetime(2026, 4, 13, 2, 0, tzinfo=ny)
class _FakeDatetime(datetime):
@classmethod
def now(cls, tz=None):
return fake_now
with patch.object(tz_mod, "get_user_tz", AsyncMock(return_value=ny)), \
patch.object(tz_mod, "datetime", _FakeDatetime):
day = await tz_mod.user_briefing_date(1)
assert day.isoformat() == "2026-04-12"
@pytest.mark.asyncio
async def test_user_briefing_date_after_4am_returns_today():
from fabledassistant.services import tz as tz_mod
ny = ZoneInfo("America/New_York")
fake_now = datetime(2026, 4, 13, 4, 30, tzinfo=ny)
class _FakeDatetime(datetime):
@classmethod
def now(cls, tz=None):
return fake_now
with patch.object(tz_mod, "get_user_tz", AsyncMock(return_value=ny)), \
patch.object(tz_mod, "datetime", _FakeDatetime):
day = await tz_mod.user_briefing_date(1)
assert day.isoformat() == "2026-04-13"