refactor: Phase 10 — Ollama service, image cache, config, frontend orphans
Final cleanup phase of the MCP-first pivot.
docker-compose:
- docker-compose.yml: drop ollama service + OLLAMA_URL/MODEL env vars +
IMAGE_CACHE / VAPID env comments
- docker-compose.prod.yml: drop ollama service + Ollama env + GPU
reservation
- docker-compose.quickstart.yml: drop ollama service + Ollama env +
GPU-reservation comment; quickstart instructions now point at the
MCP Access tab instead of model-pull
Config:
- Drop OLLAMA_URL, OLLAMA_MODEL, OLLAMA_BACKGROUND_MODEL,
OLLAMA_KEEP_ALIVE_*, OLLAMA_NUM_CTX, EMBEDDING_MODEL (fastembed
is hard-coded inside services/embeddings.py)
- Drop IMAGE_CACHE_DIR, IMAGE_MAX_BYTES (image cache subsystem
deleted)
- Drop VAPID_PRIVATE_KEY, VAPID_PUBLIC_KEY, VAPID_CLAIMS_SUB (push
deleted in phase 8)
- Drop VOICE_ENABLED, STT_BACKEND, STT_MODEL, TTS_BACKEND (voice
deleted in phase 8)
- Drop Config.validate() rules for those keys
Image cache deletion:
- services/images.py, routes/images.py, models/image_cache.py
- models/__init__.py: drop ImageCache import
- app.py: drop images_bp registration
- alembic/versions/0054_drop_image_cache.py: DROP TABLE image_cache
Frontend client.ts orphan exports stripped:
- getVoiceStatus, getVoiceList, getVoiceLibrary, installVoice,
uninstallVoice, transcribeAudio, synthesiseSpeech,
VoiceStatusResult / VoiceEntry / VoiceLibraryEntry types
- getJournalConfig, saveJournalConfig, getJournalToday/Day/Days,
triggerJournalPrep, runJournalCurator, listPendingActions,
approvePendingAction, rejectPendingAction, listJournalMoments,
updateJournalMoment, deleteJournalMoment, geocodeAddress
- JournalConfig / JournalLocation / JournalConversation /
JournalMessage / JournalDayPayload / JournalMoment /
CuratorRunResult / PendingCuratorAction types
- consolidateProfile, clearProfileObservations, listProfileObservations
- ProfileObservationEntry, learned_summary/observations_* fields on
UserProfile
- consolidateTask (cascading update to TaskEditorView)
- getFableMcpInfo, getNewsItems, GetNewsItemsParams, NewsItem import
TaskEditorView:
- Drop the auto-summary banner + Re-consolidate button
- Drop isBodyAutoMaintained gate (editor is always user-controlled now)
- Drop reconsolidate function + reconsolidating ref
SettingsView:
- profile ref no longer initialises learned_summary /
observations_count / observations_updated_at (those fields are
gone from UserProfile type)
Surviving frontend composables/components flagged for likely future
cleanup but not deleted in this commit (no compile errors, just
unreferenced after Phase 7-8):
- useAssist, useFloatingAssist, useTagSuggestions, useVad,
useListenMode, useOnnxPreloader (composables)
- WorkspaceNoteEditor, WorkspaceTaskPanel, WeatherCard, InlineAssistPanel
(components)
- api/client.ts still references /api/notes/assist/* and
/api/notes/suggest-tags via useAssist + useTagSuggestions — those
endpoints 404 now but no caller hits them; dead at runtime, harmless.
Compose stack collapses to two services: `app` + `db`. No Ollama, no
voice models, no fable-mcp wheel build. First-boot install reduces to:
docker compose up -d
→ visit web UI → register → Settings → MCP Access → copy snippet
→ claude mcp add … → done.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,6 @@ from fabledassistant.routes.api import api
|
||||
from fabledassistant.routes.auth import auth_bp
|
||||
from fabledassistant.routes.export import export_bp
|
||||
from fabledassistant.routes.notes import notes_bp
|
||||
from fabledassistant.routes.images import images_bp
|
||||
from fabledassistant.routes.milestones import milestones_bp
|
||||
from fabledassistant.routes.task_logs import task_logs_bp
|
||||
from fabledassistant.routes.projects import projects_bp
|
||||
@@ -69,7 +68,6 @@ def create_app() -> Quart:
|
||||
app.register_blueprint(api)
|
||||
app.register_blueprint(auth_bp)
|
||||
app.register_blueprint(export_bp)
|
||||
app.register_blueprint(images_bp)
|
||||
app.register_blueprint(milestones_bp)
|
||||
app.register_blueprint(notes_bp)
|
||||
app.register_blueprint(projects_bp)
|
||||
|
||||
@@ -22,30 +22,11 @@ class Config:
|
||||
"DATABASE_URL_FILE",
|
||||
"postgresql+asyncpg://fabled:fabled@localhost:5432/fabledassistant",
|
||||
)
|
||||
OLLAMA_URL: str = os.environ.get("OLLAMA_URL", "http://localhost:11434")
|
||||
OLLAMA_MODEL: str = os.environ.get("OLLAMA_MODEL", "qwen3:latest")
|
||||
# Lightweight model for background tasks (title generation, tag suggestions,
|
||||
# project summaries). Using a separate model keeps the
|
||||
# main model's KV cache intact between user messages, enabling prefix cache hits.
|
||||
OLLAMA_BACKGROUND_MODEL: str = os.environ.get("OLLAMA_BACKGROUND_MODEL", "gemma3:4b")
|
||||
# Ollama keep_alive — how long a model stays resident in VRAM after its last
|
||||
# request. Main model gets a longer window since it's used interactively;
|
||||
# the background model is called sporadically and doesn't need to camp VRAM.
|
||||
# Format matches Ollama's duration strings: "30m", "10m", "1h", "0s", "-1" (forever).
|
||||
OLLAMA_KEEP_ALIVE_MAIN: str = os.environ.get("OLLAMA_KEEP_ALIVE_MAIN", "30m")
|
||||
OLLAMA_KEEP_ALIVE_BACKGROUND: str = os.environ.get("OLLAMA_KEEP_ALIVE_BACKGROUND", "10m")
|
||||
# KV cache context window for generation. Keep this as small as practical —
|
||||
# a larger context forces more KV cache into CPU RAM, drastically slowing prefill.
|
||||
# 16384 covers ~30+ message conversations with our system prompt comfortably.
|
||||
OLLAMA_NUM_CTX: int = int(os.environ.get("OLLAMA_NUM_CTX", "16384"))
|
||||
SECRET_KEY: str = _read_secret("SECRET_KEY", "SECRET_KEY_FILE", "dev-secret-change-me")
|
||||
SECURE_COOKIES: bool = os.environ.get("SECURE_COOKIES", "").lower() in ("1", "true", "yes")
|
||||
LOG_LEVEL: str = os.environ.get("LOG_LEVEL", "INFO")
|
||||
LOG_RETENTION_DAYS: int = int(os.environ.get("LOG_RETENTION_DAYS", "90"))
|
||||
|
||||
# Embedding model for semantic note search (served by Ollama)
|
||||
EMBEDDING_MODEL: str = os.environ.get("EMBEDDING_MODEL", "nomic-embed-text")
|
||||
|
||||
# SMTP defaults (overridden by DB settings when configured via admin UI)
|
||||
SMTP_HOST: str = os.environ.get("SMTP_HOST", "")
|
||||
SMTP_PORT: int = int(os.environ.get("SMTP_PORT", "587"))
|
||||
@@ -64,25 +45,11 @@ class Config:
|
||||
OIDC_SCOPES: str = os.environ.get("OIDC_SCOPES", "openid profile email")
|
||||
LOCAL_AUTH_ENABLED: bool = os.environ.get("LOCAL_AUTH_ENABLED", "true").lower() not in ("0", "false", "no")
|
||||
|
||||
# SearXNG web search (external instance)
|
||||
# SearXNG web search (external instance). Currently only surfaced via
|
||||
# /api/settings/search for the Integrations tab's status indicator —
|
||||
# the MCP layer doesn't proxy web search (Claude has its own).
|
||||
SEARXNG_URL: str = os.environ.get("SEARXNG_URL", "")
|
||||
|
||||
# Image cache — images fetched from the web are stored here and served locally
|
||||
IMAGE_CACHE_DIR: str = os.environ.get("IMAGE_CACHE_DIR", "/data/images")
|
||||
# Maximum size of a single image to cache (default 5 MB)
|
||||
IMAGE_MAX_BYTES: int = int(os.environ.get("IMAGE_MAX_BYTES", str(5 * 1024 * 1024)))
|
||||
|
||||
# VAPID keys for browser push notifications
|
||||
VAPID_PRIVATE_KEY: str = os.environ.get("VAPID_PRIVATE_KEY", "")
|
||||
VAPID_PUBLIC_KEY: str = os.environ.get("VAPID_PUBLIC_KEY", "")
|
||||
VAPID_CLAIMS_SUB: str = os.environ.get("VAPID_CLAIMS_SUB", "mailto:admin@fabledassistant.local")
|
||||
|
||||
# Voice (Speech-to-Speech) feature
|
||||
VOICE_ENABLED: bool = os.environ.get("VOICE_ENABLED", "").lower() in ("1", "true", "yes")
|
||||
STT_BACKEND: str = os.environ.get("STT_BACKEND", "faster-whisper")
|
||||
STT_MODEL: str = os.environ.get("STT_MODEL", "base.en")
|
||||
TTS_BACKEND: str = os.environ.get("TTS_BACKEND", "kokoro")
|
||||
|
||||
@classmethod
|
||||
def oidc_enabled(cls) -> bool:
|
||||
return bool(cls.OIDC_ISSUER and cls.OIDC_CLIENT_ID and cls.OIDC_CLIENT_SECRET)
|
||||
@@ -95,12 +62,8 @@ class Config:
|
||||
def validate(cls) -> None:
|
||||
"""Validate critical config values at startup. Raises ValueError on misconfiguration."""
|
||||
errors: list[str] = []
|
||||
if cls.OLLAMA_NUM_CTX < 512:
|
||||
errors.append(f"OLLAMA_NUM_CTX={cls.OLLAMA_NUM_CTX} is too small (minimum 512)")
|
||||
if cls.LOG_RETENTION_DAYS < 1:
|
||||
errors.append(f"LOG_RETENTION_DAYS={cls.LOG_RETENTION_DAYS} must be >= 1")
|
||||
if cls.IMAGE_MAX_BYTES < 1024:
|
||||
errors.append(f"IMAGE_MAX_BYTES={cls.IMAGE_MAX_BYTES} must be >= 1024")
|
||||
if not (1 <= cls.SMTP_PORT <= 65535):
|
||||
errors.append(f"SMTP_PORT={cls.SMTP_PORT} must be between 1 and 65535")
|
||||
if cls.oidc_enabled() and not cls.BASE_URL.startswith(("http://", "https://")):
|
||||
@@ -110,11 +73,5 @@ class Config:
|
||||
"SECRET_KEY is set to the insecure default but SECURE_COOKIES=true indicates "
|
||||
"a production deployment. Set SECRET_KEY or SECRET_KEY_FILE before starting."
|
||||
)
|
||||
_valid_stt_models = {"tiny.en", "base.en", "small.en", "medium.en"}
|
||||
if cls.VOICE_ENABLED and cls.STT_MODEL not in _valid_stt_models:
|
||||
errors.append(
|
||||
f"STT_MODEL='{cls.STT_MODEL}' is not supported. "
|
||||
f"Valid values: {', '.join(sorted(_valid_stt_models))}"
|
||||
)
|
||||
if errors:
|
||||
raise ValueError("Configuration errors:\n" + "\n".join(f" - {e}" for e in errors))
|
||||
|
||||
@@ -26,7 +26,6 @@ from fabledassistant.models.app_log import AppLog # noqa: E402, F401
|
||||
from fabledassistant.models.password_reset import PasswordResetToken # noqa: E402, F401
|
||||
from fabledassistant.models.invitation import InvitationToken # noqa: E402, F401
|
||||
from fabledassistant.models.embedding import NoteEmbedding # noqa: E402, F401
|
||||
from fabledassistant.models.image_cache import ImageCache # noqa: E402, F401
|
||||
from fabledassistant.models.project import Project # noqa: E402, F401
|
||||
from fabledassistant.models.event import Event # noqa: E402, F401
|
||||
from fabledassistant.models.milestone import Milestone # noqa: E402, F401
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import DateTime, Integer, Text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from fabledassistant.models import Base
|
||||
|
||||
|
||||
class ImageCache(Base):
|
||||
"""Metadata for an image fetched from the web and stored locally on disk."""
|
||||
|
||||
__tablename__ = "image_cache"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
url_hash: Mapped[str] = mapped_column(Text, unique=True, nullable=False, index=True)
|
||||
original_url: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
source_domain: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
content_type: Mapped[str] = mapped_column(Text, nullable=False, default="image/jpeg")
|
||||
file_size: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
file_ext: Mapped[str] = mapped_column(Text, nullable=False, default="jpg")
|
||||
fetched_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
)
|
||||
@@ -1,32 +0,0 @@
|
||||
"""Serve locally-cached images."""
|
||||
|
||||
import logging
|
||||
|
||||
from quart import Blueprint, jsonify, send_file
|
||||
|
||||
from fabledassistant.auth import login_required
|
||||
from fabledassistant.services.images import get_image_path, get_image_record
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
images_bp = Blueprint("images", __name__, url_prefix="/api/images")
|
||||
|
||||
|
||||
@images_bp.route("/<int:image_id>", methods=["GET"])
|
||||
@login_required
|
||||
async def serve_image(image_id: int):
|
||||
"""Serve a locally-cached image by its DB ID."""
|
||||
record = await get_image_record(image_id)
|
||||
if record is None:
|
||||
return jsonify({"error": "Image not found"}), 404
|
||||
|
||||
file_path = get_image_path(record)
|
||||
if not file_path.exists():
|
||||
logger.warning("Image file missing on disk for id=%d (%s)", image_id, file_path)
|
||||
return jsonify({"error": "Image file not found"}), 404
|
||||
|
||||
return await send_file(
|
||||
file_path,
|
||||
mimetype=record.content_type,
|
||||
cache_timeout=86400,
|
||||
)
|
||||
@@ -1,198 +0,0 @@
|
||||
"""Image cache service: fetch from web, store on disk, serve locally.
|
||||
|
||||
Images are stored in IMAGE_CACHE_DIR (default /data/images) and tracked in
|
||||
the image_cache DB table. The original URL is preserved for citation.
|
||||
Deduplication is by SHA-256 of the original URL — the same image requested
|
||||
twice shares one file and one DB row.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import logging
|
||||
import socket
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from fabledassistant.config import Config
|
||||
from fabledassistant.models import async_session
|
||||
from fabledassistant.models.image_cache import ImageCache
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _is_safe_image_url(url: str) -> bool:
|
||||
"""Return True only if the URL is a public http/https address (SSRF guard)."""
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
return False
|
||||
host = parsed.hostname
|
||||
if not host:
|
||||
return False
|
||||
if host.lower() in ("localhost", "::1"):
|
||||
return False
|
||||
addr_info = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
|
||||
for entry in addr_info:
|
||||
ip = ipaddress.ip_address(entry[4][0])
|
||||
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_reserved or ip.is_multicast:
|
||||
return False
|
||||
return True
|
||||
except Exception:
|
||||
return False # Block on resolution failure
|
||||
|
||||
_ALLOWED_TYPES = {
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/gif",
|
||||
"image/webp",
|
||||
"image/avif",
|
||||
}
|
||||
|
||||
_EXT_MAP = {
|
||||
"image/jpeg": "jpg",
|
||||
"image/png": "png",
|
||||
"image/gif": "gif",
|
||||
"image/webp": "webp",
|
||||
"image/avif": "avif",
|
||||
}
|
||||
|
||||
|
||||
def _cache_dir() -> Path:
|
||||
d = Path(Config.IMAGE_CACHE_DIR)
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def _url_hash(url: str) -> str:
|
||||
return hashlib.sha256(url.encode()).hexdigest()
|
||||
|
||||
|
||||
async def fetch_and_store_image(
|
||||
url: str,
|
||||
title: str | None = None,
|
||||
source_domain: str | None = None,
|
||||
) -> ImageCache | None:
|
||||
"""Fetch an image URL, write it to disk, persist metadata to DB.
|
||||
|
||||
Returns the ImageCache record (new or existing).
|
||||
Returns None if the URL is unreachable, not a valid image type, or
|
||||
exceeds IMAGE_MAX_BYTES.
|
||||
"""
|
||||
if not _is_safe_image_url(url):
|
||||
logger.warning("Blocked image fetch of private/internal URL: %s", url[:80])
|
||||
return None
|
||||
|
||||
url_hash = _url_hash(url)
|
||||
|
||||
# Return existing record if already cached
|
||||
async with async_session() as session:
|
||||
existing = (
|
||||
await session.execute(
|
||||
select(ImageCache).where(ImageCache.url_hash == url_hash)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing:
|
||||
logger.debug("Image cache hit id=%d for %s", existing.id, url[:80])
|
||||
return existing
|
||||
|
||||
# Infer source domain from URL if not supplied
|
||||
if not source_domain:
|
||||
try:
|
||||
source_domain = urlparse(url).netloc or None
|
||||
except Exception:
|
||||
source_domain = None
|
||||
|
||||
# Fetch from origin
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (compatible; FabledAssistant/1.0)",
|
||||
}
|
||||
# Provide a same-origin Referer to bypass basic hotlink protection
|
||||
if source_domain:
|
||||
headers["Referer"] = f"https://{source_domain}/"
|
||||
resp = await client.get(url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
|
||||
ct = resp.headers.get("content-type", "").split(";")[0].strip().lower()
|
||||
if ct not in _ALLOWED_TYPES:
|
||||
logger.warning("Skipping non-image content-type '%s' for %s", ct, url[:80])
|
||||
return None
|
||||
|
||||
# Reject oversized images before buffering all bytes
|
||||
cl = resp.headers.get("content-length")
|
||||
if cl and int(cl) > Config.IMAGE_MAX_BYTES:
|
||||
logger.warning("Image too large (%s bytes, limit %d) at %s", cl, Config.IMAGE_MAX_BYTES, url[:80])
|
||||
return None
|
||||
|
||||
data = resp.content
|
||||
if len(data) > Config.IMAGE_MAX_BYTES:
|
||||
logger.warning("Image too large (%d bytes) at %s", len(data), url[:80])
|
||||
return None
|
||||
if not data:
|
||||
return None
|
||||
|
||||
except Exception:
|
||||
logger.warning("Failed to fetch image from %s", url[:80], exc_info=True)
|
||||
return None
|
||||
|
||||
# Write to disk
|
||||
file_ext = _EXT_MAP.get(ct, "jpg")
|
||||
file_path = _cache_dir() / f"{url_hash}.{file_ext}"
|
||||
try:
|
||||
file_path.write_bytes(data)
|
||||
except Exception:
|
||||
logger.warning("Failed to write image to disk: %s", file_path, exc_info=True)
|
||||
return None
|
||||
|
||||
# Persist metadata — handle the rare race-condition duplicate gracefully
|
||||
record = ImageCache(
|
||||
url_hash=url_hash,
|
||||
original_url=url,
|
||||
source_domain=source_domain,
|
||||
title=title,
|
||||
content_type=ct,
|
||||
file_size=len(data),
|
||||
file_ext=file_ext,
|
||||
)
|
||||
async with async_session() as session:
|
||||
try:
|
||||
session.add(record)
|
||||
await session.commit()
|
||||
await session.refresh(record)
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
existing = (
|
||||
await session.execute(
|
||||
select(ImageCache).where(ImageCache.url_hash == url_hash)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
if existing:
|
||||
return existing
|
||||
logger.warning("Failed to persist image cache record for %s", url[:80], exc_info=True)
|
||||
return None
|
||||
|
||||
logger.info(
|
||||
"Cached image id=%d (%s, %d bytes) from %s",
|
||||
record.id, ct, len(data), url[:80],
|
||||
)
|
||||
return record
|
||||
|
||||
|
||||
async def get_image_record(image_id: int) -> ImageCache | None:
|
||||
"""Return the ImageCache row for a given ID, or None."""
|
||||
async with async_session() as session:
|
||||
return (
|
||||
await session.execute(
|
||||
select(ImageCache).where(ImageCache.id == image_id)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
def get_image_path(record: ImageCache) -> Path:
|
||||
"""Filesystem path for a cached image record."""
|
||||
return _cache_dir() / f"{record.url_hash}.{record.file_ext}"
|
||||
Reference in New Issue
Block a user