Files
FabledScribe/src/fabledassistant/services/backup.py
T
bvandeusen 91bafb641f refactor: Phase 8 — backend deletion (chat / voice / push / journal / curator)
Mega-commit. Strips all server-side LLM machinery now that Phase 7 has
removed the corresponding UI surfaces and the MCP HTTP endpoint is the
sole assistant interface.

Deleted (services/):
  chat, generation_buffer, generation_log, generation_task, llm, tools/
  (entire package), stt, tts, voice_config, voice_library, push,
  journal_closeout, journal_pipeline, journal_prep, journal_scheduler,
  journal_search, curator, curator_scheduler, consolidation,
  tag_suggestions, research, weather, article_fetcher, pending_actions,
  moments, assist, wikipedia.

Deleted (routes/):
  chat, voice, push, journal, quick_capture, fable_mcp_dist.

Deleted (models/):
  conversation, generation_tool_log, push_subscription,
  pending_curator_action, moment, weather_cache.

Deleted (tests/):
  test_generation_log, test_journal_*, test_consolidation, test_lookup_tool,
  test_notes_consolidation_trigger, test_record_moment_guards,
  test_research_pipeline, test_tools_*, test_tool_use_fixes,
  test_voice_library, test_weather_service, test_calendar_tool_tz,
  test_wikipedia.

Deleted (top-level):
  fable-mcp/ (legacy standalone stdio package — wheel-build pipeline
  also removed from Dockerfile).

app.py:
  - blueprint registrations for the 6 deleted routes
  - startup hook trimmed: no more Ollama warmup, KV-cache priming,
    journal/curator schedulers, voice model loading
  - shutdown hook simplified
  - httpx import dropped (was for Ollama calls)

pyproject.toml:
  - removed deps: pywebpush, feedparser, html2text, trafilatura
  - removed [voice] extras entirely
  - description updated for the MCP-first architecture

Dockerfile:
  - removed faster-whisper / piper-tts install steps
  - removed bundled piper voice download stage
  - removed fable-mcp wheel build stage

Surviving-file edits:
  - services/auth.py: drop Conversation table claim on first-user setup
  - services/backup.py: drop conversation / push-subscription export+restore;
    v1/v2 restore now silently skip pre-pivot conversation data
  - services/notes.py: drop maybe_consolidate trigger on task done/cancelled;
    drop _maybe_trigger_project_summary (LLM auto-summary)
  - services/projects.py: drop generate_project_summary + backfill_project_summaries
    (both LLM-driven)
  - services/user_profile.py: drop append_observations / consolidate /
    clear_learned_data (curator-tied) and build_profile_context
    (was LLM system-prompt builder)
  - services/notifications.py: stub out _fire_push_notif (was send_push_notification)
  - services/event_scheduler.py: drop event-reminder push + chat-retention
    cleanup job; keep CalDAV pull-sync + reminders job (in-app)
  - services/diagnostics.py: _curator_busy() always False
  - routes/notes.py: drop /assist, /assist/stream, /suggest-tags endpoints
  - routes/tasks.py: drop /<id>/consolidate endpoint
  - routes/settings.py: drop /models, KV-cache-prime-on-save, journal-schedule
    timezone hook, and the SearXNG search-test endpoint; inline _is_private_url
    (was in services/llm.py)
  - routes/admin.py: drop /voice, /voice/reload endpoints
  - routes/profile.py: drop /consolidate, /observations (GET, DELETE)
  - models/__init__.py: drop the 6 dead model imports

Frontend cascade:
  - stores/push.ts: deleted entirely (no callers after Phase 7)
  - stores/settings.ts: drop checkVoiceStatus + voice-status state
  - views/SettingsView.vue: drop Locations section + journalConfig state
    (was tied to /api/journal/config); drop JournalConfig + journal/voice
    api/client imports
  - frontend/api/client.ts: orphaned voice/journal/profile-observation/
    fable-mcp-dist exports are left as dead but harmless (call them and
    they 404; type-check is clean).

Pre-existing v1 backups that contained conversations/messages still
restore — those tables are silently dropped from the import path.
Anyone pulling the new image with a populated database will need the
Phase 9 migration to drop the dead tables (coming next).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 17:47:18 -04:00

546 lines
20 KiB
Python

import logging
from datetime import date, datetime, timezone
from sqlalchemy import select
from fabledassistant.models import async_session
from fabledassistant.models.milestone import Milestone
from fabledassistant.models.note import Note
from fabledassistant.models.note_draft import NoteDraft
from fabledassistant.models.note_version import NoteVersion
from fabledassistant.models.project import Project
from fabledassistant.models.setting import Setting
from fabledassistant.models.task_log import TaskLog
from fabledassistant.models.user import User
logger = logging.getLogger(__name__)
def _dt(val: str | None) -> datetime:
return datetime.fromisoformat(val) if val else datetime.now(timezone.utc)
def _d(val: str | None) -> date | None:
return date.fromisoformat(val) if val else None
# ---------------------------------------------------------------------------
# Export
# ---------------------------------------------------------------------------
async def export_full_backup() -> dict:
"""Export all data as version-2 JSON backup."""
async with async_session() as session:
users = (await session.execute(select(User))).scalars().all()
projects = (await session.execute(select(Project))).scalars().all()
milestones = (await session.execute(select(Milestone))).scalars().all()
notes = (await session.execute(select(Note))).scalars().all()
task_logs = (await session.execute(select(TaskLog))).scalars().all()
note_drafts = (await session.execute(select(NoteDraft))).scalars().all()
note_versions = (await session.execute(
select(NoteVersion).order_by(NoteVersion.note_id, NoteVersion.id)
)).scalars().all()
settings = (await session.execute(select(Setting))).scalars().all()
return {
"version": 2,
"scope": "full",
"exported_at": datetime.now(timezone.utc).isoformat(),
"_security_notice": (
"This backup contains hashed passwords. "
"Store it securely and restrict access."
),
"users": [
{
"id": u.id,
"username": u.username,
"email": u.email,
"password_hash": u.password_hash,
"oauth_sub": u.oauth_sub,
"role": u.role,
"session_version": u.session_version,
"created_at": u.created_at.isoformat(),
}
for u in users
],
"projects": [
{
"id": p.id,
"user_id": p.user_id,
"title": p.title,
"description": p.description,
"goal": p.goal,
"status": p.status,
"color": p.color,
"created_at": p.created_at.isoformat(),
"updated_at": p.updated_at.isoformat(),
}
for p in projects
],
"milestones": [
{
"id": m.id,
"user_id": m.user_id,
"project_id": m.project_id,
"title": m.title,
"description": m.description,
"status": m.status,
"order_index": m.order_index,
"created_at": m.created_at.isoformat(),
"updated_at": m.updated_at.isoformat(),
}
for m in milestones
],
"notes": [
{
"id": n.id,
"user_id": n.user_id,
"title": n.title,
"body": n.body,
"tags": n.tags or [],
"parent_id": n.parent_id,
"project_id": n.project_id,
"milestone_id": n.milestone_id,
"status": n.status,
"priority": n.priority,
"due_date": n.due_date.isoformat() if n.due_date else None,
"created_at": n.created_at.isoformat(),
"updated_at": n.updated_at.isoformat(),
}
for n in notes
],
"task_logs": [
{
"id": tl.id,
"user_id": tl.user_id,
"task_id": tl.task_id,
"content": tl.content,
"duration_minutes": tl.duration_minutes,
"created_at": tl.created_at.isoformat(),
"updated_at": tl.updated_at.isoformat(),
}
for tl in task_logs
],
"note_drafts": [
{
"id": nd.id,
"user_id": nd.user_id,
"note_id": nd.note_id,
"proposed_body": nd.proposed_body,
"original_body": nd.original_body,
"instruction": nd.instruction,
"scope": nd.scope,
"created_at": nd.created_at.isoformat(),
"updated_at": nd.updated_at.isoformat(),
}
for nd in note_drafts
],
"note_versions": [
{
"id": nv.id,
"user_id": nv.user_id,
"note_id": nv.note_id,
"title": nv.title,
"body": nv.body,
"tags": nv.tags or [],
"pin_kind": nv.pin_kind,
"pin_label": nv.pin_label,
"created_at": nv.created_at.isoformat(),
}
for nv in note_versions
],
"settings": [
{"user_id": s.user_id, "key": s.key, "value": s.value}
for s in settings
],
}
async def export_user_backup(user_id: int) -> dict:
"""Export a single user's data as version-2 JSON backup."""
async with async_session() as session:
user = await session.get(User, user_id)
projects = (await session.execute(
select(Project).where(Project.user_id == user_id)
)).scalars().all()
milestones = (await session.execute(
select(Milestone).where(Milestone.user_id == user_id)
)).scalars().all()
notes = (await session.execute(
select(Note).where(Note.user_id == user_id)
)).scalars().all()
task_logs = (await session.execute(
select(TaskLog).where(TaskLog.user_id == user_id)
)).scalars().all()
note_drafts = (await session.execute(
select(NoteDraft).where(NoteDraft.user_id == user_id)
)).scalars().all()
note_versions = (await session.execute(
select(NoteVersion).where(NoteVersion.user_id == user_id)
.order_by(NoteVersion.note_id, NoteVersion.id)
)).scalars().all()
settings = (await session.execute(
select(Setting).where(Setting.user_id == user_id)
)).scalars().all()
return {
"version": 2,
"scope": "user",
"exported_at": datetime.now(timezone.utc).isoformat(),
"user": {
"id": user.id,
"username": user.username,
"email": user.email,
"role": user.role,
"created_at": user.created_at.isoformat(),
} if user else None,
"projects": [
{
"id": p.id,
"user_id": p.user_id,
"title": p.title,
"description": p.description,
"goal": p.goal,
"status": p.status,
"color": p.color,
"created_at": p.created_at.isoformat(),
"updated_at": p.updated_at.isoformat(),
}
for p in projects
],
"milestones": [
{
"id": m.id,
"user_id": m.user_id,
"project_id": m.project_id,
"title": m.title,
"description": m.description,
"status": m.status,
"order_index": m.order_index,
"created_at": m.created_at.isoformat(),
"updated_at": m.updated_at.isoformat(),
}
for m in milestones
],
"notes": [
{
"id": n.id,
"title": n.title,
"body": n.body,
"tags": n.tags or [],
"parent_id": n.parent_id,
"project_id": n.project_id,
"milestone_id": n.milestone_id,
"status": n.status,
"priority": n.priority,
"due_date": n.due_date.isoformat() if n.due_date else None,
"created_at": n.created_at.isoformat(),
"updated_at": n.updated_at.isoformat(),
}
for n in notes
],
"task_logs": [
{
"id": tl.id,
"task_id": tl.task_id,
"content": tl.content,
"duration_minutes": tl.duration_minutes,
"created_at": tl.created_at.isoformat(),
"updated_at": tl.updated_at.isoformat(),
}
for tl in task_logs
],
"note_drafts": [
{
"id": nd.id,
"note_id": nd.note_id,
"proposed_body": nd.proposed_body,
"original_body": nd.original_body,
"instruction": nd.instruction,
"scope": nd.scope,
"created_at": nd.created_at.isoformat(),
"updated_at": nd.updated_at.isoformat(),
}
for nd in note_drafts
],
"note_versions": [
{
"id": nv.id,
"note_id": nv.note_id,
"title": nv.title,
"body": nv.body,
"tags": nv.tags or [],
"pin_kind": nv.pin_kind,
"pin_label": nv.pin_label,
"created_at": nv.created_at.isoformat(),
}
for nv in note_versions
],
"settings": [
{"key": s.key, "value": s.value}
for s in settings
],
}
# ---------------------------------------------------------------------------
# Restore
# ---------------------------------------------------------------------------
async def restore_full_backup(data: dict) -> dict:
"""Restore from backup JSON. Dispatches by version."""
version = data.get("version", 1)
if version == 1:
return await _restore_v1(data)
return await _restore_v2(data)
async def _restore_v1(data: dict) -> dict:
"""Restore legacy v1 backup (original format).
Pre-pivot v1 backups included conversations + messages; those are
skipped during restore now that the chat subsystem is gone.
"""
stats = {"users": 0, "notes": 0, "settings": 0}
async with async_session() as session:
user_id_map: dict[int, int] = {}
for u_data in data.get("users", []):
old_id = u_data["id"]
user = User(
username=u_data["username"],
email=u_data.get("email"),
password_hash=u_data["password_hash"],
role=u_data.get("role", "user"),
created_at=_dt(u_data.get("created_at")),
)
session.add(user)
await session.flush()
user_id_map[old_id] = user.id
stats["users"] += 1
note_id_map: dict[int, int] = {}
for n_data in data.get("notes", []):
old_id = n_data.get("id")
mapped_user_id = user_id_map.get(n_data.get("user_id", 0))
if mapped_user_id is None:
continue
note = Note(
user_id=mapped_user_id,
title=n_data.get("title", ""),
body=n_data.get("body", ""),
tags=n_data.get("tags", []),
parent_id=None, # patched below
status=n_data.get("status"),
priority=n_data.get("priority"),
due_date=_d(n_data.get("due_date")),
created_at=_dt(n_data.get("created_at")),
updated_at=_dt(n_data.get("updated_at")),
)
session.add(note)
await session.flush()
if old_id is not None:
note_id_map[old_id] = note.id
stats["notes"] += 1
# Patch parent_id now that all notes have new IDs
for n_data in data.get("notes", []):
old_id = n_data.get("id")
old_parent = n_data.get("parent_id")
if old_id and old_parent and old_id in note_id_map and old_parent in note_id_map:
note_row = await session.get(Note, note_id_map[old_id])
if note_row:
note_row.parent_id = note_id_map[old_parent]
for s_data in data.get("settings", []):
mapped_user_id = user_id_map.get(s_data.get("user_id", 0))
if mapped_user_id is None:
continue
session.add(Setting(user_id=mapped_user_id, key=s_data["key"], value=s_data.get("value", "")))
stats["settings"] += 1
await session.commit()
logger.info("Restored v1 backup: %s", stats)
return stats
async def _restore_v2(data: dict) -> dict:
"""Restore v2 backup with full FK re-mapping.
Conversations + push subscriptions in pre-pivot backups are silently
skipped — those subsystems were removed in the MCP-first pivot.
"""
stats: dict[str, int] = {
"users": 0, "projects": 0, "milestones": 0, "notes": 0,
"task_logs": 0, "note_drafts": 0, "note_versions": 0,
"settings": 0,
}
async with async_session() as session:
user_id_map: dict[int, int] = {}
project_id_map: dict[int, int] = {}
milestone_id_map: dict[int, int] = {}
note_id_map: dict[int, int] = {}
# 1. Users
for u_data in data.get("users", []):
old_id = u_data["id"]
user = User(
username=u_data["username"],
email=u_data.get("email"),
password_hash=u_data.get("password_hash"),
oauth_sub=u_data.get("oauth_sub"),
role=u_data.get("role", "user"),
session_version=u_data.get("session_version", 1),
created_at=_dt(u_data.get("created_at")),
)
session.add(user)
await session.flush()
user_id_map[old_id] = user.id
stats["users"] += 1
# 2. Projects
for p_data in data.get("projects", []):
mapped_uid = user_id_map.get(p_data.get("user_id", 0))
if mapped_uid is None:
continue
proj = Project(
user_id=mapped_uid,
title=p_data.get("title", ""),
description=p_data.get("description", ""),
goal=p_data.get("goal", ""),
status=p_data.get("status", "active"),
color=p_data.get("color"),
created_at=_dt(p_data.get("created_at")),
updated_at=_dt(p_data.get("updated_at")),
)
session.add(proj)
await session.flush()
project_id_map[p_data["id"]] = proj.id
stats["projects"] += 1
# 3. Milestones
for m_data in data.get("milestones", []):
mapped_uid = user_id_map.get(m_data.get("user_id", 0))
mapped_pid = project_id_map.get(m_data.get("project_id", 0))
if mapped_uid is None or mapped_pid is None:
continue
ms = Milestone(
user_id=mapped_uid,
project_id=mapped_pid,
title=m_data.get("title", ""),
description=m_data.get("description"),
status=m_data.get("status", "active"),
order_index=m_data.get("order_index", 0),
created_at=_dt(m_data.get("created_at")),
updated_at=_dt(m_data.get("updated_at")),
)
session.add(ms)
await session.flush()
milestone_id_map[m_data["id"]] = ms.id
stats["milestones"] += 1
# 4a. Notes — first pass (no parent_id yet)
notes_with_parents: list[tuple[int, int]] = [] # (new_note_id, old_parent_id)
for n_data in data.get("notes", []):
mapped_uid = user_id_map.get(n_data.get("user_id", 0))
if mapped_uid is None:
continue
note = Note(
user_id=mapped_uid,
title=n_data.get("title", ""),
body=n_data.get("body", ""),
tags=n_data.get("tags", []),
parent_id=None,
project_id=project_id_map.get(n_data["project_id"]) if n_data.get("project_id") else None,
milestone_id=milestone_id_map.get(n_data["milestone_id"]) if n_data.get("milestone_id") else None,
status=n_data.get("status"),
priority=n_data.get("priority"),
due_date=_d(n_data.get("due_date")),
created_at=_dt(n_data.get("created_at")),
updated_at=_dt(n_data.get("updated_at")),
)
session.add(note)
await session.flush()
note_id_map[n_data["id"]] = note.id
if n_data.get("parent_id"):
notes_with_parents.append((note.id, n_data["parent_id"]))
stats["notes"] += 1
# 4b. Patch parent_id
for new_note_id, old_parent_id in notes_with_parents:
new_parent_id = note_id_map.get(old_parent_id)
if new_parent_id:
note_row = await session.get(Note, new_note_id)
if note_row:
note_row.parent_id = new_parent_id
# 5. TaskLogs
for tl_data in data.get("task_logs", []):
mapped_uid = user_id_map.get(tl_data.get("user_id", 0))
mapped_tid = note_id_map.get(tl_data.get("task_id", 0))
if mapped_uid is None or mapped_tid is None:
continue
tl = TaskLog(
user_id=mapped_uid,
task_id=mapped_tid,
content=tl_data.get("content", ""),
duration_minutes=tl_data.get("duration_minutes"),
created_at=_dt(tl_data.get("created_at")),
updated_at=_dt(tl_data.get("updated_at")),
)
session.add(tl)
stats["task_logs"] += 1
# 6. NoteDrafts
for nd_data in data.get("note_drafts", []):
mapped_uid = user_id_map.get(nd_data.get("user_id", 0))
mapped_nid = note_id_map.get(nd_data.get("note_id", 0))
if mapped_uid is None or mapped_nid is None:
continue
nd = NoteDraft(
user_id=mapped_uid,
note_id=mapped_nid,
proposed_body=nd_data.get("proposed_body", ""),
original_body=nd_data.get("original_body", ""),
instruction=nd_data.get("instruction", ""),
scope=nd_data.get("scope", "document"),
created_at=_dt(nd_data.get("created_at")),
updated_at=_dt(nd_data.get("updated_at")),
)
session.add(nd)
stats["note_drafts"] += 1
# 7. NoteVersions
for nv_data in data.get("note_versions", []):
mapped_uid = user_id_map.get(nv_data.get("user_id", 0))
mapped_nid = note_id_map.get(nv_data.get("note_id", 0))
if mapped_uid is None or mapped_nid is None:
continue
nv = NoteVersion(
user_id=mapped_uid,
note_id=mapped_nid,
title=nv_data.get("title", ""),
body=nv_data.get("body", ""),
tags=nv_data.get("tags", []),
pin_kind=nv_data.get("pin_kind"),
pin_label=nv_data.get("pin_label"),
created_at=_dt(nv_data.get("created_at")),
)
session.add(nv)
stats["note_versions"] += 1
# 8. Settings
for s_data in data.get("settings", []):
mapped_uid = user_id_map.get(s_data.get("user_id", 0))
if mapped_uid is None:
continue
session.add(Setting(user_id=mapped_uid, key=s_data["key"], value=s_data.get("value", "")))
stats["settings"] += 1
await session.commit()
logger.info("Restored v2 backup: %s", stats)
return stats