FabledScribe/docs/plans/2026-03-11-backup-rewrite.md

Backup Service Rewrite Plan

For agentic workers: REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Rewrite services/backup.py so that full and user backup/restore correctly includes every model added since the original implementation (projects, milestones, task logs, drafts, versions), with correct FK re-mapping on restore.

Architecture: Bump backup JSON format to version 2. Export is additive — all new tables exported. Restore builds an ID map for each table and patches FK references in the correct dependency order. V1 backups continue to restore via the existing code path.

Tech Stack: Python/SQLAlchemy 2.0 async, no new dependencies.

Spec: docs/superpowers/specs/2026-03-11-backup-rewrite-design.md

Dependency note: This plan can be split into Part A (pre-sharing models) and Part B (sharing models). Part A is independent and should be done first.

---

Task 1: Extend export — full backup

Files:

• Modify: src/fabledassistant/services/backup.py

• Step 1: Read the current export_full_backup() function

Understand what is currently exported: users, notes (partial fields), conversations+messages, settings.

• Step 2: Rewrite export_full_backup() to version 2

Add all missing models to the import list at the top of backup.py:

from fabledassistant.models.project import Project
from fabledassistant.models.milestone import Milestone
from fabledassistant.models.task_log import TaskLog
from fabledassistant.models.note_draft import NoteDraft
from fabledassistant.models.note_version import NoteVersion
from fabledassistant.models.push_subscription import PushSubscription

Rewrite export_full_backup():

async def export_full_backup() -> dict:
    async with async_session() as session:
        users = (await session.execute(select(User))).scalars().all()
        projects = (await session.execute(select(Project))).scalars().all()
        milestones = (await session.execute(select(Milestone))).scalars().all()
        notes = (await session.execute(select(Note))).scalars().all()
        task_logs = (await session.execute(select(TaskLog))).scalars().all()
        note_drafts = (await session.execute(select(NoteDraft))).scalars().all()
        note_versions = (await session.execute(select(NoteVersion).order_by(NoteVersion.note_id, NoteVersion.version_number))).scalars().all()
        conversations = (await session.execute(
            select(Conversation).options(selectinload(Conversation.messages))
        )).scalars().all()
        settings = (await session.execute(select(Setting))).scalars().all()
        push_subs = (await session.execute(select(PushSubscription))).scalars().all()

        return {
            "version": 2,
            "scope": "full",
            "exported_at": datetime.now(timezone.utc).isoformat(),
            "_security_notice": (
                "This backup contains hashed passwords and push subscription keys. "
                "Store it securely and restrict access."
            ),
            "users": [
                {
                    "id": u.id,
                    "username": u.username,
                    "email": u.email,
                    "password_hash": u.password_hash,
                    "oauth_sub": u.oauth_sub,
                    "role": u.role,
                    "session_version": u.session_version,
                    "created_at": u.created_at.isoformat(),
                }
                for u in users
            ],
            "projects": [
                {
                    "id": p.id,
                    "user_id": p.user_id,
                    "title": p.title,
                    "description": p.description,
                    "goal": p.goal,
                    "status": p.status,
                    "color": p.color,
                    "created_at": p.created_at.isoformat(),
                    "updated_at": p.updated_at.isoformat(),
                }
                for p in projects
            ],
            "milestones": [
                {
                    "id": m.id,
                    "user_id": m.user_id,
                    "project_id": m.project_id,
                    "title": m.title,
                    "description": m.description,
                    "status": m.status,
                    "order_index": m.order_index,
                    "created_at": m.created_at.isoformat(),
                    "updated_at": m.updated_at.isoformat(),
                }
                for m in milestones
            ],
            "notes": [
                {
                    "id": n.id,
                    "user_id": n.user_id,
                    "title": n.title,
                    "body": n.body,
                    "tags": n.tags or [],
                    "parent_id": n.parent_id,
                    "project_id": n.project_id,
                    "milestone_id": n.milestone_id,
                    "is_task": n.is_task,
                    "status": n.status,
                    "priority": n.priority,
                    "due_date": n.due_date.isoformat() if n.due_date else None,
                    "is_starred": getattr(n, "is_starred", False),
                    "is_pinned": getattr(n, "is_pinned", False),
                    "created_at": n.created_at.isoformat(),
                    "updated_at": n.updated_at.isoformat(),
                }
                for n in notes
            ],
            "task_logs": [
                {
                    "id": tl.id,
                    "user_id": tl.user_id,
                    "note_id": tl.note_id,
                    "description": tl.description,
                    "duration_minutes": tl.duration_minutes,
                    "logged_at": tl.logged_at.isoformat() if tl.logged_at else None,
                    "created_at": tl.created_at.isoformat(),
                }
                for tl in task_logs
            ],
            "note_drafts": [
                {
                    "id": nd.id,
                    "user_id": nd.user_id,
                    "note_id": nd.note_id,
                    "title": nd.title,
                    "body": nd.body,
                    "saved_at": nd.saved_at.isoformat() if nd.saved_at else None,
                }
                for nd in note_drafts
            ],
            "note_versions": [
                {
                    "id": nv.id,
                    "user_id": nv.user_id,
                    "note_id": nv.note_id,
                    "title": nv.title,
                    "body": nv.body,
                    "version_number": nv.version_number,
                    "created_at": nv.created_at.isoformat(),
                }
                for nv in note_versions
            ],
            "conversations": [
                {
                    "id": c.id,
                    "user_id": c.user_id,
                    "title": c.title,
                    "created_at": c.created_at.isoformat(),
                    "updated_at": c.updated_at.isoformat(),
                    "messages": [
                        {
                            "id": m.id,
                            "role": m.role,
                            "content": m.content,
                            "context_note_id": m.context_note_id,
                            "created_at": m.created_at.isoformat(),
                        }
                        for m in c.messages
                    ],
                }
                for c in conversations
            ],
            "settings": [
                {"user_id": s.user_id, "key": s.key, "value": s.value}
                for s in settings
            ],
            "push_subscriptions": [
                {
                    "user_id": ps.user_id,
                    "endpoint": ps.endpoint,
                    "p256dh": ps.p256dh,
                    "auth": ps.auth,
                    "created_at": ps.created_at.isoformat(),
                }
                for ps in push_subs
            ],
        }

Check what exact field names exist on each model before assuming. Use getattr(obj, "field", default) for fields that may not exist on older DB versions.

• Step 3: Commit

git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 full export with projects, milestones, task_logs, drafts, versions"

---

Task 2: Extend export — user backup

Files:

• Modify: src/fabledassistant/services/backup.py

• Step 1: Rewrite export_user_backup(user_id)

Same structure as full backup but filtered to WHERE user_id = uid. Apply same field additions. Omit sensitive fields (password_hash, oauth_sub) from user self-export.

async def export_user_backup(user_id: int) -> dict:
    async with async_session() as session:
        user = await session.get(User, user_id)
        projects = (await session.execute(select(Project).where(Project.user_id == user_id))).scalars().all()
        milestones = (await session.execute(select(Milestone).where(Milestone.user_id == user_id))).scalars().all()
        notes = (await session.execute(select(Note).where(Note.user_id == user_id))).scalars().all()
        task_logs = (await session.execute(select(TaskLog).where(TaskLog.user_id == user_id))).scalars().all()
        note_drafts = (await session.execute(select(NoteDraft).where(NoteDraft.user_id == user_id))).scalars().all()
        note_versions = (await session.execute(
            select(NoteVersion).where(NoteVersion.user_id == user_id)
            .order_by(NoteVersion.note_id, NoteVersion.version_number)
        )).scalars().all()
        conversations = (await session.execute(
            select(Conversation).options(selectinload(Conversation.messages))
            .where(Conversation.user_id == user_id)
        )).scalars().all()
        settings = (await session.execute(select(Setting).where(Setting.user_id == user_id))).scalars().all()

        return {
            "version": 2,
            "scope": "user",
            "exported_at": datetime.now(timezone.utc).isoformat(),
            "user": {
                "id": user.id,
                "username": user.username,
                "email": user.email,
                "role": user.role,
                "created_at": user.created_at.isoformat(),
            } if user else None,
            "projects": [...],   # same as full but user-filtered
            "milestones": [...],
            "notes": [...],
            "task_logs": [...],
            "note_drafts": [...],
            "note_versions": [...],
            "conversations": [...],
            "settings": [...],
        }

• Step 2: Commit

git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 user backup with all models"

---

Task 3: Rewrite restore for v2

Files:

• Modify: src/fabledassistant/services/backup.py

• Step 1: Add version dispatch to restore_full_backup

async def restore_full_backup(data: dict) -> dict:
    version = data.get("version", 1)
    if version == 1:
        return await _restore_v1(data)
    return await _restore_v2(data)

Move existing restore code into _restore_v1(data) with no changes.

• Step 2: Implement _restore_v2(data) with full FK re-mapping

Restore order (respects FK dependencies):

1. Users → build user_id_map
2. Projects (fk: user_id) → build project_id_map
3. Milestones (fk: user_id, project_id) → build milestone_id_map
4. Notes — first pass: insert without parent_id (fk: user_id, project_id, milestone_id) → build note_id_map
5. Notes — second pass: patch parent_id using note_id_map
6. TaskLogs (fk: user_id, note_id via note_id_map)
7. NoteDrafts (fk: user_id, note_id via note_id_map)
8. NoteVersions (fk: user_id, note_id via note_id_map) — export only, no restore by default (skip unless flag set)
9. Conversations (fk: user_id) → Messages (fk: conversation_id, context_note_id via note_id_map)
10. Settings (fk: user_id)

async def _restore_v2(data: dict) -> dict:
    from datetime import date, datetime, timezone

    stats = {
        "users": 0, "projects": 0, "milestones": 0, "notes": 0,
        "task_logs": 0, "note_drafts": 0, "conversations": 0,
        "messages": 0, "settings": 0
    }

    async with async_session() as session:
        user_id_map: dict[int, int] = {}
        project_id_map: dict[int, int] = {}
        milestone_id_map: dict[int, int] = {}
        note_id_map: dict[int, int] = {}

        # 1. Users
        for u_data in data.get("users", []):
            old_id = u_data["id"]
            user = User(
                username=u_data["username"],
                email=u_data.get("email"),
                password_hash=u_data.get("password_hash"),
                oauth_sub=u_data.get("oauth_sub"),
                role=u_data.get("role", "user"),
                session_version=u_data.get("session_version", 1),
                created_at=datetime.fromisoformat(u_data["created_at"]) if u_data.get("created_at") else datetime.now(timezone.utc),
            )
            session.add(user)
            await session.flush()
            user_id_map[old_id] = user.id
            stats["users"] += 1

        # 2. Projects
        for p_data in data.get("projects", []):
            mapped_uid = user_id_map.get(p_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            proj = Project(
                user_id=mapped_uid,
                title=p_data.get("title", ""),
                description=p_data.get("description"),
                goal=p_data.get("goal"),
                status=p_data.get("status", "active"),
                color=p_data.get("color"),
                created_at=datetime.fromisoformat(p_data["created_at"]) if p_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(p_data["updated_at"]) if p_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(proj)
            await session.flush()
            project_id_map[p_data["id"]] = proj.id
            stats["projects"] += 1

        # 3. Milestones
        for m_data in data.get("milestones", []):
            mapped_uid = user_id_map.get(m_data.get("user_id", 0))
            mapped_pid = project_id_map.get(m_data.get("project_id", 0))
            if mapped_uid is None:
                continue
            ms = Milestone(
                user_id=mapped_uid,
                project_id=mapped_pid,
                title=m_data.get("title", ""),
                description=m_data.get("description"),
                status=m_data.get("status", "open"),
                order_index=m_data.get("order_index", 0),
                created_at=datetime.fromisoformat(m_data["created_at"]) if m_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(m_data["updated_at"]) if m_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(ms)
            await session.flush()
            milestone_id_map[m_data["id"]] = ms.id
            stats["milestones"] += 1

        # 4. Notes — first pass (no parent_id)
        note_objects: list[tuple[int, int]] = []  # (old_parent_id, new_note_id)
        for n_data in data.get("notes", []):
            mapped_uid = user_id_map.get(n_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            due = None
            if n_data.get("due_date"):
                due = date.fromisoformat(n_data["due_date"])
            note = Note(
                user_id=mapped_uid,
                title=n_data.get("title", ""),
                body=n_data.get("body", ""),
                tags=n_data.get("tags", []),
                parent_id=None,  # patched in second pass
                project_id=project_id_map.get(n_data.get("project_id")) if n_data.get("project_id") else None,
                milestone_id=milestone_id_map.get(n_data.get("milestone_id")) if n_data.get("milestone_id") else None,
                is_task=n_data.get("is_task", False),
                status=n_data.get("status"),
                priority=n_data.get("priority"),
                due_date=due,
                created_at=datetime.fromisoformat(n_data["created_at"]) if n_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(n_data["updated_at"]) if n_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(note)
            await session.flush()
            note_id_map[n_data["id"]] = note.id
            if n_data.get("parent_id"):
                note_objects.append((n_data["parent_id"], note.id))
            stats["notes"] += 1

        # 5. Notes — second pass: patch parent_id
        for old_parent_id, new_note_id in note_objects:
            new_parent_id = note_id_map.get(old_parent_id)
            if new_parent_id:
                note_row = await session.get(Note, new_note_id)
                if note_row:
                    note_row.parent_id = new_parent_id

        # 6. TaskLogs
        for tl_data in data.get("task_logs", []):
            mapped_uid = user_id_map.get(tl_data.get("user_id", 0))
            mapped_nid = note_id_map.get(tl_data.get("note_id", 0))
            if mapped_uid is None or mapped_nid is None:
                continue
            from fabledassistant.models.task_log import TaskLog
            tl = TaskLog(
                user_id=mapped_uid,
                note_id=mapped_nid,
                description=tl_data.get("description", ""),
                duration_minutes=tl_data.get("duration_minutes"),
                logged_at=datetime.fromisoformat(tl_data["logged_at"]) if tl_data.get("logged_at") else None,
                created_at=datetime.fromisoformat(tl_data["created_at"]) if tl_data.get("created_at") else datetime.now(timezone.utc),
            )
            session.add(tl)
            stats["task_logs"] += 1

        # 7. NoteDrafts
        for nd_data in data.get("note_drafts", []):
            mapped_uid = user_id_map.get(nd_data.get("user_id", 0))
            mapped_nid = note_id_map.get(nd_data.get("note_id", 0))
            if mapped_uid is None or mapped_nid is None:
                continue
            from fabledassistant.models.note_draft import NoteDraft
            nd = NoteDraft(
                user_id=mapped_uid,
                note_id=mapped_nid,
                title=nd_data.get("title", ""),
                body=nd_data.get("body", ""),
                saved_at=datetime.fromisoformat(nd_data["saved_at"]) if nd_data.get("saved_at") else None,
            )
            session.add(nd)
            stats["note_drafts"] += 1

        # 8. Conversations + Messages
        for c_data in data.get("conversations", []):
            mapped_uid = user_id_map.get(c_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            conv = Conversation(
                user_id=mapped_uid,
                title=c_data.get("title", ""),
                created_at=datetime.fromisoformat(c_data["created_at"]) if c_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(c_data["updated_at"]) if c_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(conv)
            await session.flush()
            stats["conversations"] += 1
            for m_data in c_data.get("messages", []):
                msg = Message(
                    conversation_id=conv.id,
                    role=m_data["role"],
                    content=m_data.get("content", ""),
                    context_note_id=note_id_map.get(m_data["context_note_id"]) if m_data.get("context_note_id") else None,
                    created_at=datetime.fromisoformat(m_data["created_at"]) if m_data.get("created_at") else datetime.now(timezone.utc),
                )
                session.add(msg)
                stats["messages"] += 1

        # 9. Settings
        for s_data in data.get("settings", []):
            mapped_uid = user_id_map.get(s_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            setting = Setting(user_id=mapped_uid, key=s_data["key"], value=s_data.get("value", ""))
            session.add(setting)
            stats["settings"] += 1

        await session.commit()

    logger.info("Restored v2 backup: %s", stats)
    return stats

• Step 3: Add missing imports at top of file

from datetime import datetime, timezone
from fabledassistant.models.project import Project
from fabledassistant.models.milestone import Milestone

• Step 4: Test restore round-trip

# Export
curl -s -b cookies.txt "http://localhost:5000/api/admin/backup?scope=full" -o /tmp/backup_v2.json
# Inspect structure
python3 -c "import json; d=json.load(open('/tmp/backup_v2.json')); print(list(d.keys()))"
# Expected keys: version, scope, exported_at, users, projects, milestones, notes, task_logs, ...

• Step 5: Typecheck

docker compose exec app python -m py_compile src/fabledassistant/services/backup.py

Expected: No errors.

• Step 6: Commit

git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 restore with full FK re-mapping; v1 restore preserved"

---

Task 4: Verification

• Step 1: Full round-trip test

1. Export full backup as admin
2. Verify JSON has "version": 2 and all expected top-level keys
3. Verify notes have project_id, milestone_id, is_task fields
4. Verify task_logs array has entries (create a task log entry first if needed)
5. Restore backup to a clean test instance (or verify restore code path runs without error in dry-run)

• Step 2: V1 backward compat test

Take an old v1 backup JSON (or construct one without the version key) and run restore. Verify it takes the v1 path and doesn't error.

• Step 3: Commit final verification note

git commit --allow-empty -m "chore(backup): v2 backup rewrite verified"