# Backup Service Rewrite Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Rewrite `services/backup.py` so that full and user backup/restore correctly includes every model added since the original implementation (projects, milestones, task logs, drafts, versions), with correct FK re-mapping on restore.

**Architecture:** Bump backup JSON format to version 2. Export is additive — all new tables exported. Restore builds an ID map for each table and patches FK references in the correct dependency order. V1 backups continue to restore via the existing code path.

**Tech Stack:** Python/SQLAlchemy 2.0 async, no new dependencies.

**Spec:** `docs/superpowers/specs/2026-03-11-backup-rewrite-design.md`

**Dependency note:** This plan can be split into Part A (pre-sharing models) and Part B (sharing models). Part A is independent and should be done first.

---

## Task 1: Extend export — full backup

**Files:**
- Modify: `src/fabledassistant/services/backup.py`

- [ ] **Step 1: Read the current `export_full_backup()` function**

Understand what is currently exported: users, notes (partial fields), conversations+messages, settings.

- [ ] **Step 2: Rewrite `export_full_backup()` to version 2**

Add all missing models to the import list at the top of `backup.py`:
```python
from fabledassistant.models.project import Project
from fabledassistant.models.milestone import Milestone
from fabledassistant.models.task_log import TaskLog
from fabledassistant.models.note_draft import NoteDraft
from fabledassistant.models.note_version import NoteVersion
from fabledassistant.models.push_subscription import PushSubscription
```

Rewrite `export_full_backup()`:
```python
async def export_full_backup() -> dict:
    async with async_session() as session:
        users = (await session.execute(select(User))).scalars().all()
        projects = (await session.execute(select(Project))).scalars().all()
        milestones = (await session.execute(select(Milestone))).scalars().all()
        notes = (await session.execute(select(Note))).scalars().all()
        task_logs = (await session.execute(select(TaskLog))).scalars().all()
        note_drafts = (await session.execute(select(NoteDraft))).scalars().all()
        note_versions = (await session.execute(select(NoteVersion).order_by(NoteVersion.note_id, NoteVersion.version_number))).scalars().all()
        conversations = (await session.execute(
            select(Conversation).options(selectinload(Conversation.messages))
        )).scalars().all()
        settings = (await session.execute(select(Setting))).scalars().all()
        push_subs = (await session.execute(select(PushSubscription))).scalars().all()

        return {
            "version": 2,
            "scope": "full",
            "exported_at": datetime.now(timezone.utc).isoformat(),
            "_security_notice": (
                "This backup contains hashed passwords and push subscription keys. "
                "Store it securely and restrict access."
            ),
            "users": [
                {
                    "id": u.id,
                    "username": u.username,
                    "email": u.email,
                    "password_hash": u.password_hash,
                    "oauth_sub": u.oauth_sub,
                    "role": u.role,
                    "session_version": u.session_version,
                    "created_at": u.created_at.isoformat(),
                }
                for u in users
            ],
            "projects": [
                {
                    "id": p.id,
                    "user_id": p.user_id,
                    "title": p.title,
                    "description": p.description,
                    "goal": p.goal,
                    "status": p.status,
                    "color": p.color,
                    "created_at": p.created_at.isoformat(),
                    "updated_at": p.updated_at.isoformat(),
                }
                for p in projects
            ],
            "milestones": [
                {
                    "id": m.id,
                    "user_id": m.user_id,
                    "project_id": m.project_id,
                    "title": m.title,
                    "description": m.description,
                    "status": m.status,
                    "order_index": m.order_index,
                    "created_at": m.created_at.isoformat(),
                    "updated_at": m.updated_at.isoformat(),
                }
                for m in milestones
            ],
            "notes": [
                {
                    "id": n.id,
                    "user_id": n.user_id,
                    "title": n.title,
                    "body": n.body,
                    "tags": n.tags or [],
                    "parent_id": n.parent_id,
                    "project_id": n.project_id,
                    "milestone_id": n.milestone_id,
                    "is_task": n.is_task,
                    "status": n.status,
                    "priority": n.priority,
                    "due_date": n.due_date.isoformat() if n.due_date else None,
                    "is_starred": getattr(n, "is_starred", False),
                    "is_pinned": getattr(n, "is_pinned", False),
                    "created_at": n.created_at.isoformat(),
                    "updated_at": n.updated_at.isoformat(),
                }
                for n in notes
            ],
            "task_logs": [
                {
                    "id": tl.id,
                    "user_id": tl.user_id,
                    "note_id": tl.note_id,
                    "description": tl.description,
                    "duration_minutes": tl.duration_minutes,
                    "logged_at": tl.logged_at.isoformat() if tl.logged_at else None,
                    "created_at": tl.created_at.isoformat(),
                }
                for tl in task_logs
            ],
            "note_drafts": [
                {
                    "id": nd.id,
                    "user_id": nd.user_id,
                    "note_id": nd.note_id,
                    "title": nd.title,
                    "body": nd.body,
                    "saved_at": nd.saved_at.isoformat() if nd.saved_at else None,
                }
                for nd in note_drafts
            ],
            "note_versions": [
                {
                    "id": nv.id,
                    "user_id": nv.user_id,
                    "note_id": nv.note_id,
                    "title": nv.title,
                    "body": nv.body,
                    "version_number": nv.version_number,
                    "created_at": nv.created_at.isoformat(),
                }
                for nv in note_versions
            ],
            "conversations": [
                {
                    "id": c.id,
                    "user_id": c.user_id,
                    "title": c.title,
                    "created_at": c.created_at.isoformat(),
                    "updated_at": c.updated_at.isoformat(),
                    "messages": [
                        {
                            "id": m.id,
                            "role": m.role,
                            "content": m.content,
                            "context_note_id": m.context_note_id,
                            "created_at": m.created_at.isoformat(),
                        }
                        for m in c.messages
                    ],
                }
                for c in conversations
            ],
            "settings": [
                {"user_id": s.user_id, "key": s.key, "value": s.value}
                for s in settings
            ],
            "push_subscriptions": [
                {
                    "user_id": ps.user_id,
                    "endpoint": ps.endpoint,
                    "p256dh": ps.p256dh,
                    "auth": ps.auth,
                    "created_at": ps.created_at.isoformat(),
                }
                for ps in push_subs
            ],
        }
```

Check what exact field names exist on each model before assuming. Use `getattr(obj, "field", default)` for fields that may not exist on older DB versions.

- [ ] **Step 3: Commit**

```bash
git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 full export with projects, milestones, task_logs, drafts, versions"
```

---

## Task 2: Extend export — user backup

**Files:**
- Modify: `src/fabledassistant/services/backup.py`

- [ ] **Step 1: Rewrite `export_user_backup(user_id)`**

Same structure as full backup but filtered to `WHERE user_id = uid`. Apply same field additions. Omit sensitive fields (password_hash, oauth_sub) from user self-export.

```python
async def export_user_backup(user_id: int) -> dict:
    async with async_session() as session:
        user = await session.get(User, user_id)
        projects = (await session.execute(select(Project).where(Project.user_id == user_id))).scalars().all()
        milestones = (await session.execute(select(Milestone).where(Milestone.user_id == user_id))).scalars().all()
        notes = (await session.execute(select(Note).where(Note.user_id == user_id))).scalars().all()
        task_logs = (await session.execute(select(TaskLog).where(TaskLog.user_id == user_id))).scalars().all()
        note_drafts = (await session.execute(select(NoteDraft).where(NoteDraft.user_id == user_id))).scalars().all()
        note_versions = (await session.execute(
            select(NoteVersion).where(NoteVersion.user_id == user_id)
            .order_by(NoteVersion.note_id, NoteVersion.version_number)
        )).scalars().all()
        conversations = (await session.execute(
            select(Conversation).options(selectinload(Conversation.messages))
            .where(Conversation.user_id == user_id)
        )).scalars().all()
        settings = (await session.execute(select(Setting).where(Setting.user_id == user_id))).scalars().all()

        return {
            "version": 2,
            "scope": "user",
            "exported_at": datetime.now(timezone.utc).isoformat(),
            "user": {
                "id": user.id,
                "username": user.username,
                "email": user.email,
                "role": user.role,
                "created_at": user.created_at.isoformat(),
            } if user else None,
            "projects": [...],   # same as full but user-filtered
            "milestones": [...],
            "notes": [...],
            "task_logs": [...],
            "note_drafts": [...],
            "note_versions": [...],
            "conversations": [...],
            "settings": [...],
        }
```

- [ ] **Step 2: Commit**

```bash
git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 user backup with all models"
```

---

## Task 3: Rewrite restore for v2

**Files:**
- Modify: `src/fabledassistant/services/backup.py`

- [ ] **Step 1: Add version dispatch to `restore_full_backup`**

```python
async def restore_full_backup(data: dict) -> dict:
    version = data.get("version", 1)
    if version == 1:
        return await _restore_v1(data)
    return await _restore_v2(data)
```

Move existing restore code into `_restore_v1(data)` with no changes.

- [ ] **Step 2: Implement `_restore_v2(data)` with full FK re-mapping**

Restore order (respects FK dependencies):
1. Users → build `user_id_map`
2. Projects (fk: user_id) → build `project_id_map`
3. Milestones (fk: user_id, project_id) → build `milestone_id_map`
4. Notes — first pass: insert without `parent_id` (fk: user_id, project_id, milestone_id) → build `note_id_map`
5. Notes — second pass: patch `parent_id` using `note_id_map`
6. TaskLogs (fk: user_id, note_id via note_id_map)
7. NoteDrafts (fk: user_id, note_id via note_id_map)
8. NoteVersions (fk: user_id, note_id via note_id_map) — export only, no restore by default (skip unless flag set)
9. Conversations (fk: user_id) → Messages (fk: conversation_id, context_note_id via note_id_map)
10. Settings (fk: user_id)

```python
async def _restore_v2(data: dict) -> dict:
    from datetime import date, datetime, timezone

    stats = {
        "users": 0, "projects": 0, "milestones": 0, "notes": 0,
        "task_logs": 0, "note_drafts": 0, "conversations": 0,
        "messages": 0, "settings": 0
    }

    async with async_session() as session:
        user_id_map: dict[int, int] = {}
        project_id_map: dict[int, int] = {}
        milestone_id_map: dict[int, int] = {}
        note_id_map: dict[int, int] = {}

        # 1. Users
        for u_data in data.get("users", []):
            old_id = u_data["id"]
            user = User(
                username=u_data["username"],
                email=u_data.get("email"),
                password_hash=u_data.get("password_hash"),
                oauth_sub=u_data.get("oauth_sub"),
                role=u_data.get("role", "user"),
                session_version=u_data.get("session_version", 1),
                created_at=datetime.fromisoformat(u_data["created_at"]) if u_data.get("created_at") else datetime.now(timezone.utc),
            )
            session.add(user)
            await session.flush()
            user_id_map[old_id] = user.id
            stats["users"] += 1

        # 2. Projects
        for p_data in data.get("projects", []):
            mapped_uid = user_id_map.get(p_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            proj = Project(
                user_id=mapped_uid,
                title=p_data.get("title", ""),
                description=p_data.get("description"),
                goal=p_data.get("goal"),
                status=p_data.get("status", "active"),
                color=p_data.get("color"),
                created_at=datetime.fromisoformat(p_data["created_at"]) if p_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(p_data["updated_at"]) if p_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(proj)
            await session.flush()
            project_id_map[p_data["id"]] = proj.id
            stats["projects"] += 1

        # 3. Milestones
        for m_data in data.get("milestones", []):
            mapped_uid = user_id_map.get(m_data.get("user_id", 0))
            mapped_pid = project_id_map.get(m_data.get("project_id", 0))
            if mapped_uid is None:
                continue
            ms = Milestone(
                user_id=mapped_uid,
                project_id=mapped_pid,
                title=m_data.get("title", ""),
                description=m_data.get("description"),
                status=m_data.get("status", "open"),
                order_index=m_data.get("order_index", 0),
                created_at=datetime.fromisoformat(m_data["created_at"]) if m_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(m_data["updated_at"]) if m_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(ms)
            await session.flush()
            milestone_id_map[m_data["id"]] = ms.id
            stats["milestones"] += 1

        # 4. Notes — first pass (no parent_id)
        note_objects: list[tuple[int, int]] = []  # (old_parent_id, new_note_id)
        for n_data in data.get("notes", []):
            mapped_uid = user_id_map.get(n_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            due = None
            if n_data.get("due_date"):
                due = date.fromisoformat(n_data["due_date"])
            note = Note(
                user_id=mapped_uid,
                title=n_data.get("title", ""),
                body=n_data.get("body", ""),
                tags=n_data.get("tags", []),
                parent_id=None,  # patched in second pass
                project_id=project_id_map.get(n_data.get("project_id")) if n_data.get("project_id") else None,
                milestone_id=milestone_id_map.get(n_data.get("milestone_id")) if n_data.get("milestone_id") else None,
                is_task=n_data.get("is_task", False),
                status=n_data.get("status"),
                priority=n_data.get("priority"),
                due_date=due,
                created_at=datetime.fromisoformat(n_data["created_at"]) if n_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(n_data["updated_at"]) if n_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(note)
            await session.flush()
            note_id_map[n_data["id"]] = note.id
            if n_data.get("parent_id"):
                note_objects.append((n_data["parent_id"], note.id))
            stats["notes"] += 1

        # 5. Notes — second pass: patch parent_id
        for old_parent_id, new_note_id in note_objects:
            new_parent_id = note_id_map.get(old_parent_id)
            if new_parent_id:
                note_row = await session.get(Note, new_note_id)
                if note_row:
                    note_row.parent_id = new_parent_id

        # 6. TaskLogs
        for tl_data in data.get("task_logs", []):
            mapped_uid = user_id_map.get(tl_data.get("user_id", 0))
            mapped_nid = note_id_map.get(tl_data.get("note_id", 0))
            if mapped_uid is None or mapped_nid is None:
                continue
            from fabledassistant.models.task_log import TaskLog
            tl = TaskLog(
                user_id=mapped_uid,
                note_id=mapped_nid,
                description=tl_data.get("description", ""),
                duration_minutes=tl_data.get("duration_minutes"),
                logged_at=datetime.fromisoformat(tl_data["logged_at"]) if tl_data.get("logged_at") else None,
                created_at=datetime.fromisoformat(tl_data["created_at"]) if tl_data.get("created_at") else datetime.now(timezone.utc),
            )
            session.add(tl)
            stats["task_logs"] += 1

        # 7. NoteDrafts
        for nd_data in data.get("note_drafts", []):
            mapped_uid = user_id_map.get(nd_data.get("user_id", 0))
            mapped_nid = note_id_map.get(nd_data.get("note_id", 0))
            if mapped_uid is None or mapped_nid is None:
                continue
            from fabledassistant.models.note_draft import NoteDraft
            nd = NoteDraft(
                user_id=mapped_uid,
                note_id=mapped_nid,
                title=nd_data.get("title", ""),
                body=nd_data.get("body", ""),
                saved_at=datetime.fromisoformat(nd_data["saved_at"]) if nd_data.get("saved_at") else None,
            )
            session.add(nd)
            stats["note_drafts"] += 1

        # 8. Conversations + Messages
        for c_data in data.get("conversations", []):
            mapped_uid = user_id_map.get(c_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            conv = Conversation(
                user_id=mapped_uid,
                title=c_data.get("title", ""),
                created_at=datetime.fromisoformat(c_data["created_at"]) if c_data.get("created_at") else datetime.now(timezone.utc),
                updated_at=datetime.fromisoformat(c_data["updated_at"]) if c_data.get("updated_at") else datetime.now(timezone.utc),
            )
            session.add(conv)
            await session.flush()
            stats["conversations"] += 1
            for m_data in c_data.get("messages", []):
                msg = Message(
                    conversation_id=conv.id,
                    role=m_data["role"],
                    content=m_data.get("content", ""),
                    context_note_id=note_id_map.get(m_data["context_note_id"]) if m_data.get("context_note_id") else None,
                    created_at=datetime.fromisoformat(m_data["created_at"]) if m_data.get("created_at") else datetime.now(timezone.utc),
                )
                session.add(msg)
                stats["messages"] += 1

        # 9. Settings
        for s_data in data.get("settings", []):
            mapped_uid = user_id_map.get(s_data.get("user_id", 0))
            if mapped_uid is None:
                continue
            setting = Setting(user_id=mapped_uid, key=s_data["key"], value=s_data.get("value", ""))
            session.add(setting)
            stats["settings"] += 1

        await session.commit()

    logger.info("Restored v2 backup: %s", stats)
    return stats
```

- [ ] **Step 3: Add missing imports at top of file**

```python
from datetime import datetime, timezone
from fabledassistant.models.project import Project
from fabledassistant.models.milestone import Milestone
```

- [ ] **Step 4: Test restore round-trip**

```bash
# Export
curl -s -b cookies.txt "http://localhost:5000/api/admin/backup?scope=full" -o /tmp/backup_v2.json
# Inspect structure
python3 -c "import json; d=json.load(open('/tmp/backup_v2.json')); print(list(d.keys()))"
# Expected keys: version, scope, exported_at, users, projects, milestones, notes, task_logs, ...
```

- [ ] **Step 5: Typecheck**

```bash
docker compose exec app python -m py_compile src/fabledassistant/services/backup.py
```
Expected: No errors.

- [ ] **Step 6: Commit**

```bash
git add src/fabledassistant/services/backup.py
git commit -m "feat(backup): v2 restore with full FK re-mapping; v1 restore preserved"
```

---

## Task 4: Verification

- [ ] **Step 1: Full round-trip test**

1. Export full backup as admin
2. Verify JSON has `"version": 2` and all expected top-level keys
3. Verify notes have `project_id`, `milestone_id`, `is_task` fields
4. Verify `task_logs` array has entries (create a task log entry first if needed)
5. Restore backup to a clean test instance (or verify restore code path runs without error in dry-run)

- [ ] **Step 2: V1 backward compat test**

Take an old v1 backup JSON (or construct one without the `version` key) and run restore. Verify it takes the v1 path and doesn't error.

- [ ] **Step 3: Commit final verification note**

```bash
git commit --allow-empty -m "chore(backup): v2 backup rewrite verified"
```