# Gallery Subscriber

A Docker-based web application for managing gallery-dl downloads with a modern web interface.

---

## Table of Contents

1. [Project Overview](#project-overview)
2. [Architecture](#architecture)
3. [Technology Stack](#technology-stack)
4. [Project Structure](#project-structure)
5. [Database Schema](#database-schema)
6. [API Specification](#api-specification)
7. [Firefox Extension](#firefox-extension)
8. [Configuration](#configuration)
9. [Development Phases](#development-phases)
10. [Migration from Legacy Project](#migration-from-legacy-project)

---

## Project Overview

### Goals

- **Containerized Deployment**: Run the entire download manager in Docker for portability
- **Web Interface**: Modern UI for managing subscriptions, viewing downloads, and monitoring new content
- **Flexible Authentication**: Firefox extension for cookie export OR manual cookie upload via web UI
- **Notifications**: Real-time updates when new content is downloaded
- **Reliability**: Maintain the robust scheduling and error handling from the original project

### Non-Goals (Initial Release)

- Multi-user support (single-user initially)
- Mobile app
- SSL termination (handled by external reverse proxy)

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    External Reverse Proxy                        │
│              (Traefik/Caddy/nginx - handles SSL)                 │
└──────────────────────────┬──────────────────────────────────────┘
                           │ HTTP (port 8080)
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Docker Compose Stack                        │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    Quart Backend                         │    │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐  │    │
│  │  │ REST API │ │WebSocket │ │ Static   │ │ gallery-dl│  │    │
│  │  │ /api/*   │ │ /ws/*    │ │ Files    │ │ Wrapper   │  │    │
│  │  └──────────┘ └──────────┘ └──────────┘ └───────────┘  │    │
│  └─────────────────────────────────────────────────────────┘    │
│                              │                                   │
│  ┌───────────────────────────┴───────────────────────────────┐  │
│  │                     Celery Worker                          │  │
│  │  • Download tasks      • Scheduled checks                  │  │
│  │  • Manifest generation • Error retry logic                 │  │
│  └───────────────────────────────────────────────────────────┘  │
│                              │                                   │
│  ┌──────────────┐  ┌─────────┴───────┐  ┌──────────────────┐   │
│  │  PostgreSQL  │  │      Redis      │  │   Vue Frontend   │   │
│  │  (data)      │  │  (task queue)   │  │   (static build) │   │
│  └──────────────┘  └─────────────────┘  └──────────────────┘   │
│                                                                  │
│  Volumes:                                                        │
│  ├── /data/downloads  (downloaded content)                      │
│  ├── /data/config     (gallery-dl.conf)                         │
│  ├── /data/cookies    (cookie files)                            │
│  └── /data/db         (PostgreSQL data)                         │
└─────────────────────────────────────────────────────────────────┘
```

---

## Technology Stack

### Backend
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Web Framework | **Quart** | Flask-like API with native async, WebSocket support |
| Task Queue | **Celery + Redis** | Reliable background tasks, scheduling with Celery Beat |
| Database | **PostgreSQL** | JSON fields for metadata, robust concurrency |
| ORM | **SQLAlchemy 2.0** | Async support, mature ecosystem |
| Migrations | **Alembic** | Standard for SQLAlchemy |

### Frontend
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Framework | **Vue 3** | Reactive, component-based |
| UI Library | **Vuetify 3** | Material Design, comprehensive components |
| State | **Pinia** | Official Vue state management |
| HTTP Client | **Axios** | Request interceptors for auth |
| WebSocket | **Native** | Real-time updates |

### Infrastructure
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Container | **Docker Compose** | Easy deployment |
| Process Manager | **Supervisor** | Manage Quart + Celery in container |

---

## Project Structure

```
gallery-subscriber/
├── docker-compose.yml
├── docker-compose.dev.yml
├── .env.example
├── README.md
│
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── supervisord.conf
│   │
│   ├── app/
│   │   ├── __init__.py
│   │   ├── main.py              # Quart app factory
│   │   ├── config.py            # Settings (from env)
│   │   │
│   │   ├── api/
│   │   │   ├── __init__.py
│   │   │   ├── sources.py       # Source CRUD
│   │   │   ├── downloads.py     # Download history
│   │   │   ├── credentials.py   # Cookie management
│   │   │   ├── settings.py      # App settings
│   │   │   └── websocket.py     # Real-time events
│   │   │
│   │   ├── models/
│   │   │   ├── __init__.py
│   │   │   ├── source.py
│   │   │   ├── download.py
│   │   │   ├── credential.py
│   │   │   └── content.py
│   │   │
│   │   ├── services/
│   │   │   ├── __init__.py
│   │   │   ├── gallery_dl.py    # gallery-dl subprocess wrapper
│   │   │   ├── download_manager.py
│   │   │   ├── credential_manager.py
│   │   │   └── manifest.py      # Manifest generation
│   │   │
│   │   ├── tasks/
│   │   │   ├── __init__.py
│   │   │   ├── celery_app.py    # Celery configuration
│   │   │   ├── downloads.py     # Download tasks
│   │   │   └── scheduler.py     # Scheduled jobs
│   │   │
│   │   └── utils/
│   │       ├── __init__.py
│   │       ├── cookies.py       # Cookie parsing
│   │       └── encryption.py    # Credential encryption
│   │
│   ├── alembic/
│   │   ├── env.py
│   │   └── versions/
│   │
│   └── tests/
│
├── frontend/
│   ├── Dockerfile
│   ├── package.json
│   ├── vite.config.js
│   │
│   └── src/
│       ├── main.js
│       ├── App.vue
│       ├── router/
│       ├── stores/
│       ├── views/
│       │   ├── Dashboard.vue
│       │   ├── Sources.vue
│       │   ├── Downloads.vue
│       │   ├── NewContent.vue
│       │   └── Settings.vue
│       ├── components/
│       └── services/
│           └── api.js
│
├── extension/
│   ├── manifest.json
│   ├── popup/
│   │   ├── popup.html
│   │   ├── popup.css
│   │   └── popup.js
│   ├── background/
│   │   └── background.js
│   └── icons/
│
└── scripts/
    ├── migrate_legacy.py        # Import from old project
    └── init_db.py
```

---

## Database Schema

```sql
-- Sources (artists/creators to download from)
CREATE TABLE sources (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    platform VARCHAR(50) NOT NULL,  -- patreon, subscribestar, hentaifoundry, discord
    url TEXT NOT NULL,
    enabled BOOLEAN DEFAULT true,
    priority INTEGER DEFAULT 0,
    check_interval INTEGER DEFAULT 3600,  -- seconds between checks
    last_check TIMESTAMP,
    last_success TIMESTAMP,
    error_count INTEGER DEFAULT 0,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(platform, url)
);

-- Downloads (individual download jobs)
CREATE TABLE downloads (
    id SERIAL PRIMARY KEY,
    source_id INTEGER REFERENCES sources(id) ON DELETE SET NULL,
    url TEXT NOT NULL,
    status VARCHAR(20) DEFAULT 'pending',  -- pending, running, completed, failed, skipped
    error_type VARCHAR(50),  -- auth_error, rate_limited, not_found, etc.
    error_message TEXT,
    file_count INTEGER DEFAULT 0,
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_downloads_status ON downloads(status);
CREATE INDEX idx_downloads_source ON downloads(source_id);

-- Content Items (discovered files)
CREATE TABLE content_items (
    id SERIAL PRIMARY KEY,
    source_id INTEGER REFERENCES sources(id) ON DELETE CASCADE,
    download_id INTEGER REFERENCES downloads(id) ON DELETE SET NULL,
    external_id VARCHAR(255),       -- Platform-specific ID
    title VARCHAR(500),
    content_type VARCHAR(50),       -- image, video, attachment
    file_path VARCHAR(500),
    file_size BIGINT,
    thumbnail_path VARCHAR(500),
    published_at TIMESTAMP,
    discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    metadata JSONB DEFAULT '{}',
    UNIQUE(source_id, external_id)
);

CREATE INDEX idx_content_discovered ON content_items(discovered_at DESC);

-- Credentials (encrypted cookie/token storage)
CREATE TABLE credentials (
    id SERIAL PRIMARY KEY,
    platform VARCHAR(50) NOT NULL UNIQUE,
    credential_type VARCHAR(20) NOT NULL,  -- cookies, token
    data BYTEA NOT NULL,                   -- Encrypted
    expires_at TIMESTAMP,
    last_verified TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Settings (key-value store)
CREATE TABLE settings (
    key VARCHAR(100) PRIMARY KEY,
    value JSONB NOT NULL,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

---

## API Specification

### Sources

```
GET    /api/sources              List all sources (with pagination)
POST   /api/sources              Add new source
GET    /api/sources/:id          Get source details
PATCH  /api/sources/:id          Update source
DELETE /api/sources/:id          Delete source
POST   /api/sources/:id/check    Trigger immediate download check
```

### Downloads

```
GET    /api/downloads            List download history
GET    /api/downloads/:id        Get download details with logs
POST   /api/downloads/:id/retry  Retry failed download
GET    /api/downloads/stats      Get statistics (by platform, by day)
```

### Content

```
GET    /api/content              List content items (filterable)
GET    /api/content/new          Get new content since timestamp
```

### Credentials

```
GET    /api/credentials          List credential status (no secrets)
POST   /api/credentials          Upload cookies (manual or from extension)
DELETE /api/credentials/:platform  Remove credentials for platform
POST   /api/credentials/verify   Verify credentials are working
```

### Settings

```
GET    /api/settings             Get all settings
PATCH  /api/settings             Update settings
GET    /api/settings/gallery-dl  Get gallery-dl config
PUT    /api/settings/gallery-dl  Update gallery-dl config
```

### WebSocket

```
WS /ws/events

Events (server -> client):
  download.started    { download_id, source_id, url }
  download.progress   { download_id, file_count }
  download.completed  { download_id, file_count }
  download.failed     { download_id, error_type, error_message }
  content.new         { content_id, source_name, title, thumbnail }
  credential.expiring { platform, expires_in_hours }
```

---

## Firefox Extension

### Features

- Export cookies for supported platforms (Patreon, SubscribeStar, HentaiFoundry)
- Export Discord token
- Auto-refresh option (re-export when cookies change)
- Connection status indicator
- Manual export button per platform

### Supported Platforms

| Platform | Auth Type | Domains |
|----------|-----------|---------|
| Patreon | Cookies | `.patreon.com` |
| SubscribeStar | Cookies | `.subscribestar.com`, `.subscribestar.adult` |
| HentaiFoundry | Cookies | `.hentai-foundry.com` |
| Discord | Token | `.discord.com` |

### Manual Alternative

For users who don't want the extension:
1. Export cookies using browser dev tools or "cookies.txt" extension
2. Upload via Settings page in web UI
3. Paste cookie content or upload file

---

## Configuration

### Environment Variables (.env)

```bash
# Database
DATABASE_URL=postgresql://gdl:password@db:5432/gallery_subscriber

# Redis
REDIS_URL=redis://redis:6379/0

# Security
SECRET_KEY=your-secret-key-at-least-32-characters
EXTENSION_API_KEY=key-for-firefox-extension

# Paths (inside container)
DOWNLOAD_PATH=/data/downloads
CONFIG_PATH=/data/config
COOKIES_PATH=/data/cookies

# Download settings
DOWNLOAD_PARALLEL_LIMIT=3
DOWNLOAD_RATE_LIMIT=3.0
DEFAULT_CHECK_INTERVAL=3600

# Server
HOST=0.0.0.0
PORT=8080
```

### docker-compose.yml

```yaml
services:
  app:
    build: ./backend
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
      - REDIS_URL=redis://redis:6379/0
      - SECRET_KEY=${SECRET_KEY}
      - EXTENSION_API_KEY=${EXTENSION_API_KEY}
    volumes:
      - downloads:/data/downloads
      - config:/data/config
      - cookies:/data/cookies
    depends_on:
      - db
      - redis
    restart: unless-stopped

  celery:
    build: ./backend
    command: celery -A app.tasks.celery_app worker -B --loglevel=info
    environment:
      - DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
      - REDIS_URL=redis://redis:6379/0
      - SECRET_KEY=${SECRET_KEY}
    volumes:
      - downloads:/data/downloads
      - config:/data/config
      - cookies:/data/cookies
    depends_on:
      - db
      - redis
    restart: unless-stopped

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=gdl
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=gallery_subscriber
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    restart: unless-stopped

volumes:
  postgres_data:
  downloads:
  config:
  cookies:
```

---

## Development Phases

### Phase 1: Foundation
- [ ] Set up project structure
- [ ] Create Docker Compose development environment
- [ ] Initialize Quart app with basic routing
- [ ] Set up PostgreSQL + Alembic migrations
- [ ] Create database models
- [ ] Basic API endpoints (sources CRUD)

### Phase 2: Download Engine
- [ ] Port gallery-dl wrapper from legacy project
- [ ] Implement Celery tasks for downloads
- [ ] Add platform-specific locking (from legacy)
- [ ] Error categorization and handling
- [ ] Download history tracking

### Phase 3: Scheduling & State
- [ ] Celery Beat for scheduled downloads
- [ ] Round-robin scheduling logic (from legacy)
- [ ] Priority queue for failed tasks
- [ ] State persistence in PostgreSQL

### Phase 4: Frontend
- [ ] Vue 3 + Vuetify scaffold
- [ ] Dashboard view
- [ ] Sources management UI
- [ ] Download history view
- [ ] Settings page

### Phase 5: Authentication
- [ ] Credential storage (encrypted)
- [ ] Manual cookie upload via web UI
- [ ] Firefox extension
- [ ] Credential verification

### Phase 6: Real-time & Content
- [ ] WebSocket event streaming
- [ ] New content feed
- [ ] Content browsing with thumbnails
- [ ] Notifications

### Phase 7: Polish & Migration
- [ ] Migration script for legacy sources.yaml
- [ ] Import legacy gallery-dl.conf
- [ ] Testing
- [ ] Documentation

---

## Migration from Legacy Project

### What Gets Migrated

| Legacy File | Destination |
|-------------|-------------|
| `sources.yaml` | `sources` table |
| `gallery-dl.conf` | `/data/config/gallery-dl.conf` (paths updated) |
| `state.json` | `sources` table (last_check, error_count) |
| Download history | Optional - start fresh or import logs |

### Migration Script Usage

```bash
python scripts/migrate_legacy.py \
  --sources /path/to/sources.yaml \
  --config /path/to/gallery-dl.conf \
  --state /path/to/state.json
```

### Key Patterns Preserved from Legacy

1. **Platform locking** - One download per platform at a time
2. **Round-robin scheduling** - Fair distribution across sources
3. **Priority queue** - Failed tasks get priority on next run
4. **Error categorization** - AUTH_ERROR, RATE_LIMITED, NOT_FOUND, etc.
5. **Rate limiting** - Configurable delay between requests

---

*Document Version: 2.0*
*Created: 2025-01-24*
*Project: Gallery Subscriber*