Bryan Van Deusen
549 lines
18 KiB
Markdown
549 lines
18 KiB
Markdown
# Gallery Subscriber
|
|
|
|
A Docker-based web application for managing gallery-dl downloads with a modern web interface.
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Project Overview](#project-overview)
|
|
2. [Architecture](#architecture)
|
|
3. [Technology Stack](#technology-stack)
|
|
4. [Project Structure](#project-structure)
|
|
5. [Database Schema](#database-schema)
|
|
6. [API Specification](#api-specification)
|
|
7. [Firefox Extension](#firefox-extension)
|
|
8. [Configuration](#configuration)
|
|
9. [Development Phases](#development-phases)
|
|
10. [Migration from Legacy Project](#migration-from-legacy-project)
|
|
|
|
---
|
|
|
|
## Project Overview
|
|
|
|
### Goals
|
|
|
|
- **Containerized Deployment**: Run the entire download manager in Docker for portability
|
|
- **Web Interface**: Modern UI for managing subscriptions, viewing downloads, and monitoring new content
|
|
- **Flexible Authentication**: Firefox extension for cookie export OR manual cookie upload via web UI
|
|
- **Notifications**: Real-time updates when new content is downloaded
|
|
- **Reliability**: Maintain the robust scheduling and error handling from the original project
|
|
|
|
### Non-Goals (Initial Release)
|
|
|
|
- Multi-user support (single-user initially)
|
|
- Mobile app
|
|
- SSL termination (handled by external reverse proxy)
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ External Reverse Proxy │
|
|
│ (Traefik/Caddy/nginx - handles SSL) │
|
|
└──────────────────────────┬──────────────────────────────────────┘
|
|
│ HTTP (port 8080)
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Docker Compose Stack │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ Quart Backend │ │
|
|
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ │ │
|
|
│ │ │ REST API │ │WebSocket │ │ Static │ │ gallery-dl│ │ │
|
|
│ │ │ /api/* │ │ /ws/* │ │ Files │ │ Wrapper │ │ │
|
|
│ │ └──────────┘ └──────────┘ └──────────┘ └───────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌───────────────────────────┴───────────────────────────────┐ │
|
|
│ │ Celery Worker │ │
|
|
│ │ • Download tasks • Scheduled checks │ │
|
|
│ │ • Manifest generation • Error retry logic │ │
|
|
│ └───────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────────┐ ┌─────────┴───────┐ ┌──────────────────┐ │
|
|
│ │ PostgreSQL │ │ Redis │ │ Vue Frontend │ │
|
|
│ │ (data) │ │ (task queue) │ │ (static build) │ │
|
|
│ └──────────────┘ └─────────────────┘ └──────────────────┘ │
|
|
│ │
|
|
│ Volumes: │
|
|
│ ├── /data/downloads (downloaded content) │
|
|
│ ├── /data/config (gallery-dl.conf) │
|
|
│ ├── /data/cookies (cookie files) │
|
|
│ └── /data/db (PostgreSQL data) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Technology Stack
|
|
|
|
### Backend
|
|
| Component | Technology | Rationale |
|
|
|-----------|------------|-----------|
|
|
| Web Framework | **Quart** | Flask-like API with native async, WebSocket support |
|
|
| Task Queue | **Celery + Redis** | Reliable background tasks, scheduling with Celery Beat |
|
|
| Database | **PostgreSQL** | JSON fields for metadata, robust concurrency |
|
|
| ORM | **SQLAlchemy 2.0** | Async support, mature ecosystem |
|
|
| Migrations | **Alembic** | Standard for SQLAlchemy |
|
|
|
|
### Frontend
|
|
| Component | Technology | Rationale |
|
|
|-----------|------------|-----------|
|
|
| Framework | **Vue 3** | Reactive, component-based |
|
|
| UI Library | **Vuetify 3** | Material Design, comprehensive components |
|
|
| State | **Pinia** | Official Vue state management |
|
|
| HTTP Client | **Axios** | Request interceptors for auth |
|
|
| WebSocket | **Native** | Real-time updates |
|
|
|
|
### Infrastructure
|
|
| Component | Technology | Rationale |
|
|
|-----------|------------|-----------|
|
|
| Container | **Docker Compose** | Easy deployment |
|
|
| Process Manager | **Supervisor** | Manage Quart + Celery in container |
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
gallery-subscriber/
|
|
├── docker-compose.yml
|
|
├── docker-compose.dev.yml
|
|
├── .env.example
|
|
├── README.md
|
|
│
|
|
├── backend/
|
|
│ ├── Dockerfile
|
|
│ ├── requirements.txt
|
|
│ ├── supervisord.conf
|
|
│ │
|
|
│ ├── app/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── main.py # Quart app factory
|
|
│ │ ├── config.py # Settings (from env)
|
|
│ │ │
|
|
│ │ ├── api/
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── sources.py # Source CRUD
|
|
│ │ │ ├── downloads.py # Download history
|
|
│ │ │ ├── credentials.py # Cookie management
|
|
│ │ │ ├── settings.py # App settings
|
|
│ │ │ └── websocket.py # Real-time events
|
|
│ │ │
|
|
│ │ ├── models/
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── source.py
|
|
│ │ │ ├── download.py
|
|
│ │ │ ├── credential.py
|
|
│ │ │ └── content.py
|
|
│ │ │
|
|
│ │ ├── services/
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── gallery_dl.py # gallery-dl subprocess wrapper
|
|
│ │ │ ├── download_manager.py
|
|
│ │ │ ├── credential_manager.py
|
|
│ │ │ └── manifest.py # Manifest generation
|
|
│ │ │
|
|
│ │ ├── tasks/
|
|
│ │ │ ├── __init__.py
|
|
│ │ │ ├── celery_app.py # Celery configuration
|
|
│ │ │ ├── downloads.py # Download tasks
|
|
│ │ │ └── scheduler.py # Scheduled jobs
|
|
│ │ │
|
|
│ │ └── utils/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── cookies.py # Cookie parsing
|
|
│ │ └── encryption.py # Credential encryption
|
|
│ │
|
|
│ ├── alembic/
|
|
│ │ ├── env.py
|
|
│ │ └── versions/
|
|
│ │
|
|
│ └── tests/
|
|
│
|
|
├── frontend/
|
|
│ ├── Dockerfile
|
|
│ ├── package.json
|
|
│ ├── vite.config.js
|
|
│ │
|
|
│ └── src/
|
|
│ ├── main.js
|
|
│ ├── App.vue
|
|
│ ├── router/
|
|
│ ├── stores/
|
|
│ ├── views/
|
|
│ │ ├── Dashboard.vue
|
|
│ │ ├── Sources.vue
|
|
│ │ ├── Downloads.vue
|
|
│ │ ├── NewContent.vue
|
|
│ │ └── Settings.vue
|
|
│ ├── components/
|
|
│ └── services/
|
|
│ └── api.js
|
|
│
|
|
├── extension/
|
|
│ ├── manifest.json
|
|
│ ├── popup/
|
|
│ │ ├── popup.html
|
|
│ │ ├── popup.css
|
|
│ │ └── popup.js
|
|
│ ├── background/
|
|
│ │ └── background.js
|
|
│ └── icons/
|
|
│
|
|
└── scripts/
|
|
├── migrate_legacy.py # Import from old project
|
|
└── init_db.py
|
|
```
|
|
|
|
---
|
|
|
|
## Database Schema
|
|
|
|
```sql
|
|
-- Sources (artists/creators to download from)
|
|
CREATE TABLE sources (
|
|
id SERIAL PRIMARY KEY,
|
|
name VARCHAR(255) NOT NULL,
|
|
platform VARCHAR(50) NOT NULL, -- patreon, subscribestar, hentaifoundry, discord
|
|
url TEXT NOT NULL,
|
|
enabled BOOLEAN DEFAULT true,
|
|
priority INTEGER DEFAULT 0,
|
|
check_interval INTEGER DEFAULT 3600, -- seconds between checks
|
|
last_check TIMESTAMP,
|
|
last_success TIMESTAMP,
|
|
error_count INTEGER DEFAULT 0,
|
|
metadata JSONB DEFAULT '{}',
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
UNIQUE(platform, url)
|
|
);
|
|
|
|
-- Downloads (individual download jobs)
|
|
CREATE TABLE downloads (
|
|
id SERIAL PRIMARY KEY,
|
|
source_id INTEGER REFERENCES sources(id) ON DELETE SET NULL,
|
|
url TEXT NOT NULL,
|
|
status VARCHAR(20) DEFAULT 'pending', -- pending, running, completed, failed, skipped
|
|
error_type VARCHAR(50), -- auth_error, rate_limited, not_found, etc.
|
|
error_message TEXT,
|
|
file_count INTEGER DEFAULT 0,
|
|
started_at TIMESTAMP,
|
|
completed_at TIMESTAMP,
|
|
metadata JSONB DEFAULT '{}',
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
CREATE INDEX idx_downloads_status ON downloads(status);
|
|
CREATE INDEX idx_downloads_source ON downloads(source_id);
|
|
|
|
-- Content Items (discovered files)
|
|
CREATE TABLE content_items (
|
|
id SERIAL PRIMARY KEY,
|
|
source_id INTEGER REFERENCES sources(id) ON DELETE CASCADE,
|
|
download_id INTEGER REFERENCES downloads(id) ON DELETE SET NULL,
|
|
external_id VARCHAR(255), -- Platform-specific ID
|
|
title VARCHAR(500),
|
|
content_type VARCHAR(50), -- image, video, attachment
|
|
file_path VARCHAR(500),
|
|
file_size BIGINT,
|
|
thumbnail_path VARCHAR(500),
|
|
published_at TIMESTAMP,
|
|
discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
metadata JSONB DEFAULT '{}',
|
|
UNIQUE(source_id, external_id)
|
|
);
|
|
|
|
CREATE INDEX idx_content_discovered ON content_items(discovered_at DESC);
|
|
|
|
-- Credentials (encrypted cookie/token storage)
|
|
CREATE TABLE credentials (
|
|
id SERIAL PRIMARY KEY,
|
|
platform VARCHAR(50) NOT NULL UNIQUE,
|
|
credential_type VARCHAR(20) NOT NULL, -- cookies, token
|
|
data BYTEA NOT NULL, -- Encrypted
|
|
expires_at TIMESTAMP,
|
|
last_verified TIMESTAMP,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
-- Settings (key-value store)
|
|
CREATE TABLE settings (
|
|
key VARCHAR(100) PRIMARY KEY,
|
|
value JSONB NOT NULL,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
---
|
|
|
|
## API Specification
|
|
|
|
### Sources
|
|
|
|
```
|
|
GET /api/sources List all sources (with pagination)
|
|
POST /api/sources Add new source
|
|
GET /api/sources/:id Get source details
|
|
PATCH /api/sources/:id Update source
|
|
DELETE /api/sources/:id Delete source
|
|
POST /api/sources/:id/check Trigger immediate download check
|
|
```
|
|
|
|
### Downloads
|
|
|
|
```
|
|
GET /api/downloads List download history
|
|
GET /api/downloads/:id Get download details with logs
|
|
POST /api/downloads/:id/retry Retry failed download
|
|
GET /api/downloads/stats Get statistics (by platform, by day)
|
|
```
|
|
|
|
### Content
|
|
|
|
```
|
|
GET /api/content List content items (filterable)
|
|
GET /api/content/new Get new content since timestamp
|
|
```
|
|
|
|
### Credentials
|
|
|
|
```
|
|
GET /api/credentials List credential status (no secrets)
|
|
POST /api/credentials Upload cookies (manual or from extension)
|
|
DELETE /api/credentials/:platform Remove credentials for platform
|
|
POST /api/credentials/verify Verify credentials are working
|
|
```
|
|
|
|
### Settings
|
|
|
|
```
|
|
GET /api/settings Get all settings
|
|
PATCH /api/settings Update settings
|
|
GET /api/settings/gallery-dl Get gallery-dl config
|
|
PUT /api/settings/gallery-dl Update gallery-dl config
|
|
```
|
|
|
|
### WebSocket
|
|
|
|
```
|
|
WS /ws/events
|
|
|
|
Events (server -> client):
|
|
download.started { download_id, source_id, url }
|
|
download.progress { download_id, file_count }
|
|
download.completed { download_id, file_count }
|
|
download.failed { download_id, error_type, error_message }
|
|
content.new { content_id, source_name, title, thumbnail }
|
|
credential.expiring { platform, expires_in_hours }
|
|
```
|
|
|
|
---
|
|
|
|
## Firefox Extension
|
|
|
|
### Features
|
|
|
|
- Export cookies for supported platforms (Patreon, SubscribeStar, HentaiFoundry)
|
|
- Export Discord token
|
|
- Auto-refresh option (re-export when cookies change)
|
|
- Connection status indicator
|
|
- Manual export button per platform
|
|
|
|
### Supported Platforms
|
|
|
|
| Platform | Auth Type | Domains |
|
|
|----------|-----------|---------|
|
|
| Patreon | Cookies | `.patreon.com` |
|
|
| SubscribeStar | Cookies | `.subscribestar.com`, `.subscribestar.adult` |
|
|
| HentaiFoundry | Cookies | `.hentai-foundry.com` |
|
|
| Discord | Token | `.discord.com` |
|
|
|
|
### Manual Alternative
|
|
|
|
For users who don't want the extension:
|
|
1. Export cookies using browser dev tools or "cookies.txt" extension
|
|
2. Upload via Settings page in web UI
|
|
3. Paste cookie content or upload file
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables (.env)
|
|
|
|
```bash
|
|
# Database
|
|
DATABASE_URL=postgresql://gdl:password@db:5432/gallery_subscriber
|
|
|
|
# Redis
|
|
REDIS_URL=redis://redis:6379/0
|
|
|
|
# Security
|
|
SECRET_KEY=your-secret-key-at-least-32-characters
|
|
EXTENSION_API_KEY=key-for-firefox-extension
|
|
|
|
# Paths (inside container)
|
|
DOWNLOAD_PATH=/data/downloads
|
|
CONFIG_PATH=/data/config
|
|
COOKIES_PATH=/data/cookies
|
|
|
|
# Download settings
|
|
DOWNLOAD_PARALLEL_LIMIT=3
|
|
DOWNLOAD_RATE_LIMIT=3.0
|
|
DEFAULT_CHECK_INTERVAL=3600
|
|
|
|
# Server
|
|
HOST=0.0.0.0
|
|
PORT=8080
|
|
```
|
|
|
|
### docker-compose.yml
|
|
|
|
```yaml
|
|
services:
|
|
app:
|
|
build: ./backend
|
|
ports:
|
|
- "8080:8080"
|
|
environment:
|
|
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
|
|
- REDIS_URL=redis://redis:6379/0
|
|
- SECRET_KEY=${SECRET_KEY}
|
|
- EXTENSION_API_KEY=${EXTENSION_API_KEY}
|
|
volumes:
|
|
- downloads:/data/downloads
|
|
- config:/data/config
|
|
- cookies:/data/cookies
|
|
depends_on:
|
|
- db
|
|
- redis
|
|
restart: unless-stopped
|
|
|
|
celery:
|
|
build: ./backend
|
|
command: celery -A app.tasks.celery_app worker -B --loglevel=info
|
|
environment:
|
|
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
|
|
- REDIS_URL=redis://redis:6379/0
|
|
- SECRET_KEY=${SECRET_KEY}
|
|
volumes:
|
|
- downloads:/data/downloads
|
|
- config:/data/config
|
|
- cookies:/data/cookies
|
|
depends_on:
|
|
- db
|
|
- redis
|
|
restart: unless-stopped
|
|
|
|
db:
|
|
image: postgres:15-alpine
|
|
environment:
|
|
- POSTGRES_USER=gdl
|
|
- POSTGRES_PASSWORD=${DB_PASSWORD}
|
|
- POSTGRES_DB=gallery_subscriber
|
|
volumes:
|
|
- postgres_data:/var/lib/postgresql/data
|
|
restart: unless-stopped
|
|
|
|
redis:
|
|
image: redis:7-alpine
|
|
restart: unless-stopped
|
|
|
|
volumes:
|
|
postgres_data:
|
|
downloads:
|
|
config:
|
|
cookies:
|
|
```
|
|
|
|
---
|
|
|
|
## Development Phases
|
|
|
|
### Phase 1: Foundation
|
|
- [ ] Set up project structure
|
|
- [ ] Create Docker Compose development environment
|
|
- [ ] Initialize Quart app with basic routing
|
|
- [ ] Set up PostgreSQL + Alembic migrations
|
|
- [ ] Create database models
|
|
- [ ] Basic API endpoints (sources CRUD)
|
|
|
|
### Phase 2: Download Engine
|
|
- [ ] Port gallery-dl wrapper from legacy project
|
|
- [ ] Implement Celery tasks for downloads
|
|
- [ ] Add platform-specific locking (from legacy)
|
|
- [ ] Error categorization and handling
|
|
- [ ] Download history tracking
|
|
|
|
### Phase 3: Scheduling & State
|
|
- [ ] Celery Beat for scheduled downloads
|
|
- [ ] Round-robin scheduling logic (from legacy)
|
|
- [ ] Priority queue for failed tasks
|
|
- [ ] State persistence in PostgreSQL
|
|
|
|
### Phase 4: Frontend
|
|
- [ ] Vue 3 + Vuetify scaffold
|
|
- [ ] Dashboard view
|
|
- [ ] Sources management UI
|
|
- [ ] Download history view
|
|
- [ ] Settings page
|
|
|
|
### Phase 5: Authentication
|
|
- [ ] Credential storage (encrypted)
|
|
- [ ] Manual cookie upload via web UI
|
|
- [ ] Firefox extension
|
|
- [ ] Credential verification
|
|
|
|
### Phase 6: Real-time & Content
|
|
- [ ] WebSocket event streaming
|
|
- [ ] New content feed
|
|
- [ ] Content browsing with thumbnails
|
|
- [ ] Notifications
|
|
|
|
### Phase 7: Polish & Migration
|
|
- [ ] Migration script for legacy sources.yaml
|
|
- [ ] Import legacy gallery-dl.conf
|
|
- [ ] Testing
|
|
- [ ] Documentation
|
|
|
|
---
|
|
|
|
## Migration from Legacy Project
|
|
|
|
### What Gets Migrated
|
|
|
|
| Legacy File | Destination |
|
|
|-------------|-------------|
|
|
| `sources.yaml` | `sources` table |
|
|
| `gallery-dl.conf` | `/data/config/gallery-dl.conf` (paths updated) |
|
|
| `state.json` | `sources` table (last_check, error_count) |
|
|
| Download history | Optional - start fresh or import logs |
|
|
|
|
### Migration Script Usage
|
|
|
|
```bash
|
|
python scripts/migrate_legacy.py \
|
|
--sources /path/to/sources.yaml \
|
|
--config /path/to/gallery-dl.conf \
|
|
--state /path/to/state.json
|
|
```
|
|
|
|
### Key Patterns Preserved from Legacy
|
|
|
|
1. **Platform locking** - One download per platform at a time
|
|
2. **Round-robin scheduling** - Fair distribution across sources
|
|
3. **Priority queue** - Failed tasks get priority on next run
|
|
4. **Error categorization** - AUTH_ERROR, RATE_LIMITED, NOT_FOUND, etc.
|
|
5. **Rate limiting** - Configurable delay between requests
|
|
|
|
---
|
|
|
|
*Document Version: 2.0*
|
|
*Created: 2025-01-24*
|
|
*Project: Gallery Subscriber*
|