Bryan Van Deusen
18 KiB
18 KiB
Gallery Subscriber
A Docker-based web application for managing gallery-dl downloads with a modern web interface.
Table of Contents
- Project Overview
- Architecture
- Technology Stack
- Project Structure
- Database Schema
- API Specification
- Firefox Extension
- Configuration
- Development Phases
- Migration from Legacy Project
Project Overview
Goals
- Containerized Deployment: Run the entire download manager in Docker for portability
- Web Interface: Modern UI for managing subscriptions, viewing downloads, and monitoring new content
- Flexible Authentication: Firefox extension for cookie export OR manual cookie upload via web UI
- Notifications: Real-time updates when new content is downloaded
- Reliability: Maintain the robust scheduling and error handling from the original project
Non-Goals (Initial Release)
- Multi-user support (single-user initially)
- Mobile app
- SSL termination (handled by external reverse proxy)
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ External Reverse Proxy │
│ (Traefik/Caddy/nginx - handles SSL) │
└──────────────────────────┬──────────────────────────────────────┘
│ HTTP (port 8080)
▼
┌─────────────────────────────────────────────────────────────────┐
│ Docker Compose Stack │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Quart Backend │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ │ │
│ │ │ REST API │ │WebSocket │ │ Static │ │ gallery-dl│ │ │
│ │ │ /api/* │ │ /ws/* │ │ Files │ │ Wrapper │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └───────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────┴───────────────────────────────┐ │
│ │ Celery Worker │ │
│ │ • Download tasks • Scheduled checks │ │
│ │ • Manifest generation • Error retry logic │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┐ ┌─────────┴───────┐ ┌──────────────────┐ │
│ │ PostgreSQL │ │ Redis │ │ Vue Frontend │ │
│ │ (data) │ │ (task queue) │ │ (static build) │ │
│ └──────────────┘ └─────────────────┘ └──────────────────┘ │
│ │
│ Volumes: │
│ ├── /data/downloads (downloaded content) │
│ ├── /data/config (gallery-dl.conf) │
│ ├── /data/cookies (cookie files) │
│ └── /data/db (PostgreSQL data) │
└─────────────────────────────────────────────────────────────────┘
Technology Stack
Backend
| Component | Technology | Rationale |
|---|---|---|
| Web Framework | Quart | Flask-like API with native async, WebSocket support |
| Task Queue | Celery + Redis | Reliable background tasks, scheduling with Celery Beat |
| Database | PostgreSQL | JSON fields for metadata, robust concurrency |
| ORM | SQLAlchemy 2.0 | Async support, mature ecosystem |
| Migrations | Alembic | Standard for SQLAlchemy |
Frontend
| Component | Technology | Rationale |
|---|---|---|
| Framework | Vue 3 | Reactive, component-based |
| UI Library | Vuetify 3 | Material Design, comprehensive components |
| State | Pinia | Official Vue state management |
| HTTP Client | Axios | Request interceptors for auth |
| WebSocket | Native | Real-time updates |
Infrastructure
| Component | Technology | Rationale |
|---|---|---|
| Container | Docker Compose | Easy deployment |
| Process Manager | Supervisor | Manage Quart + Celery in container |
Project Structure
gallery-subscriber/
├── docker-compose.yml
├── docker-compose.dev.yml
├── .env.example
├── README.md
│
├── backend/
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── supervisord.conf
│ │
│ ├── app/
│ │ ├── __init__.py
│ │ ├── main.py # Quart app factory
│ │ ├── config.py # Settings (from env)
│ │ │
│ │ ├── api/
│ │ │ ├── __init__.py
│ │ │ ├── sources.py # Source CRUD
│ │ │ ├── downloads.py # Download history
│ │ │ ├── credentials.py # Cookie management
│ │ │ ├── settings.py # App settings
│ │ │ └── websocket.py # Real-time events
│ │ │
│ │ ├── models/
│ │ │ ├── __init__.py
│ │ │ ├── source.py
│ │ │ ├── download.py
│ │ │ ├── credential.py
│ │ │ └── content.py
│ │ │
│ │ ├── services/
│ │ │ ├── __init__.py
│ │ │ ├── gallery_dl.py # gallery-dl subprocess wrapper
│ │ │ ├── download_manager.py
│ │ │ ├── credential_manager.py
│ │ │ └── manifest.py # Manifest generation
│ │ │
│ │ ├── tasks/
│ │ │ ├── __init__.py
│ │ │ ├── celery_app.py # Celery configuration
│ │ │ ├── downloads.py # Download tasks
│ │ │ └── scheduler.py # Scheduled jobs
│ │ │
│ │ └── utils/
│ │ ├── __init__.py
│ │ ├── cookies.py # Cookie parsing
│ │ └── encryption.py # Credential encryption
│ │
│ ├── alembic/
│ │ ├── env.py
│ │ └── versions/
│ │
│ └── tests/
│
├── frontend/
│ ├── Dockerfile
│ ├── package.json
│ ├── vite.config.js
│ │
│ └── src/
│ ├── main.js
│ ├── App.vue
│ ├── router/
│ ├── stores/
│ ├── views/
│ │ ├── Dashboard.vue
│ │ ├── Sources.vue
│ │ ├── Downloads.vue
│ │ ├── NewContent.vue
│ │ └── Settings.vue
│ ├── components/
│ └── services/
│ └── api.js
│
├── extension/
│ ├── manifest.json
│ ├── popup/
│ │ ├── popup.html
│ │ ├── popup.css
│ │ └── popup.js
│ ├── background/
│ │ └── background.js
│ └── icons/
│
└── scripts/
├── migrate_legacy.py # Import from old project
└── init_db.py
Database Schema
-- Sources (artists/creators to download from)
CREATE TABLE sources (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
platform VARCHAR(50) NOT NULL, -- patreon, subscribestar, hentaifoundry, discord
url TEXT NOT NULL,
enabled BOOLEAN DEFAULT true,
priority INTEGER DEFAULT 0,
check_interval INTEGER DEFAULT 3600, -- seconds between checks
last_check TIMESTAMP,
last_success TIMESTAMP,
error_count INTEGER DEFAULT 0,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(platform, url)
);
-- Downloads (individual download jobs)
CREATE TABLE downloads (
id SERIAL PRIMARY KEY,
source_id INTEGER REFERENCES sources(id) ON DELETE SET NULL,
url TEXT NOT NULL,
status VARCHAR(20) DEFAULT 'pending', -- pending, running, completed, failed, skipped
error_type VARCHAR(50), -- auth_error, rate_limited, not_found, etc.
error_message TEXT,
file_count INTEGER DEFAULT 0,
started_at TIMESTAMP,
completed_at TIMESTAMP,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_downloads_status ON downloads(status);
CREATE INDEX idx_downloads_source ON downloads(source_id);
-- Content Items (discovered files)
CREATE TABLE content_items (
id SERIAL PRIMARY KEY,
source_id INTEGER REFERENCES sources(id) ON DELETE CASCADE,
download_id INTEGER REFERENCES downloads(id) ON DELETE SET NULL,
external_id VARCHAR(255), -- Platform-specific ID
title VARCHAR(500),
content_type VARCHAR(50), -- image, video, attachment
file_path VARCHAR(500),
file_size BIGINT,
thumbnail_path VARCHAR(500),
published_at TIMESTAMP,
discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
metadata JSONB DEFAULT '{}',
UNIQUE(source_id, external_id)
);
CREATE INDEX idx_content_discovered ON content_items(discovered_at DESC);
-- Credentials (encrypted cookie/token storage)
CREATE TABLE credentials (
id SERIAL PRIMARY KEY,
platform VARCHAR(50) NOT NULL UNIQUE,
credential_type VARCHAR(20) NOT NULL, -- cookies, token
data BYTEA NOT NULL, -- Encrypted
expires_at TIMESTAMP,
last_verified TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Settings (key-value store)
CREATE TABLE settings (
key VARCHAR(100) PRIMARY KEY,
value JSONB NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
API Specification
Sources
GET /api/sources List all sources (with pagination)
POST /api/sources Add new source
GET /api/sources/:id Get source details
PATCH /api/sources/:id Update source
DELETE /api/sources/:id Delete source
POST /api/sources/:id/check Trigger immediate download check
Downloads
GET /api/downloads List download history
GET /api/downloads/:id Get download details with logs
POST /api/downloads/:id/retry Retry failed download
GET /api/downloads/stats Get statistics (by platform, by day)
Content
GET /api/content List content items (filterable)
GET /api/content/new Get new content since timestamp
Credentials
GET /api/credentials List credential status (no secrets)
POST /api/credentials Upload cookies (manual or from extension)
DELETE /api/credentials/:platform Remove credentials for platform
POST /api/credentials/verify Verify credentials are working
Settings
GET /api/settings Get all settings
PATCH /api/settings Update settings
GET /api/settings/gallery-dl Get gallery-dl config
PUT /api/settings/gallery-dl Update gallery-dl config
WebSocket
WS /ws/events
Events (server -> client):
download.started { download_id, source_id, url }
download.progress { download_id, file_count }
download.completed { download_id, file_count }
download.failed { download_id, error_type, error_message }
content.new { content_id, source_name, title, thumbnail }
credential.expiring { platform, expires_in_hours }
Firefox Extension
Features
- Export cookies for supported platforms (Patreon, SubscribeStar, HentaiFoundry)
- Export Discord token
- Auto-refresh option (re-export when cookies change)
- Connection status indicator
- Manual export button per platform
Supported Platforms
| Platform | Auth Type | Domains |
|---|---|---|
| Patreon | Cookies | .patreon.com |
| SubscribeStar | Cookies | .subscribestar.com, .subscribestar.adult |
| HentaiFoundry | Cookies | .hentai-foundry.com |
| Discord | Token | .discord.com |
Manual Alternative
For users who don't want the extension:
- Export cookies using browser dev tools or "cookies.txt" extension
- Upload via Settings page in web UI
- Paste cookie content or upload file
Configuration
Environment Variables (.env)
# Database
DATABASE_URL=postgresql://gdl:password@db:5432/gallery_subscriber
# Redis
REDIS_URL=redis://redis:6379/0
# Security
SECRET_KEY=your-secret-key-at-least-32-characters
EXTENSION_API_KEY=key-for-firefox-extension
# Paths (inside container)
DOWNLOAD_PATH=/data/downloads
CONFIG_PATH=/data/config
COOKIES_PATH=/data/cookies
# Download settings
DOWNLOAD_PARALLEL_LIMIT=3
DOWNLOAD_RATE_LIMIT=3.0
DEFAULT_CHECK_INTERVAL=3600
# Server
HOST=0.0.0.0
PORT=8080
docker-compose.yml
services:
app:
build: ./backend
ports:
- "8080:8080"
environment:
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
- REDIS_URL=redis://redis:6379/0
- SECRET_KEY=${SECRET_KEY}
- EXTENSION_API_KEY=${EXTENSION_API_KEY}
volumes:
- downloads:/data/downloads
- config:/data/config
- cookies:/data/cookies
depends_on:
- db
- redis
restart: unless-stopped
celery:
build: ./backend
command: celery -A app.tasks.celery_app worker -B --loglevel=info
environment:
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
- REDIS_URL=redis://redis:6379/0
- SECRET_KEY=${SECRET_KEY}
volumes:
- downloads:/data/downloads
- config:/data/config
- cookies:/data/cookies
depends_on:
- db
- redis
restart: unless-stopped
db:
image: postgres:15-alpine
environment:
- POSTGRES_USER=gdl
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=gallery_subscriber
volumes:
- postgres_data:/var/lib/postgresql/data
restart: unless-stopped
redis:
image: redis:7-alpine
restart: unless-stopped
volumes:
postgres_data:
downloads:
config:
cookies:
Development Phases
Phase 1: Foundation
- Set up project structure
- Create Docker Compose development environment
- Initialize Quart app with basic routing
- Set up PostgreSQL + Alembic migrations
- Create database models
- Basic API endpoints (sources CRUD)
Phase 2: Download Engine
- Port gallery-dl wrapper from legacy project
- Implement Celery tasks for downloads
- Add platform-specific locking (from legacy)
- Error categorization and handling
- Download history tracking
Phase 3: Scheduling & State
- Celery Beat for scheduled downloads
- Round-robin scheduling logic (from legacy)
- Priority queue for failed tasks
- State persistence in PostgreSQL
Phase 4: Frontend
- Vue 3 + Vuetify scaffold
- Dashboard view
- Sources management UI
- Download history view
- Settings page
Phase 5: Authentication
- Credential storage (encrypted)
- Manual cookie upload via web UI
- Firefox extension
- Credential verification
Phase 6: Real-time & Content
- WebSocket event streaming
- New content feed
- Content browsing with thumbnails
- Notifications
Phase 7: Polish & Migration
- Migration script for legacy sources.yaml
- Import legacy gallery-dl.conf
- Testing
- Documentation
Migration from Legacy Project
What Gets Migrated
| Legacy File | Destination |
|---|---|
sources.yaml |
sources table |
gallery-dl.conf |
/data/config/gallery-dl.conf (paths updated) |
state.json |
sources table (last_check, error_count) |
| Download history | Optional - start fresh or import logs |
Migration Script Usage
python scripts/migrate_legacy.py \
--sources /path/to/sources.yaml \
--config /path/to/gallery-dl.conf \
--state /path/to/state.json
Key Patterns Preserved from Legacy
- Platform locking - One download per platform at a time
- Round-robin scheduling - Fair distribution across sources
- Priority queue - Failed tasks get priority on next run
- Error categorization - AUTH_ERROR, RATE_LIMITED, NOT_FOUND, etc.
- Rate limiting - Configurable delay between requests
Document Version: 2.0 Created: 2025-01-24 Project: Gallery Subscriber