bvandeusen/GallerySubscriber
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

rapid interations of server side app and firefox extension

Browse Source
This commit is contained in:
Bryan Van Deusen
2026-01-24 22:52:51 -05:00
commit b9b8048a2d
81 changed files with 9675 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_PLAN.md
+548
View File
@@ -0,0 +1,548 @@
# Gallery Subscriber
A Docker-based web application for managing gallery-dl downloads with a modern web interface.
---
## Table of Contents
1. [Project Overview](#project-overview)
2. [Architecture](#architecture)
3. [Technology Stack](#technology-stack)
4. [Project Structure](#project-structure)
5. [Database Schema](#database-schema)
6. [API Specification](#api-specification)
7. [Firefox Extension](#firefox-extension)
8. [Configuration](#configuration)
9. [Development Phases](#development-phases)
10. [Migration from Legacy Project](#migration-from-legacy-project)
---
## Project Overview
### Goals
- **Containerized Deployment**: Run the entire download manager in Docker for portability
- **Web Interface**: Modern UI for managing subscriptions, viewing downloads, and monitoring new content
- **Flexible Authentication**: Firefox extension for cookie export OR manual cookie upload via web UI
- **Notifications**: Real-time updates when new content is downloaded
- **Reliability**: Maintain the robust scheduling and error handling from the original project
### Non-Goals (Initial Release)
- Multi-user support (single-user initially)
- Mobile app
- SSL termination (handled by external reverse proxy)
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ External Reverse Proxy │
│ (Traefik/Caddy/nginx - handles SSL) │
└──────────────────────────┬──────────────────────────────────────┘
│ HTTP (port 8080)
▼
┌─────────────────────────────────────────────────────────────────┐
│ Docker Compose Stack │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Quart Backend │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ │ │
│ │ │ REST API │ │WebSocket │ │ Static │ │ gallery-dl│ │ │
│ │ │ /api/* │ │ /ws/* │ │ Files │ │ Wrapper │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └───────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────┴───────────────────────────────┐ │
│ │ Celery Worker │ │
│ │ • Download tasks • Scheduled checks │ │
│ │ • Manifest generation • Error retry logic │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┐ ┌─────────┴───────┐ ┌──────────────────┐ │
│ │ PostgreSQL │ │ Redis │ │ Vue Frontend │ │
│ │ (data) │ │ (task queue) │ │ (static build) │ │
│ └──────────────┘ └─────────────────┘ └──────────────────┘ │
│ │
│ Volumes: │
│ ├── /data/downloads (downloaded content) │
│ ├── /data/config (gallery-dl.conf) │
│ ├── /data/cookies (cookie files) │
│ └── /data/db (PostgreSQL data) │
└─────────────────────────────────────────────────────────────────┘
```
---
## Technology Stack
### Backend
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Web Framework | **Quart** | Flask-like API with native async, WebSocket support |
| Task Queue | **Celery + Redis** | Reliable background tasks, scheduling with Celery Beat |
| Database | **PostgreSQL** | JSON fields for metadata, robust concurrency |
| ORM | **SQLAlchemy 2.0** | Async support, mature ecosystem |
| Migrations | **Alembic** | Standard for SQLAlchemy |
### Frontend
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Framework | **Vue 3** | Reactive, component-based |
| UI Library | **Vuetify 3** | Material Design, comprehensive components |
| State | **Pinia** | Official Vue state management |
| HTTP Client | **Axios** | Request interceptors for auth |
| WebSocket | **Native** | Real-time updates |
### Infrastructure
| Component | Technology | Rationale |
|-----------|------------|-----------|
| Container | **Docker Compose** | Easy deployment |
| Process Manager | **Supervisor** | Manage Quart + Celery in container |
---
## Project Structure
```
gallery-subscriber/
├── docker-compose.yml
├── docker-compose.dev.yml
├── .env.example
├── README.md
│
├── backend/
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── supervisord.conf
│ │
│ ├── app/
│ │ ├── __init__.py
│ │ ├── main.py # Quart app factory
│ │ ├── config.py # Settings (from env)
│ │ │
│ │ ├── api/
│ │ │ ├── __init__.py
│ │ │ ├── sources.py # Source CRUD
│ │ │ ├── downloads.py # Download history
│ │ │ ├── credentials.py # Cookie management
│ │ │ ├── settings.py # App settings
│ │ │ └── websocket.py # Real-time events
│ │ │
│ │ ├── models/
│ │ │ ├── __init__.py
│ │ │ ├── source.py
│ │ │ ├── download.py
│ │ │ ├── credential.py
│ │ │ └── content.py
│ │ │
│ │ ├── services/
│ │ │ ├── __init__.py
│ │ │ ├── gallery_dl.py # gallery-dl subprocess wrapper
│ │ │ ├── download_manager.py
│ │ │ ├── credential_manager.py
│ │ │ └── manifest.py # Manifest generation
│ │ │
│ │ ├── tasks/
│ │ │ ├── __init__.py
│ │ │ ├── celery_app.py # Celery configuration
│ │ │ ├── downloads.py # Download tasks
│ │ │ └── scheduler.py # Scheduled jobs
│ │ │
│ │ └── utils/
│ │ ├── __init__.py
│ │ ├── cookies.py # Cookie parsing
│ │ └── encryption.py # Credential encryption
│ │
│ ├── alembic/
│ │ ├── env.py
│ │ └── versions/
│ │
│ └── tests/
│
├── frontend/
│ ├── Dockerfile
│ ├── package.json
│ ├── vite.config.js
│ │
│ └── src/
│ ├── main.js
│ ├── App.vue
│ ├── router/
│ ├── stores/
│ ├── views/
│ │ ├── Dashboard.vue
│ │ ├── Sources.vue
│ │ ├── Downloads.vue
│ │ ├── NewContent.vue
│ │ └── Settings.vue
│ ├── components/
│ └── services/
│ └── api.js
│
├── extension/
│ ├── manifest.json
│ ├── popup/
│ │ ├── popup.html
│ │ ├── popup.css
│ │ └── popup.js
│ ├── background/
│ │ └── background.js
│ └── icons/
│
└── scripts/
├── migrate_legacy.py # Import from old project
└── init_db.py
```
---
## Database Schema
```sql
-- Sources (artists/creators to download from)
CREATE TABLE sources (
id SERIAL PRIMARY KEY,
name VARCHAR(255) NOT NULL,
platform VARCHAR(50) NOT NULL, -- patreon, subscribestar, hentaifoundry, discord
url TEXT NOT NULL,
enabled BOOLEAN DEFAULT true,
priority INTEGER DEFAULT 0,
check_interval INTEGER DEFAULT 3600, -- seconds between checks
last_check TIMESTAMP,
last_success TIMESTAMP,
error_count INTEGER DEFAULT 0,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(platform, url)
);
-- Downloads (individual download jobs)
CREATE TABLE downloads (
id SERIAL PRIMARY KEY,
source_id INTEGER REFERENCES sources(id) ON DELETE SET NULL,
url TEXT NOT NULL,
status VARCHAR(20) DEFAULT 'pending', -- pending, running, completed, failed, skipped
error_type VARCHAR(50), -- auth_error, rate_limited, not_found, etc.
error_message TEXT,
file_count INTEGER DEFAULT 0,
started_at TIMESTAMP,
completed_at TIMESTAMP,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_downloads_status ON downloads(status);
CREATE INDEX idx_downloads_source ON downloads(source_id);
-- Content Items (discovered files)
CREATE TABLE content_items (
id SERIAL PRIMARY KEY,
source_id INTEGER REFERENCES sources(id) ON DELETE CASCADE,
download_id INTEGER REFERENCES downloads(id) ON DELETE SET NULL,
external_id VARCHAR(255), -- Platform-specific ID
title VARCHAR(500),
content_type VARCHAR(50), -- image, video, attachment
file_path VARCHAR(500),
file_size BIGINT,
thumbnail_path VARCHAR(500),
published_at TIMESTAMP,
discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
metadata JSONB DEFAULT '{}',
UNIQUE(source_id, external_id)
);
CREATE INDEX idx_content_discovered ON content_items(discovered_at DESC);
-- Credentials (encrypted cookie/token storage)
CREATE TABLE credentials (
id SERIAL PRIMARY KEY,
platform VARCHAR(50) NOT NULL UNIQUE,
credential_type VARCHAR(20) NOT NULL, -- cookies, token
data BYTEA NOT NULL, -- Encrypted
expires_at TIMESTAMP,
last_verified TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Settings (key-value store)
CREATE TABLE settings (
key VARCHAR(100) PRIMARY KEY,
value JSONB NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
---
## API Specification
### Sources
```
GET /api/sources List all sources (with pagination)
POST /api/sources Add new source
GET /api/sources/:id Get source details
PATCH /api/sources/:id Update source
DELETE /api/sources/:id Delete source
POST /api/sources/:id/check Trigger immediate download check
```
### Downloads
```
GET /api/downloads List download history
GET /api/downloads/:id Get download details with logs
POST /api/downloads/:id/retry Retry failed download
GET /api/downloads/stats Get statistics (by platform, by day)
```
### Content
```
GET /api/content List content items (filterable)
GET /api/content/new Get new content since timestamp
```
### Credentials
```
GET /api/credentials List credential status (no secrets)
POST /api/credentials Upload cookies (manual or from extension)
DELETE /api/credentials/:platform Remove credentials for platform
POST /api/credentials/verify Verify credentials are working
```
### Settings
```
GET /api/settings Get all settings
PATCH /api/settings Update settings
GET /api/settings/gallery-dl Get gallery-dl config
PUT /api/settings/gallery-dl Update gallery-dl config
```
### WebSocket
```
WS /ws/events
Events (server -> client):
download.started { download_id, source_id, url }
download.progress { download_id, file_count }
download.completed { download_id, file_count }
download.failed { download_id, error_type, error_message }
content.new { content_id, source_name, title, thumbnail }
credential.expiring { platform, expires_in_hours }
```
---
## Firefox Extension
### Features
- Export cookies for supported platforms (Patreon, SubscribeStar, HentaiFoundry)
- Export Discord token
- Auto-refresh option (re-export when cookies change)
- Connection status indicator
- Manual export button per platform
### Supported Platforms
| Platform | Auth Type | Domains |
|----------|-----------|---------|
| Patreon | Cookies | `.patreon.com` |
| SubscribeStar | Cookies | `.subscribestar.com`, `.subscribestar.adult` |
| HentaiFoundry | Cookies | `.hentai-foundry.com` |
| Discord | Token | `.discord.com` |
### Manual Alternative
For users who don't want the extension:
1. Export cookies using browser dev tools or "cookies.txt" extension
2. Upload via Settings page in web UI
3. Paste cookie content or upload file
---
## Configuration
### Environment Variables (.env)
```bash
# Database
DATABASE_URL=postgresql://gdl:password@db:5432/gallery_subscriber
# Redis
REDIS_URL=redis://redis:6379/0
# Security
SECRET_KEY=your-secret-key-at-least-32-characters
EXTENSION_API_KEY=key-for-firefox-extension
# Paths (inside container)
DOWNLOAD_PATH=/data/downloads
CONFIG_PATH=/data/config
COOKIES_PATH=/data/cookies
# Download settings
DOWNLOAD_PARALLEL_LIMIT=3
DOWNLOAD_RATE_LIMIT=3.0
DEFAULT_CHECK_INTERVAL=3600
# Server
HOST=0.0.0.0
PORT=8080
```
### docker-compose.yml
```yaml
services:
app:
build: ./backend
ports:
- "8080:8080"
environment:
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
- REDIS_URL=redis://redis:6379/0
- SECRET_KEY=${SECRET_KEY}
- EXTENSION_API_KEY=${EXTENSION_API_KEY}
volumes:
- downloads:/data/downloads
- config:/data/config
- cookies:/data/cookies
depends_on:
- db
- redis
restart: unless-stopped
celery:
build: ./backend
command: celery -A app.tasks.celery_app worker -B --loglevel=info
environment:
- DATABASE_URL=postgresql://gdl:${DB_PASSWORD}@db:5432/gallery_subscriber
- REDIS_URL=redis://redis:6379/0
- SECRET_KEY=${SECRET_KEY}
volumes:
- downloads:/data/downloads
- config:/data/config
- cookies:/data/cookies
depends_on:
- db
- redis
restart: unless-stopped
db:
image: postgres:15-alpine
environment:
- POSTGRES_USER=gdl
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=gallery_subscriber
volumes:
- postgres_data:/var/lib/postgresql/data
restart: unless-stopped
redis:
image: redis:7-alpine
restart: unless-stopped
volumes:
postgres_data:
downloads:
config:
cookies:
```
---
## Development Phases
### Phase 1: Foundation
- [ ] Set up project structure
- [ ] Create Docker Compose development environment
- [ ] Initialize Quart app with basic routing
- [ ] Set up PostgreSQL + Alembic migrations
- [ ] Create database models
- [ ] Basic API endpoints (sources CRUD)
### Phase 2: Download Engine
- [ ] Port gallery-dl wrapper from legacy project
- [ ] Implement Celery tasks for downloads
- [ ] Add platform-specific locking (from legacy)
- [ ] Error categorization and handling
- [ ] Download history tracking
### Phase 3: Scheduling & State
- [ ] Celery Beat for scheduled downloads
- [ ] Round-robin scheduling logic (from legacy)
- [ ] Priority queue for failed tasks
- [ ] State persistence in PostgreSQL
### Phase 4: Frontend
- [ ] Vue 3 + Vuetify scaffold
- [ ] Dashboard view
- [ ] Sources management UI
- [ ] Download history view
- [ ] Settings page
### Phase 5: Authentication
- [ ] Credential storage (encrypted)
- [ ] Manual cookie upload via web UI
- [ ] Firefox extension
- [ ] Credential verification
### Phase 6: Real-time & Content
- [ ] WebSocket event streaming
- [ ] New content feed
- [ ] Content browsing with thumbnails
- [ ] Notifications
### Phase 7: Polish & Migration
- [ ] Migration script for legacy sources.yaml
- [ ] Import legacy gallery-dl.conf
- [ ] Testing
- [ ] Documentation
---
## Migration from Legacy Project
### What Gets Migrated
| Legacy File | Destination |
|-------------|-------------|
| `sources.yaml` | `sources` table |
| `gallery-dl.conf` | `/data/config/gallery-dl.conf` (paths updated) |
| `state.json` | `sources` table (last_check, error_count) |
| Download history | Optional - start fresh or import logs |
### Migration Script Usage
```bash
python scripts/migrate_legacy.py \
--sources /path/to/sources.yaml \
--config /path/to/gallery-dl.conf \
--state /path/to/state.json
```
### Key Patterns Preserved from Legacy
1. **Platform locking** - One download per platform at a time
2. **Round-robin scheduling** - Fair distribution across sources
3. **Priority queue** - Failed tasks get priority on next run
4. **Error categorization** - AUTH_ERROR, RATE_LIMITED, NOT_FOUND, etc.
5. **Rate limiting** - Configurable delay between requests
---
*Document Version: 2.0*
*Created: 2025-01-24*
*Project: Gallery Subscriber*