GallerySubscriber/summary.md

GallerySubscriber - Project Summary

Updated: 2026-01-29 (task routing, scheduler/worker separation, 8-hour default interval)

Overview

GallerySubscriber is a Docker-based web application that automates downloading content from subscription platforms (Patreon, SubscribeStar, Discord, Hentai Foundry). It combines a Vue 3 web dashboard, Firefox extension for credential export, and a Python backend with Celery for background task scheduling.

Core Purpose: Automatically download and organize content from multiple subscription platforms using gallery-dl as the underlying download engine.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    GallerySubscriber Stack                      │
├─────────────────────────────────────────────────────────────────┤
│ FRONTEND (Vue 3 + Vuetify)                                      │
│ - Served as static assets from backend (/static)               │
│ - SPA with client-side routing                                 │
├─────────────────────────────────────────────────────────────────┤
│ BACKEND API (Quart + SQLAlchemy)                               │
│ - Async Python web framework                                    │
│ - RESTful API + WebSocket for real-time updates                │
├─────────────────────────────────────────────────────────────────┤
│ CELERY WORKER (Background Tasks)                               │
│ - Scheduled downloads via Celery Beat                          │
│ - gallery-dl subprocess execution                              │
├─────────────────────────────────────────────────────────────────┤
│ DATABASE (PostgreSQL 15) + CACHE (Redis 7)                     │
└─────────────────────────────────────────────────────────────────┘

Technology Stack

Layer	Technologies
Backend	Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn
Frontend	Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11
Extension	Firefox WebExtension (Manifest V2)
Infrastructure	Docker, PostgreSQL 15, Redis 7
Download Engine	gallery-dl >=1.31.0

Project Structure

GallerySubscriber/
├── backend/app/
│   ├── main.py              # Quart app entry, serves frontend
│   ├── config.py            # Pydantic Settings from env
│   ├── events.py            # Redis pub/sub for WebSocket
│   ├── api/                 # API blueprints
│   │   ├── subscriptions.py # Artist/creator CRUD + check trigger
│   │   ├── sources.py       # Platform URL management
│   │   ├── downloads.py     # Download history
│   │   ├── credentials.py   # Encrypted credential storage
│   │   ├── settings.py      # App configuration
│   │   ├── platforms.py     # Platform schemas
│   │   └── websocket.py     # Real-time events
│   ├── models/              # SQLAlchemy models
│   │   ├── subscription.py  # Artist/creator groups
│   │   ├── source.py        # Platform URLs per subscription
│   │   ├── download.py      # Job records
│   │   ├── credential.py    # Encrypted auth data
│   │   ├── content.py       # Downloaded file records
│   │   └── setting.py       # Key-value config
│   ├── services/
│   │   ├── gallery_dl.py    # gallery-dl wrapper
│   │   └── credential_manager.py
│   ├── tasks/
│   │   ├── celery_app.py    # Celery configuration
│   │   ├── downloads.py     # Download tasks + scheduler
│   │   └── maintenance.py   # Cleanup jobs
│   └── utils/
│       ├── encryption.py    # Fernet crypto
│       └── cookies.py       # Netscape format handling
├── frontend/src/
│   ├── main.js              # Vue app entry
│   ├── views/               # Page components (Dashboard, Subscriptions, etc.)
│   ├── stores/              # Pinia state management
│   ├── services/api.js      # Axios HTTP client
│   └── composables/         # Reusable logic (useWebSocket)
├── extension/
│   ├── manifest.json        # WebExtension config
│   ├── background/          # Service worker for cookie/token capture
│   ├── popup/               # Extension popup UI
│   └── lib/                 # Platform definitions, API calls
├── alembic/                 # Database migrations
├── Dockerfile               # Multi-stage build (Node + Python)
├── docker-compose.yml       # Production stack
└── docker-compose.dev.yml   # Development with hot-reload

Key Entry Points

Component	Entry Point	Command
Backend API	backend/app/main.py	hypercorn app.main:app --bind 0.0.0.0:8080
Scheduler	backend/app/tasks/celery_app.py	celery -A app.tasks.celery_app worker --queues=scheduling --beat --concurrency=1
Download Workers	backend/app/tasks/celery_app.py	celery -A app.tasks.celery_app worker --queues=downloads
Frontend	frontend/src/main.js	Built with Vite to /static

Database Schema

Core Entities:

1. subscriptions - Artist/creator groups

   • name (unique), enabled, priority, metadata (JSONB)

2. sources - Platform URLs per subscription

   • subscription_id (FK), platform, url
   • last_check, last_success, error_count, metadata (JSONB)
   • Note: All sources use the global schedule_interval setting

3. downloads - Job records

   • source_id (FK), status (queued|pending|running|completed|failed|skipped)
   • file_count, total_size, error_type, error_message

4. credentials - Encrypted platform authentication

   • platform (unique), credential_type (cookies|token), data (encrypted binary)

5. content_items - Downloaded file records

   • source_id, download_id (FKs), external_id, file_path, file_size

6. settings - Key-value configuration store

Relationships:

Subscription (1) ──→ (many) Source
Source (1) ──→ (many) Download, ContentItem
Download (1) ──→ (many) ContentItem

API Routes

Endpoint	Methods	Purpose
/api/subscriptions	GET, POST	List/create subscriptions
/api/subscriptions/<id>	GET, PUT, DELETE	Single subscription CRUD
/api/subscriptions/<id>/check	POST	Trigger download check
/api/subscriptions/<id>/sources	GET, POST	Sources for subscription
/api/sources/<id>	GET, PUT, DELETE	Single source CRUD
/api/downloads	GET	Download history with pagination
/api/downloads/stats	GET	Download counts by status/platform
/api/downloads/recent-activity	GET	Recent failures and completions
/api/downloads/reset-orphaned	POST	Reset stuck running jobs
/api/downloads/requeue-stale	POST	Re-queue lost queued jobs
/api/credentials	GET, POST	List/upload credentials
/api/credentials/<platform>	DELETE	Remove credential
/api/settings	GET, PUT	App configuration
/api/platforms	GET	Platform definitions
/ws	WebSocket	Real-time events

Supported Platforms

Platform	Auth Type	Content Types
Patreon	Cookies	images, attachments, posts
SubscribeStar	Cookies	images, attachments
SubscribeStar Adult	Cookies	images, attachments
Hentai Foundry	Cookies	images
Discord	Token	messages, attachments

Configuration

Environment Variables (.env):

• DB_PASSWORD (required) - PostgreSQL password
• SECRET_KEY (required) - Encryption key (32+ chars)
• DOWNLOAD_PARALLEL_LIMIT (default: 3) - Concurrent downloads
• DOWNLOAD_RATE_LIMIT (default: 3.0) - Seconds between requests
• LOG_LEVEL (default: INFO)
• TZ (default: America/New_York)

Key Patterns

1. Async/Await - Quart + asyncpg for non-blocking I/O
2. Event-Driven - Redis pub/sub → WebSocket for real-time updates
3. Service Layer - GalleryDLService wraps gallery-dl subprocess
4. Encrypted Credentials - Fernet symmetric encryption with PBKDF2 key derivation
5. Global Schedule Interval - All sources checked at the same interval (configured in Settings)
6. Task Routing - Separate queues for scheduling vs download tasks
7. Database-Driven Settings - Worker reads rate_limit, retry_count, parallel_limit, schedule_interval from DB
8. Orphan Job Cleanup - Scheduler startup + periodic task resets stuck "running" jobs

Docker Services

app:        Backend API + Frontend (port 8080)
scheduler:  Celery beat + scheduling worker (lightweight tasks)
worker:     Celery download workers (heavy gallery-dl tasks)
db:         PostgreSQL 15-Alpine
redis:      Redis 7-Alpine (task queue + result backend)

Volumes:
  downloads: /data/downloads (downloaded files)
  config:    /data/config (gallery-dl archive)
  postgres_data: Database persistence

Task Routing Architecture

Tasks are routed to separate queues for clean separation of concerns:

┌─────────────────────────────────────────────────────────────────┐
│ SCHEDULER (1 container)                                         │
│   - Runs Celery Beat (triggers periodic tasks)                  │
│   - 1 worker consuming from "scheduling" queue                  │
│   - Handles: scheduled_check, cleanup, storage_stats, etc.      │
│   - Resets orphaned jobs on startup (only here, not workers)    │
└─────────────────────────────────────────────────────────────────┘
         │ Queues download tasks
         ▼
┌─────────────────────────────────────────────────────────────────┐
│ Redis Queues                                                    │
│   - "scheduling" queue → scheduler only                         │
│   - "downloads" queue  → download workers only                  │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│ WORKERS (scalable, 1+ containers)                               │
│   - Consume from "downloads" queue only                         │
│   - Handles: download_source, process_download                  │
│   - Heavy work (running gallery-dl)                             │
└─────────────────────────────────────────────────────────────────┘

Task Routing Config (in celery_app.py):

• scheduled_check, cleanup_old_downloads, update_storage_stats, reset_orphaned_jobs → "scheduling" queue
• download_source, process_download → "downloads" queue

Common Operations

Run migrations:

docker compose exec app alembic upgrade head

Build and start:

docker compose up -d --build

View logs:

docker compose logs -f app scheduler worker

Development mode:

docker compose -f docker-compose.dev.yml up

Extension Workflow

1. User installs Firefox extension
2. User configures API URL and key in extension options
3. Extension detects supported platforms when user visits them
4. User clicks "Export Cookies" in popup
5. Extension sends encrypted cookies to backend API
6. Backend stores credentials for gallery-dl authentication

Download Workflow

1. User creates subscription and adds source URLs
2. User triggers check or waits for scheduled check
3. Celery task retrieves credentials and builds gallery-dl config
4. gallery-dl downloads content to /data/downloads
5. Download records and content items stored in database
6. Real-time progress broadcast via WebSocket

Source Scheduling Rules

A source gets queued for download when ALL of these conditions are met:

1. source.enabled = True
2. subscription.enabled = True
3. (now - source.last_check) >= schedule_interval OR source.last_check is NULL (never checked)
4. No existing QUEUED or RUNNING download for this source

Note: last_check is updated when a download completes (success or failure), not when queued. This means the interval is measured from completion to completion.

Scheduled Tasks (Celery Beat)

Task	Schedule	Purpose
scheduled_check	Every 15 min	Check if sources are due (based on global schedule_interval)
cleanup_old_downloads	Daily 3 AM	Remove old download records (30d completed, 7d failed)
update_storage_stats	Every 30 min	Calculate downloads folder size
reset_orphaned_jobs	Every 15 min	Reset jobs stuck in "running" > 90 min
requeue_stale_jobs	Every 15 min	Re-queue jobs stuck in "queued" > 30 min (lost Celery tasks)

Scheduler Startup Behavior: When the scheduler container starts:

1. Resets ALL "running" jobs to "queued" (handles crash recovery)
2. Re-queues ALL "queued" jobs (recovers from Redis DB mismatch or lost tasks)
3. Triggers an immediate scheduled_check (don't wait 15 min for first check)

Download workers do NOT perform these startup tasks to avoid duplicates.

Settings (Database-Driven)

Key	Default	Purpose
download.schedule_interval	28800	Seconds between source checks (8 hours, applies to all sources)
download.rate_limit	3.0	Seconds between gallery-dl requests
download.parallel_limit	3	Concurrent Celery workers (requires restart)
download.retry_count	3	Max retries for failed downloads
download.crawl_depth	0	gallery-dl post crawl depth