Bryan Van Deusen
251 lines
8.6 KiB
Markdown
251 lines
8.6 KiB
Markdown
# GallerySubscriber
|
|
|
|
A Docker-based web application for managing gallery-dl downloads with a modern web interface and browser extension.
|
|
|
|
## How It Works
|
|
|
|
GallerySubscriber automates downloading content from subscription platforms like Patreon, SubscribeStar, Discord, and more. Here's the typical workflow:
|
|
|
|
1. **Install the Firefox extension** to export your login cookies/tokens from supported platforms
|
|
2. **Add subscriptions** in the web UI (e.g., "ArtistName") and attach sources (platform URLs)
|
|
3. **Export credentials** from the extension to authenticate downloads
|
|
4. **Trigger a check** or wait for automatic scheduled downloads
|
|
5. **Content is downloaded** to organized folders: `{subscription}/{platform}/{post}/`
|
|
|
|
The app handles rate limiting, retries, and tracks what's already been downloaded so you never get duplicates.
|
|
|
|
## Features
|
|
|
|
- **Web Dashboard** - Monitor downloads, view statistics, and manage subscriptions
|
|
- **Subscription Management** - Organize artists/creators with multiple platform sources
|
|
- **Firefox Extension** - One-click cookie/token export from Patreon, SubscribeStar, Discord
|
|
- **Per-Source Configuration** - Customize content types, rate limits, and patterns
|
|
- **Credential Management** - Securely encrypted storage for cookies/tokens
|
|
- **Scheduled Downloads** - Automatic periodic checks for new content
|
|
- **Real-time Updates** - WebSocket notifications for download progress
|
|
- **Supported Platforms**: Patreon, SubscribeStar, Hentai Foundry, Discord
|
|
|
|
## Quick Start
|
|
|
|
### 1. Configure Environment
|
|
|
|
```bash
|
|
# Download the docker-compose and example env
|
|
curl -O https://git.fabledsword.com/bvandeusen/gallerysubscriber/raw/branch/main/docker-compose.yml
|
|
curl -O https://git.fabledsword.com/bvandeusen/gallerysubscriber/raw/branch/main/.env.example
|
|
|
|
# Create your .env file
|
|
cp .env.example .env
|
|
|
|
# Edit .env - you MUST change DB_PASSWORD and SECRET_KEY
|
|
# Generate a secret key with: openssl rand -hex 32
|
|
```
|
|
|
|
### 2. Start the Services
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
### 3. Run Database Migrations
|
|
|
|
```bash
|
|
docker compose exec app alembic upgrade head
|
|
```
|
|
|
|
### 4. Access the Application
|
|
|
|
- **Web UI**: http://localhost:8080
|
|
- **API**: http://localhost:8080/api
|
|
- **Health Check**: http://localhost:8080/health
|
|
|
|
### 5. Install the Firefox Extension
|
|
|
|
1. Download the extension from the `extension/` folder
|
|
2. Go to `about:debugging#/runtime/this-firefox` in Firefox
|
|
3. Click "Load Temporary Add-on" and select `extension/manifest.json`
|
|
4. Click the extension icon and configure the API URL and key (found in Settings)
|
|
|
|
### 6. Export Your Credentials
|
|
|
|
1. Log into your subscription platforms (Patreon, etc.) in Firefox
|
|
2. Click the GallerySubscriber extension icon
|
|
3. Click "Export" for each platform to send cookies to the backend
|
|
4. For Discord: just browse Discord normally - the token is captured automatically
|
|
|
|
### 7. Add Subscriptions
|
|
|
|
1. Go to **Subscriptions** in the web UI
|
|
2. Click **Add Subscription** and enter the creator's name
|
|
3. Add sources (platform URLs) like `https://www.patreon.com/artistname`
|
|
4. Click the refresh icon to trigger a download check
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `DB_PASSWORD` | PostgreSQL password | (required) |
|
|
| `SECRET_KEY` | App secret key for encryption (32+ chars) | (required) |
|
|
| `TZ` | Timezone | `America/New_York` |
|
|
| `PORT` | External port | `8080` |
|
|
| `DOWNLOAD_PARALLEL_LIMIT` | Max concurrent downloads | `3` |
|
|
| `DOWNLOAD_RATE_LIMIT` | Delay between requests (seconds) | `3.0` |
|
|
| `DEFAULT_CHECK_INTERVAL` | Auto-check interval (seconds) | `28800` (8 hours) |
|
|
| `LOG_LEVEL` | Logging verbosity | `INFO` |
|
|
|
|
See `.env.example` for the complete list with descriptions.
|
|
|
|
### Settings Page
|
|
|
|
All configuration options are available in the web UI under **Settings**:
|
|
- Download settings (parallel limit, rate limit, retries)
|
|
- Extension API key (for connecting the Firefox extension)
|
|
- Raw gallery-dl.conf editor
|
|
- Platform defaults reference
|
|
|
|
## Firefox Extension
|
|
|
|
The extension exports authentication cookies/tokens from supported platforms to the backend.
|
|
|
|
### Supported Platforms
|
|
|
|
| Platform | Auth Type | How It Works |
|
|
|----------|-----------|--------------|
|
|
| Patreon | Cookies | Exports session cookies |
|
|
| SubscribeStar | Cookies | Exports session cookies |
|
|
| Hentai Foundry | Cookies | Exports session cookies |
|
|
| Discord | Token | Automatically captures Authorization header |
|
|
|
|
### Extension Setup
|
|
|
|
1. Load the extension in Firefox (`about:debugging` → Load Temporary Add-on)
|
|
2. Click the extension icon → Settings (gear icon)
|
|
3. Enter your backend API URL (e.g., `http://localhost:8080/api`)
|
|
4. Enter the Extension API Key from Settings page in the web UI
|
|
5. Click "Test Connection" to verify
|
|
|
|
### Exporting Credentials
|
|
|
|
- **Cookie-based platforms**: Log in, then click "Export" in the extension popup
|
|
- **Discord**: Just open Discord in Firefox - the token is captured automatically from API requests
|
|
|
|
## Download Organization
|
|
|
|
Files are downloaded to: `/data/downloads/{subscription}/{platform}/{post}/`
|
|
|
|
Example structure:
|
|
```
|
|
downloads/
|
|
├── ArtistName/
|
|
│ ├── patreon/
|
|
│ │ ├── 2024-01-15_12345_Post Title/
|
|
│ │ │ ├── 01_image.png
|
|
│ │ │ ├── 02_image.jpg
|
|
│ │ │ └── 01_image.json (metadata)
|
|
│ │ └── 2024-01-10_12340_Another Post/
|
|
│ └── discord/
|
|
│ └── art-channel/
|
|
│ └── 20240115_987654321_photo.png
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `GET /api/subscriptions` | List all subscriptions |
|
|
| `POST /api/subscriptions` | Create a subscription |
|
|
| `POST /api/subscriptions/:id/sources` | Add source to subscription |
|
|
| `POST /api/subscriptions/:id/check` | Trigger download check |
|
|
| `GET /api/sources` | List all sources |
|
|
| `PATCH /api/sources/:id` | Update a source |
|
|
| `GET /api/downloads` | List download history |
|
|
| `GET /api/downloads/stats` | Get download statistics |
|
|
| `POST /api/downloads/:id/retry` | Retry failed download |
|
|
| `GET /api/credentials` | List credential status |
|
|
| `POST /api/credentials` | Upload cookies/tokens |
|
|
| `GET /api/settings` | Get application settings |
|
|
| `PATCH /api/settings` | Update settings |
|
|
| `WS /ws/events` | Real-time event stream |
|
|
|
|
## Development
|
|
|
|
For development with hot-reload and debug logging:
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone https://git.fabledsword.com/bvandeusen/gallerysubscriber.git
|
|
cd gallerysubscriber
|
|
|
|
# Start development environment
|
|
docker compose -f docker-compose.dev.yml up --build
|
|
```
|
|
|
|
This provides:
|
|
- Live code reloading (backend changes apply automatically)
|
|
- Debug logging enabled
|
|
- PostgreSQL exposed on port 5432
|
|
- Redis exposed on port 6379
|
|
|
|
### Frontend Development (standalone)
|
|
|
|
```bash
|
|
cd frontend
|
|
npm install
|
|
npm run dev
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
gallerysubscriber/
|
|
├── backend/ # Quart API server
|
|
│ ├── app/
|
|
│ │ ├── api/ # REST + WebSocket endpoints
|
|
│ │ ├── models/ # SQLAlchemy database models
|
|
│ │ ├── services/ # Business logic (gallery-dl wrapper)
|
|
│ │ ├── tasks/ # Celery background tasks
|
|
│ │ └── utils/ # Encryption, cookies, etc.
|
|
│ └── alembic/ # Database migrations
|
|
├── frontend/ # Vue 3 + Vuetify SPA
|
|
│ └── src/
|
|
│ ├── views/ # Page components
|
|
│ ├── stores/ # Pinia state management
|
|
│ └── services/ # API client
|
|
├── extension/ # Firefox extension
|
|
│ ├── manifest.json # Extension manifest (v2)
|
|
│ ├── popup/ # Extension popup UI
|
|
│ ├── options/ # Extension settings page
|
|
│ ├── background/ # Background script
|
|
│ └── lib/ # Shared utilities
|
|
└── Dockerfile # Multi-stage build
|
|
```
|
|
|
|
## Technology Stack
|
|
|
|
- **Backend**: Quart (async Flask) + SQLAlchemy + PostgreSQL
|
|
- **Task Queue**: Celery + Redis
|
|
- **Frontend**: Vue 3 + Vuetify 3 + Pinia
|
|
- **Extension**: Firefox WebExtension (Manifest V2)
|
|
- **Downloader**: gallery-dl
|
|
|
|
## Troubleshooting
|
|
|
|
### Downloads failing with "Authentication failed"
|
|
- Re-export cookies from the Firefox extension
|
|
- Check that you're still logged into the platform
|
|
- For Discord: browse Discord to refresh the captured token
|
|
|
|
### "No cookies found" in extension
|
|
- Make sure you're logged into the platform in Firefox
|
|
- Check that cookies aren't being blocked by privacy extensions
|
|
|
|
### Downloads not starting
|
|
- Check the Celery worker logs: `docker compose logs celery`
|
|
- Verify credentials are uploaded: check the Credentials page
|
|
|
|
## License
|
|
|
|
MIT
|