# GallerySubscriber - Project Summary

**Updated:** 2026-01-28 (settings integration, orphan cleanup, error categorization fixes)

## Overview

GallerySubscriber is a Docker-based web application that automates downloading content from subscription platforms (Patreon, SubscribeStar, Discord, Hentai Foundry). It combines a Vue 3 web dashboard, Firefox extension for credential export, and a Python backend with Celery for background task scheduling.

**Core Purpose:** Automatically download and organize content from multiple subscription platforms using gallery-dl as the underlying download engine.

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    GallerySubscriber Stack                      │
├─────────────────────────────────────────────────────────────────┤
│ FRONTEND (Vue 3 + Vuetify)                                      │
│ - Served as static assets from backend (/static)               │
│ - SPA with client-side routing                                 │
├─────────────────────────────────────────────────────────────────┤
│ BACKEND API (Quart + SQLAlchemy)                               │
│ - Async Python web framework                                    │
│ - RESTful API + WebSocket for real-time updates                │
├─────────────────────────────────────────────────────────────────┤
│ CELERY WORKER (Background Tasks)                               │
│ - Scheduled downloads via Celery Beat                          │
│ - gallery-dl subprocess execution                              │
├─────────────────────────────────────────────────────────────────┤
│ DATABASE (PostgreSQL 15) + CACHE (Redis 7)                     │
└─────────────────────────────────────────────────────────────────┘
```

## Technology Stack

| Layer | Technologies |
|-------|-------------|
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
| Extension | Firefox WebExtension (Manifest V2) |
| Infrastructure | Docker, PostgreSQL 15, Redis 7 |
| Download Engine | gallery-dl >=1.31.0 |

## Project Structure

```
GallerySubscriber/
├── backend/app/
│   ├── main.py              # Quart app entry, serves frontend
│   ├── config.py            # Pydantic Settings from env
│   ├── events.py            # Redis pub/sub for WebSocket
│   ├── api/                 # API blueprints
│   │   ├── subscriptions.py # Artist/creator CRUD + check trigger
│   │   ├── sources.py       # Platform URL management
│   │   ├── downloads.py     # Download history
│   │   ├── credentials.py   # Encrypted credential storage
│   │   ├── settings.py      # App configuration
│   │   ├── platforms.py     # Platform schemas
│   │   └── websocket.py     # Real-time events
│   ├── models/              # SQLAlchemy models
│   │   ├── subscription.py  # Artist/creator groups
│   │   ├── source.py        # Platform URLs per subscription
│   │   ├── download.py      # Job records
│   │   ├── credential.py    # Encrypted auth data
│   │   ├── content.py       # Downloaded file records
│   │   └── setting.py       # Key-value config
│   ├── services/
│   │   ├── gallery_dl.py    # gallery-dl wrapper
│   │   └── credential_manager.py
│   ├── tasks/
│   │   ├── celery_app.py    # Celery configuration
│   │   ├── downloads.py     # Download tasks + scheduler
│   │   └── maintenance.py   # Cleanup jobs
│   └── utils/
│       ├── encryption.py    # Fernet crypto
│       └── cookies.py       # Netscape format handling
├── frontend/src/
│   ├── main.js              # Vue app entry
│   ├── views/               # Page components (Dashboard, Subscriptions, etc.)
│   ├── stores/              # Pinia state management
│   ├── services/api.js      # Axios HTTP client
│   └── composables/         # Reusable logic (useWebSocket)
├── extension/
│   ├── manifest.json        # WebExtension config
│   ├── background/          # Service worker for cookie/token capture
│   ├── popup/               # Extension popup UI
│   └── lib/                 # Platform definitions, API calls
├── alembic/                 # Database migrations
├── Dockerfile               # Multi-stage build (Node + Python)
├── docker-compose.yml       # Production stack
└── docker-compose.dev.yml   # Development with hot-reload
```

## Key Entry Points

| Component | Entry Point | Command |
|-----------|-------------|---------|
| Backend API | `backend/app/main.py` | `hypercorn app.main:app --bind 0.0.0.0:8080` |
| Celery Worker | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker -B` |
| Frontend | `frontend/src/main.js` | Built with Vite to `/static` |

## Database Schema

**Core Entities:**

1. **subscriptions** - Artist/creator groups
   - `name` (unique), `enabled`, `priority`, `metadata` (JSONB)

2. **sources** - Platform URLs per subscription
   - `subscription_id` (FK), `platform`, `url`
   - `last_check`, `last_success`, `error_count`, `metadata` (JSONB)
   - Note: All sources use the global `schedule_interval` setting

3. **downloads** - Job records
   - `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
   - `file_count`, `total_size`, `error_type`, `error_message`

4. **credentials** - Encrypted platform authentication
   - `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)

5. **content_items** - Downloaded file records
   - `source_id`, `download_id` (FKs), `external_id`, `file_path`, `file_size`

6. **settings** - Key-value configuration store

**Relationships:**
```
Subscription (1) ──→ (many) Source
Source (1) ──→ (many) Download, ContentItem
Download (1) ──→ (many) ContentItem
```

## API Routes

| Endpoint | Methods | Purpose |
|----------|---------|---------|
| `/api/subscriptions` | GET, POST | List/create subscriptions |
| `/api/subscriptions/<id>` | GET, PUT, DELETE | Single subscription CRUD |
| `/api/subscriptions/<id>/check` | POST | Trigger download check |
| `/api/subscriptions/<id>/sources` | GET, POST | Sources for subscription |
| `/api/sources/<id>` | GET, PUT, DELETE | Single source CRUD |
| `/api/downloads` | GET | Download history with pagination |
| `/api/downloads/stats` | GET | Download counts by status/platform |
| `/api/downloads/recent-activity` | GET | Recent failures and completions |
| `/api/downloads/reset-orphaned` | POST | Reset stuck running jobs |
| `/api/credentials` | GET, POST | List/upload credentials |
| `/api/credentials/<platform>` | DELETE | Remove credential |
| `/api/settings` | GET, PUT | App configuration |
| `/api/platforms` | GET | Platform definitions |
| `/ws` | WebSocket | Real-time events |

## Supported Platforms

| Platform | Auth Type | Content Types |
|----------|-----------|---------------|
| Patreon | Cookies | images, attachments, posts |
| SubscribeStar | Cookies | images, attachments |
| SubscribeStar Adult | Cookies | images, attachments |
| Hentai Foundry | Cookies | images |
| Discord | Token | messages, attachments |

## Configuration

**Environment Variables (.env):**
- `DB_PASSWORD` (required) - PostgreSQL password
- `SECRET_KEY` (required) - Encryption key (32+ chars)
- `DOWNLOAD_PARALLEL_LIMIT` (default: 3) - Concurrent downloads
- `DOWNLOAD_RATE_LIMIT` (default: 3.0) - Seconds between requests
- `LOG_LEVEL` (default: INFO)
- `TZ` (default: America/New_York)

## Key Patterns

1. **Async/Await** - Quart + asyncpg for non-blocking I/O
2. **Event-Driven** - Redis pub/sub → WebSocket for real-time updates
3. **Service Layer** - GalleryDLService wraps gallery-dl subprocess
4. **Encrypted Credentials** - Fernet symmetric encryption with PBKDF2 key derivation
5. **Global Schedule Interval** - All sources checked at the same interval (configured in Settings)
6. **Celery Beat** - Runs `scheduled_check` every 60s to trigger due downloads
7. **Database-Driven Settings** - Worker reads rate_limit, retry_count, parallel_limit from DB
8. **Orphan Job Cleanup** - Worker startup + periodic task resets stuck "running" jobs

## Docker Services

```yaml
app:      Backend API + Frontend (port 8080)
celery:   Background worker with scheduler
db:       PostgreSQL 15-Alpine
redis:    Redis 7-Alpine

Volumes:
  downloads: /data/downloads (downloaded files)
  config:    /data/config (gallery-dl archive)
  postgres_data: Database persistence
```

## Common Operations

**Run migrations:**
```bash
docker compose exec app alembic upgrade head
```

**Build and start:**
```bash
docker compose up -d --build
```

**View logs:**
```bash
docker compose logs -f app celery
```

**Development mode:**
```bash
docker compose -f docker-compose.dev.yml up
```

## Extension Workflow

1. User installs Firefox extension
2. User configures API URL and key in extension options
3. Extension detects supported platforms when user visits them
4. User clicks "Export Cookies" in popup
5. Extension sends encrypted cookies to backend API
6. Backend stores credentials for gallery-dl authentication

## Download Workflow

1. User creates subscription and adds source URLs
2. User triggers check or waits for scheduled check
3. Celery task retrieves credentials and builds gallery-dl config
4. gallery-dl downloads content to `/data/downloads`
5. Download records and content items stored in database
6. Real-time progress broadcast via WebSocket

## Scheduled Tasks (Celery Beat)

| Task | Schedule | Purpose |
|------|----------|---------|
| `scheduled_check` | Every 60s | Check if sources are due (based on global schedule_interval) |
| `cleanup_old_downloads` | Daily 3 AM | Remove old download records (30d completed, 7d failed) |
| `update_storage_stats` | Every 30 min | Calculate downloads folder size |
| `reset_orphaned_jobs` | Every 15 min | Reset jobs stuck in "running" > 90 min |

**Worker Startup Behavior:** When Celery worker starts, it resets ALL "running" jobs to "queued" (handles crash recovery).

## Settings (Database-Driven)

| Key | Default | Purpose |
|-----|---------|---------|
| `download.schedule_interval` | 3600 | Seconds between source checks (applies to all sources) |
| `download.rate_limit` | 3.0 | Seconds between gallery-dl requests |
| `download.parallel_limit` | 3 | Concurrent Celery workers (requires restart) |
| `download.retry_count` | 3 | Max retries for failed downloads |
| `download.crawl_depth` | 0 | gallery-dl post crawl depth |