GallerySubscriber/summary.md

# GallerySubscriber - Project Summary

**Updated:** 2026-02-02 (Added Pixiv and DeviantArt platform support)

## Overview

GallerySubscriber is a Docker-based web application that automates downloading content from subscription platforms (Patreon, SubscribeStar, Discord, Hentai Foundry, Pixiv, DeviantArt). It combines a Vue 3 web dashboard, Firefox extension for credential export, and a Python backend with Celery for background task scheduling.

**Core Purpose:** Automatically download and organize content from multiple subscription platforms using gallery-dl as the underlying download engine.

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    GallerySubscriber Stack                      │
├─────────────────────────────────────────────────────────────────┤
│ FRONTEND (Vue 3 + Vuetify)                                      │
│ - Served as static assets from backend (/static)               │
│ - SPA with client-side routing                                 │
├─────────────────────────────────────────────────────────────────┤
│ BACKEND API (Quart + SQLAlchemy)                               │
│ - Async Python web framework                                    │
│ - RESTful API + WebSocket for real-time updates                │
├─────────────────────────────────────────────────────────────────┤
│ CELERY WORKER (Background Tasks)                               │
│ - Scheduled downloads via Celery Beat                          │
│ - gallery-dl subprocess execution                              │
├─────────────────────────────────────────────────────────────────┤
│ DATABASE (PostgreSQL 15) + CACHE (Redis 7)                     │
└─────────────────────────────────────────────────────────────────┘
```

## Technology Stack

| Layer | Technologies |
|-------|-------------|
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
| Extension | Firefox WebExtension (Manifest V2) |
| Infrastructure | Docker, PostgreSQL 15, Redis 7 |
| Download Engine | gallery-dl >=1.31.0 |

## Project Structure

```
GallerySubscriber/
├── backend/app/
│   ├── main.py              # Quart app entry, serves frontend
│   ├── config.py            # Pydantic Settings from env
│   ├── events.py            # Redis pub/sub for WebSocket
│   ├── api/                 # API blueprints
│   │   ├── subscriptions.py # Artist/creator CRUD + check trigger
│   │   ├── sources.py       # Platform URL management
│   │   ├── downloads.py     # Download history
│   │   ├── credentials.py   # Encrypted credential storage
│   │   ├── settings.py      # App configuration
│   │   ├── platforms.py     # Platform schemas
│   │   └── websocket.py     # Real-time events
│   ├── models/              # SQLAlchemy models
│   │   ├── subscription.py  # Artist/creator groups
│   │   ├── source.py        # Platform URLs per subscription
│   │   ├── download.py      # Job records
│   │   ├── credential.py    # Encrypted auth data
│   │   ├── content.py       # Downloaded file records
│   │   └── setting.py       # Key-value config
│   ├── services/
│   │   ├── gallery_dl.py    # gallery-dl wrapper
│   │   └── credential_manager.py
│   ├── tasks/
│   │   ├── celery_app.py    # Celery configuration
│   │   ├── downloads.py     # Download tasks + scheduler
│   │   └── maintenance.py   # Cleanup jobs
│   └── utils/
│       ├── encryption.py    # Fernet crypto
│       └── cookies.py       # Netscape format handling
├── frontend/src/
│   ├── main.js              # Vue app entry
│   ├── views/               # Page components (Dashboard, Subscriptions, etc.)
│   ├── stores/              # Pinia state management
│   ├── services/api.js      # Axios HTTP client
│   └── composables/         # Reusable logic (useWebSocket)
├── extension/
│   ├── manifest.json        # WebExtension config
│   ├── background/          # Service worker for cookie/token capture
│   ├── popup/               # Extension popup UI
│   └── lib/                 # Platform definitions, API calls
├── alembic/                 # Database migrations
├── Dockerfile               # Multi-stage build (Node + Python)
├── docker-compose.yml       # Production stack
└── docker-compose.dev.yml   # Development with hot-reload
```

## Key Entry Points

| Component | Entry Point | Command |
|-----------|-------------|---------|
| Backend API | `backend/app/main.py` | `hypercorn app.main:app --bind 0.0.0.0:8080` |
| Scheduler | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker --queues=scheduling --beat --concurrency=1` |
| Download Workers | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker --queues=downloads` |
| Frontend | `frontend/src/main.js` | Built with Vite to `/static` |

## Database Schema

**Core Entities:**

1. **subscriptions** - Artist/creator groups
   - `name` (unique), `enabled`, `priority`, `metadata` (JSONB)

2. **sources** - Platform URLs per subscription
   - `subscription_id` (FK), `platform`, `url`
   - `last_check`, `last_success`, `error_count`, `metadata` (JSONB)
   - Note: All sources use the global `schedule_interval` setting

3. **downloads** - Job records
   - `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
   - `file_count`, `total_size`, `error_type`, `error_message`

4. **credentials** - Encrypted platform authentication
   - `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)

5. **content_items** - Downloaded file records
   - `source_id`, `download_id` (FKs), `external_id`, `file_path`, `file_size`

6. **settings** - Key-value configuration store

**Relationships:**
```
Subscription (1) ──→ (many) Source
Source (1) ──→ (many) Download, ContentItem
Download (1) ──→ (many) ContentItem
```

## API Routes

| Endpoint | Methods | Purpose |
|----------|---------|---------|
| `/api/subscriptions` | GET, POST | List/create subscriptions |
| `/api/subscriptions/<id>` | GET, PUT, DELETE | Single subscription CRUD |
| `/api/subscriptions/<id>/check` | POST | Trigger download check |
| `/api/subscriptions/<id>/sources` | GET, POST | Sources for subscription |
| `/api/sources/<id>` | GET, PUT, DELETE | Single source CRUD |
| `/api/downloads` | GET | Download history with pagination |
| `/api/downloads/stats` | GET | Download counts by status/platform |
| `/api/downloads/recent-activity` | GET | Recent failures and completions |
| `/api/downloads/reset-orphaned` | POST | Reset stuck running jobs |
| `/api/downloads/requeue-stale` | POST | Re-queue lost queued jobs |
| `/api/credentials` | GET, POST | List/upload credentials |
| `/api/credentials/<platform>` | DELETE | Remove credential |
| `/api/settings` | GET, PUT | App configuration |
| `/api/platforms` | GET | Platform definitions |
| `/ws` | WebSocket | Real-time events |

## Supported Platforms

| Platform | Auth Type | Content Types | Pagination |
|----------|-----------|---------------|------------|
| Patreon | Cookies | images, attachments, posts, content | Cursor-based (complete history) |
| SubscribeStar | Cookies | images, attachments | Infinite scroll (complete) |
| SubscribeStar Adult | Cookies | images, attachments | Infinite scroll (complete) |
| Hentai Foundry | Cookies | pictures, scraps, stories | All pages (complete) |
| Discord | Token | messages, attachments, embeds | All threads included |
| Pixiv | Cookies | illusts, manga, bookmarks | All works (complete) |
| DeviantArt | Cookies (optional) | gallery, scraps, favorites | All deviations (complete) |

**Note:** All platforms fetch complete history by default - no depth limits. Gallery-dl paginates through ALL available content.

## Configuration

**Environment Variables (.env):**
- `DB_PASSWORD` (required) - PostgreSQL password
- `SECRET_KEY` (required) - Encryption key (32+ chars)
- `REDIS_DB` (default: 0) - Redis database number (must be same for all services)
- `DOWNLOAD_PARALLEL_LIMIT` (default: 3) - Concurrent downloads
- `DOWNLOAD_RATE_LIMIT` (default: 3.0) - Seconds between requests
- `LOG_LEVEL` (default: INFO)
- `TZ` (default: America/New_York)

**IMPORTANT:** All services (app, scheduler, worker) must use the same Redis database. A mismatch causes Celery tasks to be lost (jobs stuck in "queued" status). The system includes automatic detection and recovery for this scenario.

## Key Patterns

1. **Async/Await** - Quart + asyncpg for non-blocking I/O
2. **Event-Driven** - Redis pub/sub → WebSocket for real-time updates
3. **Service Layer** - GalleryDLService wraps gallery-dl subprocess
4. **Encrypted Credentials** - Fernet symmetric encryption with PBKDF2 key derivation
5. **Global Schedule Interval** - All sources checked at the same interval (configured in Settings)
6. **Task Routing** - Separate queues for scheduling vs download tasks
7. **Database-Driven Settings** - Worker reads rate_limit, retry_count, parallel_limit, schedule_interval from DB
8. **Orphan Job Cleanup** - Scheduler startup + periodic task resets stuck "running" jobs
9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing

## Docker Services

```yaml
app:        Backend API + Frontend (port 8080)
scheduler:  Celery beat + scheduling worker (lightweight tasks)
worker:     Celery download workers (heavy gallery-dl tasks)
db:         PostgreSQL 15-Alpine
redis:      Redis 7-Alpine (task queue + result backend)

Volumes:
  downloads: /data/downloads (downloaded files)
  config:    /data/config (gallery-dl archive)
  postgres_data: Database persistence
```

## Task Routing Architecture

Tasks are routed to separate queues for clean separation of concerns:

```
┌─────────────────────────────────────────────────────────────────┐
│ SCHEDULER (1 container)                                         │
│   - Runs Celery Beat (triggers periodic tasks)                  │
│   - 1 worker consuming from "scheduling" queue                  │
│   - Handles: scheduled_check, cleanup, storage_stats, etc.      │
│   - Resets orphaned jobs on startup (only here, not workers)    │
└─────────────────────────────────────────────────────────────────┘
         │ Queues download tasks
         ▼
┌─────────────────────────────────────────────────────────────────┐
│ Redis Queues                                                    │
│   - "scheduling" queue → scheduler only                         │
│   - "downloads" queue  → download workers only                  │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│ WORKERS (scalable, 1+ containers)                               │
│   - Consume from "downloads" queue only                         │
│   - Handles: download_source, process_download                  │
│   - Heavy work (running gallery-dl)                             │
└─────────────────────────────────────────────────────────────────┘
```

**Task Routing Config** (in `celery_app.py`):
- `scheduled_check`, `cleanup_old_downloads`, `update_storage_stats`, `reset_orphaned_jobs`, `requeue_stale_jobs` → "scheduling" queue
- `download_source`, `process_download` → "downloads" queue

## Common Operations

**Run migrations:**
```bash
docker compose exec app alembic upgrade head
```

**Build and start:**
```bash
docker compose up -d --build
```

**View logs:**
```bash
docker compose logs -f app scheduler worker
```

**Development mode:**
```bash
docker compose -f docker-compose.dev.yml up
```

## Extension Workflow

1. User installs Firefox extension
2. User configures API URL and key in extension options
3. Extension detects supported platforms when user visits them
4. User clicks "Export Cookies" in popup
5. Extension sends encrypted cookies to backend API
6. Backend stores credentials for gallery-dl authentication

## Download Workflow

1. User creates subscription and adds source URLs
2. User triggers check or waits for scheduled check
3. Celery task retrieves credentials and builds gallery-dl config
4. gallery-dl downloads content to `/data/downloads`
5. Download records and content items stored in database
6. Real-time progress broadcast via WebSocket

## Source Scheduling Rules

A source gets queued for download when ALL of these conditions are met:

1. `source.enabled = True`
2. `subscription.enabled = True`
3. `(now - source.last_check) >= schedule_interval` OR `source.last_check is NULL` (never checked)
4. No existing QUEUED or RUNNING download for this source

**Note:** `last_check` is updated when a download **completes** (success or failure), not when queued. This means the interval is measured from completion to completion.

## Scheduled Tasks (Celery Beat)

| Task | Schedule | Purpose |
|------|----------|---------|
| `scheduled_check` | Every 15 min | Check if sources are due (based on global schedule_interval) |
| `cleanup_old_downloads` | Daily 3 AM | Remove old download records (30d completed, 7d failed) |
| `update_storage_stats` | Every 30 min | Calculate downloads folder size |
| `reset_orphaned_jobs` | Every 15 min | Reset jobs stuck in "running" > 90 min |
| `requeue_stale_jobs` | Every 15 min | Re-queue jobs stuck in "queued" > 30 min (lost Celery tasks) |

**Scheduler Startup Behavior:** When the scheduler container starts:
1. Resets ALL "running" jobs to "queued" (handles crash recovery)
2. Re-queues ALL "queued" jobs (recovers from Redis DB mismatch or lost tasks)
3. Triggers an immediate `scheduled_check` (don't wait 15 min for first check)

Download workers do NOT perform these startup tasks to avoid duplicates.

## Settings (Database-Driven)

| Key | Default | Purpose |
|-----|---------|---------|
| `download.schedule_interval` | 28800 | Seconds between source checks (8 hours, applies to all sources) |
| `download.rate_limit` | 3.0 | Seconds between file downloads |
| `download.parallel_limit` | 3 | Concurrent Celery workers (requires restart) |
| `download.retry_count` | 3 | Max retries for failed downloads |

**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.

## Downloads Page Features

The Downloads view (`/downloads`) provides comprehensive download monitoring:

**Header Section:**
- Live stats chips showing counts for Queued, Running, Completed, Failed
- Refresh button to manually update data
- Maintenance dropdown with recovery actions

**Filters:**
- Status filter (queued, running, completed, failed, skipped)
- Source filter showing "Subscription:Platform" format for easy identification
- Date range filters (from/to)

**Table Columns:**
- Status with color-coded chips and animated spinner for running jobs
- Source showing "Subscription:Platform" label
- URL (truncated with full URL on hover)
- Error type (formatted for readability)
- File count
- Date created
- Duration (calculated from started_at to completed_at)
- Actions (retry for failed, details for all)

**Details Dialog:**
- Full download metadata
- Output log (stdout from gallery-dl)
- Error log (stderr)
- Retry button for failed downloads

**Maintenance Actions:**
- **Reset Orphaned Running Jobs**: Resets jobs stuck in "running" status back to "queued"
- **Re-queue Stale Queued Jobs**: Re-sends Celery tasks for jobs stuck in "queued" status

**Auto-Refresh:** Automatically refreshes every 30 seconds when there are running or queued jobs.

## Duplicate Download Prevention

Multiple layers prevent duplicate downloads for the same source:

1. **Re-queue Functions** (`maintenance.py`):
   - Track `queued_source_ids` set during re-queuing
   - Skip sources that already have a task re-queued
   - Mark duplicate jobs as SKIPPED with descriptive error message

2. **Download Task** (`downloads.py`):
   - Check download status is still QUEUED before claiming
   - Check if source already has a RUNNING download
   - Mark duplicates as SKIPPED to prevent reprocessing

3. **Scheduler Logic** (`downloads.py`):
   - `scheduled_check` skips sources with existing QUEUED or RUNNING downloads

## Error Classification System

The `_categorize_error()` method in `gallery_dl.py` classifies download failures into actionable error types:

| Error Type | Meaning | User Action |
|------------|---------|-------------|
| `no_new_content` | All content already downloaded (success) | None needed |
| `auth_error` | Cookies expired or invalid | Re-export cookies from extension |
| `rate_limited` | Too many requests (429) | Increase rate_limit setting |
| `not_found` | Creator/content deleted or moved | Check URL, may need to remove |
| `access_denied` | Insufficient subscription tier | Upgrade pledge or remove source |
| `network_error` | Connection issues | Check network, will auto-retry |
| `timeout` | Download took too long | Will auto-retry |
| `unsupported_url` | URL not recognized by gallery-dl | Check URL format |
| `unknown_error` | Unclassified failure | Check logs for details |

**Classification Logic:**

1. **Skip Detection First** - Checks for `#` prefixed lines in stdout (gallery-dl's skip message format) and text patterns like "skipping", "already exists". If skips are detected with no actual errors, returns `no_new_content`.

2. **Actual Error Detection** - Looks for `][error]` log level, exceptions, and tracebacks before proceeding to pattern matching.

3. **Specific Patterns** - Uses targeted patterns (e.g., `"timed out"` not just `"timeout"`) to avoid false positives from config values in debug output.

## Backend Scripts

| Script | Purpose |
|--------|---------|
| `backend/scripts/export_failed_logs.py` | Export failed download logs to JSON for analysis |

## Analysis Directory

The `analysis/` directory (gitignored) contains debugging and analysis artifacts:

- `failed_logs_export_*.json` - Exported download logs for analysis
- `analysis_results.json` - Parsed analysis results
- `ERROR_RECOGNITION_ANALYSIS.md` - Detailed findings report
- Analysis scripts for pattern debugging

These files are excluded from version control but preserved locally for ongoing analysis.