changes to task scheduling and readability change on downloads view
This commit is contained in:
Bryan Van Deusen
+66
-13
@@ -1,6 +1,6 @@
|
||||
# GallerySubscriber - Project Summary
|
||||
|
||||
**Updated:** 2026-01-28 (settings integration, orphan cleanup, error categorization fixes)
|
||||
**Updated:** 2026-01-29 (task routing, scheduler/worker separation, 8-hour default interval)
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -95,7 +95,8 @@ GallerySubscriber/
|
||||
| Component | Entry Point | Command |
|
||||
|-----------|-------------|---------|
|
||||
| Backend API | `backend/app/main.py` | `hypercorn app.main:app --bind 0.0.0.0:8080` |
|
||||
| Celery Worker | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker -B` |
|
||||
| Scheduler | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker --queues=scheduling --beat --concurrency=1` |
|
||||
| Download Workers | `backend/app/tasks/celery_app.py` | `celery -A app.tasks.celery_app worker --queues=downloads` |
|
||||
| Frontend | `frontend/src/main.js` | Built with Vite to `/static` |
|
||||
|
||||
## Database Schema
|
||||
@@ -142,6 +143,7 @@ Download (1) ──→ (many) ContentItem
|
||||
| `/api/downloads/stats` | GET | Download counts by status/platform |
|
||||
| `/api/downloads/recent-activity` | GET | Recent failures and completions |
|
||||
| `/api/downloads/reset-orphaned` | POST | Reset stuck running jobs |
|
||||
| `/api/downloads/requeue-stale` | POST | Re-queue lost queued jobs |
|
||||
| `/api/credentials` | GET, POST | List/upload credentials |
|
||||
| `/api/credentials/<platform>` | DELETE | Remove credential |
|
||||
| `/api/settings` | GET, PUT | App configuration |
|
||||
@@ -175,17 +177,18 @@ Download (1) ──→ (many) ContentItem
|
||||
3. **Service Layer** - GalleryDLService wraps gallery-dl subprocess
|
||||
4. **Encrypted Credentials** - Fernet symmetric encryption with PBKDF2 key derivation
|
||||
5. **Global Schedule Interval** - All sources checked at the same interval (configured in Settings)
|
||||
6. **Celery Beat** - Runs `scheduled_check` every 60s to trigger due downloads
|
||||
7. **Database-Driven Settings** - Worker reads rate_limit, retry_count, parallel_limit from DB
|
||||
8. **Orphan Job Cleanup** - Worker startup + periodic task resets stuck "running" jobs
|
||||
6. **Task Routing** - Separate queues for scheduling vs download tasks
|
||||
7. **Database-Driven Settings** - Worker reads rate_limit, retry_count, parallel_limit, schedule_interval from DB
|
||||
8. **Orphan Job Cleanup** - Scheduler startup + periodic task resets stuck "running" jobs
|
||||
|
||||
## Docker Services
|
||||
|
||||
```yaml
|
||||
app: Backend API + Frontend (port 8080)
|
||||
celery: Background worker with scheduler
|
||||
db: PostgreSQL 15-Alpine
|
||||
redis: Redis 7-Alpine
|
||||
app: Backend API + Frontend (port 8080)
|
||||
scheduler: Celery beat + scheduling worker (lightweight tasks)
|
||||
worker: Celery download workers (heavy gallery-dl tasks)
|
||||
db: PostgreSQL 15-Alpine
|
||||
redis: Redis 7-Alpine (task queue + result backend)
|
||||
|
||||
Volumes:
|
||||
downloads: /data/downloads (downloaded files)
|
||||
@@ -193,6 +196,39 @@ Volumes:
|
||||
postgres_data: Database persistence
|
||||
```
|
||||
|
||||
## Task Routing Architecture
|
||||
|
||||
Tasks are routed to separate queues for clean separation of concerns:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ SCHEDULER (1 container) │
|
||||
│ - Runs Celery Beat (triggers periodic tasks) │
|
||||
│ - 1 worker consuming from "scheduling" queue │
|
||||
│ - Handles: scheduled_check, cleanup, storage_stats, etc. │
|
||||
│ - Resets orphaned jobs on startup (only here, not workers) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ Queues download tasks
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Redis Queues │
|
||||
│ - "scheduling" queue → scheduler only │
|
||||
│ - "downloads" queue → download workers only │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ WORKERS (scalable, 1+ containers) │
|
||||
│ - Consume from "downloads" queue only │
|
||||
│ - Handles: download_source, process_download │
|
||||
│ - Heavy work (running gallery-dl) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Task Routing Config** (in `celery_app.py`):
|
||||
- `scheduled_check`, `cleanup_old_downloads`, `update_storage_stats`, `reset_orphaned_jobs` → "scheduling" queue
|
||||
- `download_source`, `process_download` → "downloads" queue
|
||||
|
||||
## Common Operations
|
||||
|
||||
**Run migrations:**
|
||||
@@ -207,7 +243,7 @@ docker compose up -d --build
|
||||
|
||||
**View logs:**
|
||||
```bash
|
||||
docker compose logs -f app celery
|
||||
docker compose logs -f app scheduler worker
|
||||
```
|
||||
|
||||
**Development mode:**
|
||||
@@ -233,22 +269,39 @@ docker compose -f docker-compose.dev.yml up
|
||||
5. Download records and content items stored in database
|
||||
6. Real-time progress broadcast via WebSocket
|
||||
|
||||
## Source Scheduling Rules
|
||||
|
||||
A source gets queued for download when ALL of these conditions are met:
|
||||
|
||||
1. `source.enabled = True`
|
||||
2. `subscription.enabled = True`
|
||||
3. `(now - source.last_check) >= schedule_interval` OR `source.last_check is NULL` (never checked)
|
||||
4. No existing QUEUED or RUNNING download for this source
|
||||
|
||||
**Note:** `last_check` is updated when a download **completes** (success or failure), not when queued. This means the interval is measured from completion to completion.
|
||||
|
||||
## Scheduled Tasks (Celery Beat)
|
||||
|
||||
| Task | Schedule | Purpose |
|
||||
|------|----------|---------|
|
||||
| `scheduled_check` | Every 60s | Check if sources are due (based on global schedule_interval) |
|
||||
| `scheduled_check` | Every 15 min | Check if sources are due (based on global schedule_interval) |
|
||||
| `cleanup_old_downloads` | Daily 3 AM | Remove old download records (30d completed, 7d failed) |
|
||||
| `update_storage_stats` | Every 30 min | Calculate downloads folder size |
|
||||
| `reset_orphaned_jobs` | Every 15 min | Reset jobs stuck in "running" > 90 min |
|
||||
| `requeue_stale_jobs` | Every 15 min | Re-queue jobs stuck in "queued" > 30 min (lost Celery tasks) |
|
||||
|
||||
**Worker Startup Behavior:** When Celery worker starts, it resets ALL "running" jobs to "queued" (handles crash recovery).
|
||||
**Scheduler Startup Behavior:** When the scheduler container starts:
|
||||
1. Resets ALL "running" jobs to "queued" (handles crash recovery)
|
||||
2. Re-queues ALL "queued" jobs (recovers from Redis DB mismatch or lost tasks)
|
||||
3. Triggers an immediate `scheduled_check` (don't wait 15 min for first check)
|
||||
|
||||
Download workers do NOT perform these startup tasks to avoid duplicates.
|
||||
|
||||
## Settings (Database-Driven)
|
||||
|
||||
| Key | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `download.schedule_interval` | 3600 | Seconds between source checks (applies to all sources) |
|
||||
| `download.schedule_interval` | 28800 | Seconds between source checks (8 hours, applies to all sources) |
|
||||
| `download.rate_limit` | 3.0 | Seconds between gallery-dl requests |
|
||||
| `download.parallel_limit` | 3 | Concurrent Celery workers (requires restart) |
|
||||
| `download.retry_count` | 3 | Max retries for failed downloads |
|
||||
|
||||
Reference in New Issue
Block a user