more UI polish and schedule safe guards and optimization

summary.md

@@ -1,6 +1,6 @@
 # GallerySubscriber - Project Summary
 
-**Updated:** 2026-01-29 (task routing, scheduler/worker separation, 8-hour default interval)
+**Updated:** 2026-01-29 (Redis DB config, mismatch detection, stale job recovery, Downloads UI improvements)
 
 ## Overview
 
@@ -152,24 +152,29 @@ Download (1) ──→ (many) ContentItem
 
 ## Supported Platforms
 
-| Platform | Auth Type | Content Types |
-|----------|-----------|---------------|
-| Patreon | Cookies | images, attachments, posts |
-| SubscribeStar | Cookies | images, attachments |
-| SubscribeStar Adult | Cookies | images, attachments |
-| Hentai Foundry | Cookies | images |
-| Discord | Token | messages, attachments |
+| Platform | Auth Type | Content Types | Pagination |
+|----------|-----------|---------------|------------|
+| Patreon | Cookies | images, attachments, posts, content | Cursor-based (complete history) |
+| SubscribeStar | Cookies | images, attachments | Infinite scroll (complete) |
+| SubscribeStar Adult | Cookies | images, attachments | Infinite scroll (complete) |
+| Hentai Foundry | Cookies | pictures, scraps, stories | All pages (complete) |
+| Discord | Token | messages, attachments, embeds | All threads included |
+
+**Note:** All platforms fetch complete history by default - no depth limits. Gallery-dl paginates through ALL available content.
 
 ## Configuration
 
 **Environment Variables (.env):**
 - `DB_PASSWORD` (required) - PostgreSQL password
 - `SECRET_KEY` (required) - Encryption key (32+ chars)
+- `REDIS_DB` (default: 0) - Redis database number (must be same for all services)
 - `DOWNLOAD_PARALLEL_LIMIT` (default: 3) - Concurrent downloads
 - `DOWNLOAD_RATE_LIMIT` (default: 3.0) - Seconds between requests
 - `LOG_LEVEL` (default: INFO)
 - `TZ` (default: America/New_York)
 
+**IMPORTANT:** All services (app, scheduler, worker) must use the same Redis database. A mismatch causes Celery tasks to be lost (jobs stuck in "queued" status). The system includes automatic detection and recovery for this scenario.
+
 ## Key Patterns
 
 1. **Async/Await** - Quart + asyncpg for non-blocking I/O
@@ -180,6 +185,9 @@ Download (1) ──→ (many) ContentItem
 6. **Task Routing** - Separate queues for scheduling vs download tasks
 7. **Database-Driven Settings** - Worker reads rate_limit, retry_count, parallel_limit, schedule_interval from DB
 8. **Orphan Job Cleanup** - Scheduler startup + periodic task resets stuck "running" jobs
+9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
+10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
+11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
 
 ## Docker Services
 
@@ -226,7 +234,7 @@ Tasks are routed to separate queues for clean separation of concerns:
 ```
 
 **Task Routing Config** (in `celery_app.py`):
-- `scheduled_check`, `cleanup_old_downloads`, `update_storage_stats`, `reset_orphaned_jobs` → "scheduling" queue
+- `scheduled_check`, `cleanup_old_downloads`, `update_storage_stats`, `reset_orphaned_jobs`, `requeue_stale_jobs` → "scheduling" queue
 - `download_source`, `process_download` → "downloads" queue
 
 ## Common Operations
@@ -302,7 +310,61 @@ Download workers do NOT perform these startup tasks to avoid duplicates.
 | Key | Default | Purpose |
 |-----|---------|---------|
 | `download.schedule_interval` | 28800 | Seconds between source checks (8 hours, applies to all sources) |
-| `download.rate_limit` | 3.0 | Seconds between gallery-dl requests |
+| `download.rate_limit` | 3.0 | Seconds between file downloads |
 | `download.parallel_limit` | 3 | Concurrent Celery workers (requires restart) |
 | `download.retry_count` | 3 | Max retries for failed downloads |
-| `download.crawl_depth` | 0 | gallery-dl post crawl depth |
+
+**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.
+
+## Downloads Page Features
+
+The Downloads view (`/downloads`) provides comprehensive download monitoring:
+
+**Header Section:**
+- Live stats chips showing counts for Queued, Running, Completed, Failed
+- Refresh button to manually update data
+- Maintenance dropdown with recovery actions
+
+**Filters:**
+- Status filter (queued, running, completed, failed, skipped)
+- Source filter showing "Subscription:Platform" format for easy identification
+- Date range filters (from/to)
+
+**Table Columns:**
+- Status with color-coded chips and animated spinner for running jobs
+- Source showing "Subscription:Platform" label
+- URL (truncated with full URL on hover)
+- Error type (formatted for readability)
+- File count
+- Date created
+- Duration (calculated from started_at to completed_at)
+- Actions (retry for failed, details for all)
+
+**Details Dialog:**
+- Full download metadata
+- Output log (stdout from gallery-dl)
+- Error log (stderr)
+- Retry button for failed downloads
+
+**Maintenance Actions:**
+- **Reset Orphaned Running Jobs**: Resets jobs stuck in "running" status back to "queued"
+- **Re-queue Stale Queued Jobs**: Re-sends Celery tasks for jobs stuck in "queued" status
+
+**Auto-Refresh:** Automatically refreshes every 30 seconds when there are running or queued jobs.
+
+## Duplicate Download Prevention
+
+Multiple layers prevent duplicate downloads for the same source:
+
+1. **Re-queue Functions** (`maintenance.py`):
+   - Track `queued_source_ids` set during re-queuing
+   - Skip sources that already have a task re-queued
+   - Mark duplicate jobs as SKIPPED with descriptive error message
+
+2. **Download Task** (`downloads.py`):
+   - Check download status is still QUEUED before claiming
+   - Check if source already has a RUNNING download
+   - Mark duplicates as SKIPPED to prevent reprocessing
+
+3. **Scheduler Logic** (`downloads.py`):
+   - `scheduled_check` skips sources with existing QUEUED or RUNNING downloads