chore: sane ignore files, update summary, bump base images

- Add .claude/ to .gitignore and untrack settings.local.json - Create .dockerignore to exclude docs, extension, analysis, scripts, node_modules, and other noise from the Docker build context - Dockerfile: Node 20→22-alpine, Python 3.11→3.13-slim, DEBIAN_FRONTEND set, pip upgrade before install - Track frontend/package-lock.json for reproducible builds - Update summary.md: date, db.py in project structure, superseded column in schema, two new key patterns (12/13), full 2026-03-19 changelog Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 08:20:24 -04:00
parent 8cd0ce4ecb
commit 9db27ddaae
6 changed files with 2218 additions and 34 deletions
@@ -1,24 +0,0 @@
-{
-  "permissions": {
-    "allow": [
-      "Bash(docker logs:*)",
-      "Bash(docker-compose up:*)",
-      "Bash(docker-compose exec:*)",
-      "Bash(docker compose logs:*)",
-      "Bash(docker compose exec celery sh -c \"grep -i session /data/cookies/patreon_cookies.txt || echo ''No session cookie found''\")",
-      "Bash(docker compose exec celery sh -c \"gallery-dl --cookies /data/cookies/patreon_cookies.txt -v ''https://www.patreon.com/knuxy'' 2>&1 | head -20\")",
-      "Bash(docker compose exec:*)",
-      "Bash(powershell -File - <<'EOF'\nAdd-Type -AssemblyName System.IO.Compression.FileSystem\n$zip = [System.IO.Compression.ZipFile]::OpenRead\\('c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber.xpi'\\)\n$zip.Entries | Select-Object FullName\n$zip.Dispose\\(\\)\nEOF)",
-      "Bash(npx web-ext sign:*)",
-      "Bash(where:*)",
-      "Bash(where npm:*)",
-      "Bash(powershell -Command \"Copy-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\extension\\\\web-ext-artifacts\\\\be3706e9e89640a5b70b-1.0.0.xpi'' ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' -Force; Get-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' | Select-Object Name, Length\")",
-      "WebFetch(domain:github.com)",
-      "WebFetch(domain:raw.githubusercontent.com)",
-      "Bash(python -m py_compile:*)",
-      "Bash(powershell:*)",
-      "Bash(python:*)",
-      "Bash(xargs basename:*)"
-    ]
-  }
-}
@@ -0,0 +1,60 @@
+# Version control
+.git/
+.gitignore
+.dockerignore
+
+# Claude Code
+.claude/
+
+# Documentation and specs
+docs/
+*.md
+README.md
+
+# Analysis artifacts
+analysis/
+
+# Extension packages
+*.xpi
+extension-dist/
+extension/
+
+# Utility scripts
+scripts/
+
+# Environment files (secrets must be injected via env vars, not baked in)
+.env
+.env.*
+
+# Docker compose files (not needed inside the image)
+docker-compose*.yml
+
+# Runtime data (mounted as volumes)
+data/
+config/
+logs/
+downloads/
+cookies/
+
+# Frontend build intermediates
+frontend/node_modules/
+frontend/dist/
+
+# Python cache and test artifacts
+**/__pycache__/
+**/*.py[cod]
+**/*.egg-info/
+**/.pytest_cache/
+**/.mypy_cache/
+**/.ruff_cache/
+**/.coverage
+**/htmlcov/
+
+# Node dev artifacts
+**/npm-debug.log*
+**/yarn-debug.log*
+**/yarn-error.log*
+
+# OS artifacts
+.DS_Store
+Thumbs.db
@@ -44,6 +44,9 @@ yarn-debug.log*
 yarn-error.log*
 .npm

+# Claude Code
+.claude/
+
 # IDE
 .vscode/
 .idea/
@@ -1,5 +1,5 @@
 # Stage 1: Build frontend
-FROM node:20-alpine AS frontend-build
+FROM node:22-alpine AS frontend-build

 WORKDIR /frontend
 COPY frontend/package*.json ./
@@ -8,20 +8,22 @@ COPY frontend/ .
 RUN npm run build

 # Stage 2: Python backend with frontend static files
-FROM python:3.11-slim
+FROM python:3.13-slim

 WORKDIR /app

 # Install system dependencies
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    gcc \
-    libpq-dev \
-    gosu \
+RUN DEBIAN_FRONTEND=noninteractive apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        gcc \
+        libpq-dev \
+        gosu \
    && rm -rf /var/lib/apt/lists/*

 # Install Python dependencies
 COPY backend/requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
+RUN pip install --upgrade pip \
+    && pip install --no-cache-dir -r requirements.txt

 # Copy backend application
 COPY backend/ .
@@ -1,6 +1,6 @@
 # GallerySubscriber - Project Summary

-**Updated:** 2026-02-04 (Pixiv OAuth in extension, improved failed/naming display)
+**Updated:** 2026-03-19 (Comprehensive review: bug fixes, superseded-at-write, error classification, code quality)

 ## Overview

@@ -36,7 +36,7 @@ GallerySubscriber is a Docker-based web application that automates downloading c
 |-------|-------------|
 | Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
 | Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
-| Extension | Firefox WebExtension (Manifest V2) |
+| Extension | Firefox WebExtension (Manifest V2), v1.1.1 |
 | Infrastructure | Docker, PostgreSQL 15, Redis 7 |
 | Download Engine | gallery-dl >=1.31.0 |

@@ -69,7 +69,8 @@ GallerySubscriber/
 │   ├── tasks/
 │   │   ├── celery_app.py    # Celery configuration
 │   │   ├── downloads.py     # Download tasks + scheduler
-│   │   └── maintenance.py   # Cleanup jobs
+│   │   ├── maintenance.py   # Cleanup jobs
+│   │   └── db.py            # Shared DB session helpers (engine + session factory)
 │   └── utils/
 │       ├── encryption.py    # Fernet crypto
 │       └── cookies.py       # Netscape format handling
@@ -114,6 +115,7 @@ GallerySubscriber/
 3. **downloads** - Job records
   - `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
   - `file_count`, `total_size`, `error_type`, `error_message`
+   - `superseded` (boolean, default false) — set to `true` on prior failures when a later run completes successfully

 4. **credentials** - Encrypted platform authentication
   - `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)
@@ -198,6 +200,8 @@ Alternatively, you can manually obtain a token by running `gallery-dl oauth:pixi
 9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
 10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
 11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
+12. **Superseded-at-Write** - When a download completes, all prior `FAILED` records for the same source are bulk-updated to `superseded=True`; stats/activity queries filter by column instead of correlated subquery
+13. **Session Safety** - All API handlers use `async with AsyncSession(...)` for guaranteed cleanup on all code paths

 ## Docker Services

@@ -269,6 +273,19 @@ docker compose logs -f app scheduler worker
 docker compose -f docker-compose.dev.yml up
 ```

+**Build Firefox extension:**
+```bash
+cd extension
+web-ext build
+# Output: web-ext-artifacts/gallerysubscriber-X.X.X.zip
+```
+
+**Test extension locally:**
+```bash
+cd extension
+web-ext run
+```
+
 ## Extension Workflow

 1. User installs Firefox extension
@@ -329,6 +346,25 @@ Download workers do NOT perform these startup tasks to avoid duplicates.

 **Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.

+## Dashboard Features
+
+The Dashboard view (`/`) provides at-a-glance monitoring:
+
+**Stats Cards:** Total sources, completed downloads, failed downloads, active downloads (queued + running)
+
+**Quick Actions:**
+- Check All Sources - triggers download check for all enabled sources
+- Retry All Failed - re-queues all failed downloads
+- Reset Stuck - resets jobs stuck in "running" status
+
+**Active Downloads:** Shows currently queued and running downloads with `{subscription}:{platform}` labels
+
+**Recent Activity:** Shows recent failures and downloads with new content, labeled as `{subscription}:{platform}`
+
+**Sources Needing Attention:** Shows sources with **recent** failures (within last 7 days), not cumulative historical errors. Displays the actual error type from the last failure.
+
+**Credential Status:** Shows which platforms have valid credentials configured
+
 ## Downloads Page Features

 The Downloads view (`/downloads`) provides comprehensive download monitoring:
@@ -422,3 +458,33 @@ The `analysis/` directory (gitignored) contains debugging and analysis artifacts
 - Analysis scripts for pattern debugging

 These files are excluded from version control but preserved locally for ongoing analysis.
+
+## Changelog
+
+### 2026-03-19 — Comprehensive Review
+
+**Bug fixes:**
+- `process_download` retry task now passes `download_id` to `_download_source_async`, preventing the retry from being silently swallowed by the `source_already_active` guard
+- Orphaned-job reset message no longer says "None" — `started_at` is captured before being cleared
+- `asyncio.get_event_loop()` → `asyncio.get_running_loop()` in `gallery_dl.py` (safer in async context)
+- `datetime.utcnow()` (naive, deprecated) replaced with timezone-aware `utcnow()` from `app.models.base` in `gallery_dl.py`
+
+**Superseded logic (write-time):**
+- Added `superseded` boolean column to `downloads` table (migration `003_add_superseded.py` with backfill and partial index)
+- When a download completes, all prior `FAILED` records for the same source are bulk-marked `superseded=True`
+- Removed `_not_superseded_condition()` correlated subquery; all three query sites now filter on `Download.superseded == False`
+
+**Error classification:**
+- `_categorize_error()` now logs `logger.debug(...)` identifying which pattern triggered each classification
+- Pattern lists extracted to named class-level constants on `GalleryDLService`
+- Return code `1` with empty stdout no longer treated as `no_new_content` (was masking real failures)
+
+**Code quality:**
+- `checkAllSources` in `Dashboard.vue` uses `Promise.allSettled()` instead of a sequential `for` loop
+- All API handlers across all blueprints (`downloads.py`, `credentials.py`, `settings.py`, `sources.py`, `subscriptions.py`) use `async with AsyncSession(...)` for guaranteed session cleanup
+- `_count_downloaded_files` now counts only lines starting with `/` (actual downloaded file paths) instead of using fragile heuristics
+- Duplicate `get_async_session` / `_cleanup_engine` helpers extracted to `backend/app/tasks/db.py`; both task files import from there
+
+**Tooling:**
+- `.claude/` added to `.gitignore`; `settings.local.json` untracked
+- `.dockerignore` created — excludes docs, extension, scripts, analysis, node_modules, and other build-context noise