chore: sane ignore files, update summary, bump base images

- Add .claude/ to .gitignore and untrack settings.local.json
- Create .dockerignore to exclude docs, extension, analysis, scripts,
  node_modules, and other noise from the Docker build context
- Dockerfile: Node 20→22-alpine, Python 3.11→3.13-slim, DEBIAN_FRONTEND
  set, pip upgrade before install
- Track frontend/package-lock.json for reproducible builds
- Update summary.md: date, db.py in project structure, superseded column
  in schema, two new key patterns (12/13), full 2026-03-19 changelog

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-19 08:20:24 -04:00
parent 8cd0ce4ecb
commit 9db27ddaae
6 changed files with 2218 additions and 34 deletions
-24
View File
@@ -1,24 +0,0 @@
{
"permissions": {
"allow": [
"Bash(docker logs:*)",
"Bash(docker-compose up:*)",
"Bash(docker-compose exec:*)",
"Bash(docker compose logs:*)",
"Bash(docker compose exec celery sh -c \"grep -i session /data/cookies/patreon_cookies.txt || echo ''No session cookie found''\")",
"Bash(docker compose exec celery sh -c \"gallery-dl --cookies /data/cookies/patreon_cookies.txt -v ''https://www.patreon.com/knuxy'' 2>&1 | head -20\")",
"Bash(docker compose exec:*)",
"Bash(powershell -File - <<'EOF'\nAdd-Type -AssemblyName System.IO.Compression.FileSystem\n$zip = [System.IO.Compression.ZipFile]::OpenRead\\('c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber.xpi'\\)\n$zip.Entries | Select-Object FullName\n$zip.Dispose\\(\\)\nEOF)",
"Bash(npx web-ext sign:*)",
"Bash(where:*)",
"Bash(where npm:*)",
"Bash(powershell -Command \"Copy-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\extension\\\\web-ext-artifacts\\\\be3706e9e89640a5b70b-1.0.0.xpi'' ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' -Force; Get-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' | Select-Object Name, Length\")",
"WebFetch(domain:github.com)",
"WebFetch(domain:raw.githubusercontent.com)",
"Bash(python -m py_compile:*)",
"Bash(powershell:*)",
"Bash(python:*)",
"Bash(xargs basename:*)"
]
}
}
+60
View File
@@ -0,0 +1,60 @@
# Version control
.git/
.gitignore
.dockerignore
# Claude Code
.claude/
# Documentation and specs
docs/
*.md
README.md
# Analysis artifacts
analysis/
# Extension packages
*.xpi
extension-dist/
extension/
# Utility scripts
scripts/
# Environment files (secrets must be injected via env vars, not baked in)
.env
.env.*
# Docker compose files (not needed inside the image)
docker-compose*.yml
# Runtime data (mounted as volumes)
data/
config/
logs/
downloads/
cookies/
# Frontend build intermediates
frontend/node_modules/
frontend/dist/
# Python cache and test artifacts
**/__pycache__/
**/*.py[cod]
**/*.egg-info/
**/.pytest_cache/
**/.mypy_cache/
**/.ruff_cache/
**/.coverage
**/htmlcov/
# Node dev artifacts
**/npm-debug.log*
**/yarn-debug.log*
**/yarn-error.log*
# OS artifacts
.DS_Store
Thumbs.db
+3
View File
@@ -44,6 +44,9 @@ yarn-debug.log*
yarn-error.log* yarn-error.log*
.npm .npm
# Claude Code
.claude/
# IDE # IDE
.vscode/ .vscode/
.idea/ .idea/
+9 -7
View File
@@ -1,5 +1,5 @@
# Stage 1: Build frontend # Stage 1: Build frontend
FROM node:20-alpine AS frontend-build FROM node:22-alpine AS frontend-build
WORKDIR /frontend WORKDIR /frontend
COPY frontend/package*.json ./ COPY frontend/package*.json ./
@@ -8,20 +8,22 @@ COPY frontend/ .
RUN npm run build RUN npm run build
# Stage 2: Python backend with frontend static files # Stage 2: Python backend with frontend static files
FROM python:3.11-slim FROM python:3.13-slim
WORKDIR /app WORKDIR /app
# Install system dependencies # Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
gcc \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
libpq-dev \ gcc \
gosu \ libpq-dev \
gosu \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Install Python dependencies # Install Python dependencies
COPY backend/requirements.txt . COPY backend/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --upgrade pip \
&& pip install --no-cache-dir -r requirements.txt
# Copy backend application # Copy backend application
COPY backend/ . COPY backend/ .
+2077
View File
File diff suppressed because it is too large Load Diff
+69 -3
View File
@@ -1,6 +1,6 @@
# GallerySubscriber - Project Summary # GallerySubscriber - Project Summary
**Updated:** 2026-02-04 (Pixiv OAuth in extension, improved failed/naming display) **Updated:** 2026-03-19 (Comprehensive review: bug fixes, superseded-at-write, error classification, code quality)
## Overview ## Overview
@@ -36,7 +36,7 @@ GallerySubscriber is a Docker-based web application that automates downloading c
|-------|-------------| |-------|-------------|
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn | | Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 | | Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
| Extension | Firefox WebExtension (Manifest V2) | | Extension | Firefox WebExtension (Manifest V2), v1.1.1 |
| Infrastructure | Docker, PostgreSQL 15, Redis 7 | | Infrastructure | Docker, PostgreSQL 15, Redis 7 |
| Download Engine | gallery-dl >=1.31.0 | | Download Engine | gallery-dl >=1.31.0 |
@@ -69,7 +69,8 @@ GallerySubscriber/
│ ├── tasks/ │ ├── tasks/
│ │ ├── celery_app.py # Celery configuration │ │ ├── celery_app.py # Celery configuration
│ │ ├── downloads.py # Download tasks + scheduler │ │ ├── downloads.py # Download tasks + scheduler
│ │ ── maintenance.py # Cleanup jobs │ │ ── maintenance.py # Cleanup jobs
│ │ └── db.py # Shared DB session helpers (engine + session factory)
│ └── utils/ │ └── utils/
│ ├── encryption.py # Fernet crypto │ ├── encryption.py # Fernet crypto
│ └── cookies.py # Netscape format handling │ └── cookies.py # Netscape format handling
@@ -114,6 +115,7 @@ GallerySubscriber/
3. **downloads** - Job records 3. **downloads** - Job records
- `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped) - `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
- `file_count`, `total_size`, `error_type`, `error_message` - `file_count`, `total_size`, `error_type`, `error_message`
- `superseded` (boolean, default false) — set to `true` on prior failures when a later run completes successfully
4. **credentials** - Encrypted platform authentication 4. **credentials** - Encrypted platform authentication
- `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary) - `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)
@@ -198,6 +200,8 @@ Alternatively, you can manually obtain a token by running `gallery-dl oauth:pixi
9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies 9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention 10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing 11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
12. **Superseded-at-Write** - When a download completes, all prior `FAILED` records for the same source are bulk-updated to `superseded=True`; stats/activity queries filter by column instead of correlated subquery
13. **Session Safety** - All API handlers use `async with AsyncSession(...)` for guaranteed cleanup on all code paths
## Docker Services ## Docker Services
@@ -269,6 +273,19 @@ docker compose logs -f app scheduler worker
docker compose -f docker-compose.dev.yml up docker compose -f docker-compose.dev.yml up
``` ```
**Build Firefox extension:**
```bash
cd extension
web-ext build
# Output: web-ext-artifacts/gallerysubscriber-X.X.X.zip
```
**Test extension locally:**
```bash
cd extension
web-ext run
```
## Extension Workflow ## Extension Workflow
1. User installs Firefox extension 1. User installs Firefox extension
@@ -329,6 +346,25 @@ Download workers do NOT perform these startup tasks to avoid duplicates.
**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing. **Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.
## Dashboard Features
The Dashboard view (`/`) provides at-a-glance monitoring:
**Stats Cards:** Total sources, completed downloads, failed downloads, active downloads (queued + running)
**Quick Actions:**
- Check All Sources - triggers download check for all enabled sources
- Retry All Failed - re-queues all failed downloads
- Reset Stuck - resets jobs stuck in "running" status
**Active Downloads:** Shows currently queued and running downloads with `{subscription}:{platform}` labels
**Recent Activity:** Shows recent failures and downloads with new content, labeled as `{subscription}:{platform}`
**Sources Needing Attention:** Shows sources with **recent** failures (within last 7 days), not cumulative historical errors. Displays the actual error type from the last failure.
**Credential Status:** Shows which platforms have valid credentials configured
## Downloads Page Features ## Downloads Page Features
The Downloads view (`/downloads`) provides comprehensive download monitoring: The Downloads view (`/downloads`) provides comprehensive download monitoring:
@@ -422,3 +458,33 @@ The `analysis/` directory (gitignored) contains debugging and analysis artifacts
- Analysis scripts for pattern debugging - Analysis scripts for pattern debugging
These files are excluded from version control but preserved locally for ongoing analysis. These files are excluded from version control but preserved locally for ongoing analysis.
## Changelog
### 2026-03-19 — Comprehensive Review
**Bug fixes:**
- `process_download` retry task now passes `download_id` to `_download_source_async`, preventing the retry from being silently swallowed by the `source_already_active` guard
- Orphaned-job reset message no longer says "None" — `started_at` is captured before being cleared
- `asyncio.get_event_loop()``asyncio.get_running_loop()` in `gallery_dl.py` (safer in async context)
- `datetime.utcnow()` (naive, deprecated) replaced with timezone-aware `utcnow()` from `app.models.base` in `gallery_dl.py`
**Superseded logic (write-time):**
- Added `superseded` boolean column to `downloads` table (migration `003_add_superseded.py` with backfill and partial index)
- When a download completes, all prior `FAILED` records for the same source are bulk-marked `superseded=True`
- Removed `_not_superseded_condition()` correlated subquery; all three query sites now filter on `Download.superseded == False`
**Error classification:**
- `_categorize_error()` now logs `logger.debug(...)` identifying which pattern triggered each classification
- Pattern lists extracted to named class-level constants on `GalleryDLService`
- Return code `1` with empty stdout no longer treated as `no_new_content` (was masking real failures)
**Code quality:**
- `checkAllSources` in `Dashboard.vue` uses `Promise.allSettled()` instead of a sequential `for` loop
- All API handlers across all blueprints (`downloads.py`, `credentials.py`, `settings.py`, `sources.py`, `subscriptions.py`) use `async with AsyncSession(...)` for guaranteed session cleanup
- `_count_downloaded_files` now counts only lines starting with `/` (actual downloaded file paths) instead of using fragile heuristics
- Duplicate `get_async_session` / `_cleanup_engine` helpers extracted to `backend/app/tasks/db.py`; both task files import from there
**Tooling:**
- `.claude/` added to `.gitignore`; `settings.local.json` untracked
- `.dockerignore` created — excludes docs, extension, scripts, analysis, node_modules, and other build-context noise