chore: sane ignore files, update summary, bump base images
- Add .claude/ to .gitignore and untrack settings.local.json - Create .dockerignore to exclude docs, extension, analysis, scripts, node_modules, and other noise from the Docker build context - Dockerfile: Node 20→22-alpine, Python 3.11→3.13-slim, DEBIAN_FRONTEND set, pip upgrade before install - Track frontend/package-lock.json for reproducible builds - Update summary.md: date, db.py in project structure, superseded column in schema, two new key patterns (12/13), full 2026-03-19 changelog Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,24 +0,0 @@
|
|||||||
{
|
|
||||||
"permissions": {
|
|
||||||
"allow": [
|
|
||||||
"Bash(docker logs:*)",
|
|
||||||
"Bash(docker-compose up:*)",
|
|
||||||
"Bash(docker-compose exec:*)",
|
|
||||||
"Bash(docker compose logs:*)",
|
|
||||||
"Bash(docker compose exec celery sh -c \"grep -i session /data/cookies/patreon_cookies.txt || echo ''No session cookie found''\")",
|
|
||||||
"Bash(docker compose exec celery sh -c \"gallery-dl --cookies /data/cookies/patreon_cookies.txt -v ''https://www.patreon.com/knuxy'' 2>&1 | head -20\")",
|
|
||||||
"Bash(docker compose exec:*)",
|
|
||||||
"Bash(powershell -File - <<'EOF'\nAdd-Type -AssemblyName System.IO.Compression.FileSystem\n$zip = [System.IO.Compression.ZipFile]::OpenRead\\('c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber.xpi'\\)\n$zip.Entries | Select-Object FullName\n$zip.Dispose\\(\\)\nEOF)",
|
|
||||||
"Bash(npx web-ext sign:*)",
|
|
||||||
"Bash(where:*)",
|
|
||||||
"Bash(where npm:*)",
|
|
||||||
"Bash(powershell -Command \"Copy-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\extension\\\\web-ext-artifacts\\\\be3706e9e89640a5b70b-1.0.0.xpi'' ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' -Force; Get-Item ''c:\\\\Users\\\\bvandeusen\\\\Nextcloud\\\\Projects\\\\GallerySubscriber\\\\gallerysubscriber-signed.xpi'' | Select-Object Name, Length\")",
|
|
||||||
"WebFetch(domain:github.com)",
|
|
||||||
"WebFetch(domain:raw.githubusercontent.com)",
|
|
||||||
"Bash(python -m py_compile:*)",
|
|
||||||
"Bash(powershell:*)",
|
|
||||||
"Bash(python:*)",
|
|
||||||
"Bash(xargs basename:*)"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Version control
|
||||||
|
.git/
|
||||||
|
.gitignore
|
||||||
|
.dockerignore
|
||||||
|
|
||||||
|
# Claude Code
|
||||||
|
.claude/
|
||||||
|
|
||||||
|
# Documentation and specs
|
||||||
|
docs/
|
||||||
|
*.md
|
||||||
|
README.md
|
||||||
|
|
||||||
|
# Analysis artifacts
|
||||||
|
analysis/
|
||||||
|
|
||||||
|
# Extension packages
|
||||||
|
*.xpi
|
||||||
|
extension-dist/
|
||||||
|
extension/
|
||||||
|
|
||||||
|
# Utility scripts
|
||||||
|
scripts/
|
||||||
|
|
||||||
|
# Environment files (secrets must be injected via env vars, not baked in)
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
|
||||||
|
# Docker compose files (not needed inside the image)
|
||||||
|
docker-compose*.yml
|
||||||
|
|
||||||
|
# Runtime data (mounted as volumes)
|
||||||
|
data/
|
||||||
|
config/
|
||||||
|
logs/
|
||||||
|
downloads/
|
||||||
|
cookies/
|
||||||
|
|
||||||
|
# Frontend build intermediates
|
||||||
|
frontend/node_modules/
|
||||||
|
frontend/dist/
|
||||||
|
|
||||||
|
# Python cache and test artifacts
|
||||||
|
**/__pycache__/
|
||||||
|
**/*.py[cod]
|
||||||
|
**/*.egg-info/
|
||||||
|
**/.pytest_cache/
|
||||||
|
**/.mypy_cache/
|
||||||
|
**/.ruff_cache/
|
||||||
|
**/.coverage
|
||||||
|
**/htmlcov/
|
||||||
|
|
||||||
|
# Node dev artifacts
|
||||||
|
**/npm-debug.log*
|
||||||
|
**/yarn-debug.log*
|
||||||
|
**/yarn-error.log*
|
||||||
|
|
||||||
|
# OS artifacts
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -44,6 +44,9 @@ yarn-debug.log*
|
|||||||
yarn-error.log*
|
yarn-error.log*
|
||||||
.npm
|
.npm
|
||||||
|
|
||||||
|
# Claude Code
|
||||||
|
.claude/
|
||||||
|
|
||||||
# IDE
|
# IDE
|
||||||
.vscode/
|
.vscode/
|
||||||
.idea/
|
.idea/
|
||||||
|
|||||||
+9
-7
@@ -1,5 +1,5 @@
|
|||||||
# Stage 1: Build frontend
|
# Stage 1: Build frontend
|
||||||
FROM node:20-alpine AS frontend-build
|
FROM node:22-alpine AS frontend-build
|
||||||
|
|
||||||
WORKDIR /frontend
|
WORKDIR /frontend
|
||||||
COPY frontend/package*.json ./
|
COPY frontend/package*.json ./
|
||||||
@@ -8,20 +8,22 @@ COPY frontend/ .
|
|||||||
RUN npm run build
|
RUN npm run build
|
||||||
|
|
||||||
# Stage 2: Python backend with frontend static files
|
# Stage 2: Python backend with frontend static files
|
||||||
FROM python:3.11-slim
|
FROM python:3.13-slim
|
||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
# Install system dependencies
|
# Install system dependencies
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
RUN DEBIAN_FRONTEND=noninteractive apt-get update \
|
||||||
gcc \
|
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
|
||||||
libpq-dev \
|
gcc \
|
||||||
gosu \
|
libpq-dev \
|
||||||
|
gosu \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Install Python dependencies
|
# Install Python dependencies
|
||||||
COPY backend/requirements.txt .
|
COPY backend/requirements.txt .
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
RUN pip install --upgrade pip \
|
||||||
|
&& pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
# Copy backend application
|
# Copy backend application
|
||||||
COPY backend/ .
|
COPY backend/ .
|
||||||
|
|||||||
Generated
+2077
File diff suppressed because it is too large
Load Diff
+69
-3
@@ -1,6 +1,6 @@
|
|||||||
# GallerySubscriber - Project Summary
|
# GallerySubscriber - Project Summary
|
||||||
|
|
||||||
**Updated:** 2026-02-04 (Pixiv OAuth in extension, improved failed/naming display)
|
**Updated:** 2026-03-19 (Comprehensive review: bug fixes, superseded-at-write, error classification, code quality)
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
@@ -36,7 +36,7 @@ GallerySubscriber is a Docker-based web application that automates downloading c
|
|||||||
|-------|-------------|
|
|-------|-------------|
|
||||||
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
|
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
|
||||||
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
|
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
|
||||||
| Extension | Firefox WebExtension (Manifest V2) |
|
| Extension | Firefox WebExtension (Manifest V2), v1.1.1 |
|
||||||
| Infrastructure | Docker, PostgreSQL 15, Redis 7 |
|
| Infrastructure | Docker, PostgreSQL 15, Redis 7 |
|
||||||
| Download Engine | gallery-dl >=1.31.0 |
|
| Download Engine | gallery-dl >=1.31.0 |
|
||||||
|
|
||||||
@@ -69,7 +69,8 @@ GallerySubscriber/
|
|||||||
│ ├── tasks/
|
│ ├── tasks/
|
||||||
│ │ ├── celery_app.py # Celery configuration
|
│ │ ├── celery_app.py # Celery configuration
|
||||||
│ │ ├── downloads.py # Download tasks + scheduler
|
│ │ ├── downloads.py # Download tasks + scheduler
|
||||||
│ │ └── maintenance.py # Cleanup jobs
|
│ │ ├── maintenance.py # Cleanup jobs
|
||||||
|
│ │ └── db.py # Shared DB session helpers (engine + session factory)
|
||||||
│ └── utils/
|
│ └── utils/
|
||||||
│ ├── encryption.py # Fernet crypto
|
│ ├── encryption.py # Fernet crypto
|
||||||
│ └── cookies.py # Netscape format handling
|
│ └── cookies.py # Netscape format handling
|
||||||
@@ -114,6 +115,7 @@ GallerySubscriber/
|
|||||||
3. **downloads** - Job records
|
3. **downloads** - Job records
|
||||||
- `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
|
- `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
|
||||||
- `file_count`, `total_size`, `error_type`, `error_message`
|
- `file_count`, `total_size`, `error_type`, `error_message`
|
||||||
|
- `superseded` (boolean, default false) — set to `true` on prior failures when a later run completes successfully
|
||||||
|
|
||||||
4. **credentials** - Encrypted platform authentication
|
4. **credentials** - Encrypted platform authentication
|
||||||
- `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)
|
- `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)
|
||||||
@@ -198,6 +200,8 @@ Alternatively, you can manually obtain a token by running `gallery-dl oauth:pixi
|
|||||||
9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
|
9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
|
||||||
10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
|
10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
|
||||||
11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
|
11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
|
||||||
|
12. **Superseded-at-Write** - When a download completes, all prior `FAILED` records for the same source are bulk-updated to `superseded=True`; stats/activity queries filter by column instead of correlated subquery
|
||||||
|
13. **Session Safety** - All API handlers use `async with AsyncSession(...)` for guaranteed cleanup on all code paths
|
||||||
|
|
||||||
## Docker Services
|
## Docker Services
|
||||||
|
|
||||||
@@ -269,6 +273,19 @@ docker compose logs -f app scheduler worker
|
|||||||
docker compose -f docker-compose.dev.yml up
|
docker compose -f docker-compose.dev.yml up
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Build Firefox extension:**
|
||||||
|
```bash
|
||||||
|
cd extension
|
||||||
|
web-ext build
|
||||||
|
# Output: web-ext-artifacts/gallerysubscriber-X.X.X.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test extension locally:**
|
||||||
|
```bash
|
||||||
|
cd extension
|
||||||
|
web-ext run
|
||||||
|
```
|
||||||
|
|
||||||
## Extension Workflow
|
## Extension Workflow
|
||||||
|
|
||||||
1. User installs Firefox extension
|
1. User installs Firefox extension
|
||||||
@@ -329,6 +346,25 @@ Download workers do NOT perform these startup tasks to avoid duplicates.
|
|||||||
|
|
||||||
**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.
|
**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.
|
||||||
|
|
||||||
|
## Dashboard Features
|
||||||
|
|
||||||
|
The Dashboard view (`/`) provides at-a-glance monitoring:
|
||||||
|
|
||||||
|
**Stats Cards:** Total sources, completed downloads, failed downloads, active downloads (queued + running)
|
||||||
|
|
||||||
|
**Quick Actions:**
|
||||||
|
- Check All Sources - triggers download check for all enabled sources
|
||||||
|
- Retry All Failed - re-queues all failed downloads
|
||||||
|
- Reset Stuck - resets jobs stuck in "running" status
|
||||||
|
|
||||||
|
**Active Downloads:** Shows currently queued and running downloads with `{subscription}:{platform}` labels
|
||||||
|
|
||||||
|
**Recent Activity:** Shows recent failures and downloads with new content, labeled as `{subscription}:{platform}`
|
||||||
|
|
||||||
|
**Sources Needing Attention:** Shows sources with **recent** failures (within last 7 days), not cumulative historical errors. Displays the actual error type from the last failure.
|
||||||
|
|
||||||
|
**Credential Status:** Shows which platforms have valid credentials configured
|
||||||
|
|
||||||
## Downloads Page Features
|
## Downloads Page Features
|
||||||
|
|
||||||
The Downloads view (`/downloads`) provides comprehensive download monitoring:
|
The Downloads view (`/downloads`) provides comprehensive download monitoring:
|
||||||
@@ -422,3 +458,33 @@ The `analysis/` directory (gitignored) contains debugging and analysis artifacts
|
|||||||
- Analysis scripts for pattern debugging
|
- Analysis scripts for pattern debugging
|
||||||
|
|
||||||
These files are excluded from version control but preserved locally for ongoing analysis.
|
These files are excluded from version control but preserved locally for ongoing analysis.
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-03-19 — Comprehensive Review
|
||||||
|
|
||||||
|
**Bug fixes:**
|
||||||
|
- `process_download` retry task now passes `download_id` to `_download_source_async`, preventing the retry from being silently swallowed by the `source_already_active` guard
|
||||||
|
- Orphaned-job reset message no longer says "None" — `started_at` is captured before being cleared
|
||||||
|
- `asyncio.get_event_loop()` → `asyncio.get_running_loop()` in `gallery_dl.py` (safer in async context)
|
||||||
|
- `datetime.utcnow()` (naive, deprecated) replaced with timezone-aware `utcnow()` from `app.models.base` in `gallery_dl.py`
|
||||||
|
|
||||||
|
**Superseded logic (write-time):**
|
||||||
|
- Added `superseded` boolean column to `downloads` table (migration `003_add_superseded.py` with backfill and partial index)
|
||||||
|
- When a download completes, all prior `FAILED` records for the same source are bulk-marked `superseded=True`
|
||||||
|
- Removed `_not_superseded_condition()` correlated subquery; all three query sites now filter on `Download.superseded == False`
|
||||||
|
|
||||||
|
**Error classification:**
|
||||||
|
- `_categorize_error()` now logs `logger.debug(...)` identifying which pattern triggered each classification
|
||||||
|
- Pattern lists extracted to named class-level constants on `GalleryDLService`
|
||||||
|
- Return code `1` with empty stdout no longer treated as `no_new_content` (was masking real failures)
|
||||||
|
|
||||||
|
**Code quality:**
|
||||||
|
- `checkAllSources` in `Dashboard.vue` uses `Promise.allSettled()` instead of a sequential `for` loop
|
||||||
|
- All API handlers across all blueprints (`downloads.py`, `credentials.py`, `settings.py`, `sources.py`, `subscriptions.py`) use `async with AsyncSession(...)` for guaranteed session cleanup
|
||||||
|
- `_count_downloaded_files` now counts only lines starting with `/` (actual downloaded file paths) instead of using fragile heuristics
|
||||||
|
- Duplicate `get_async_session` / `_cleanup_engine` helpers extracted to `backend/app/tasks/db.py`; both task files import from there
|
||||||
|
|
||||||
|
**Tooling:**
|
||||||
|
- `.claude/` added to `.gitignore`; `settings.local.json` untracked
|
||||||
|
- `.dockerignore` created — excludes docs, extension, scripts, analysis, node_modules, and other build-context noise
|
||||||
|
|||||||
Reference in New Issue
Block a user