chore: sane ignore files, update summary, bump base images
- Add .claude/ to .gitignore and untrack settings.local.json - Create .dockerignore to exclude docs, extension, analysis, scripts, node_modules, and other noise from the Docker build context - Dockerfile: Node 20→22-alpine, Python 3.11→3.13-slim, DEBIAN_FRONTEND set, pip upgrade before install - Track frontend/package-lock.json for reproducible builds - Update summary.md: date, db.py in project structure, superseded column in schema, two new key patterns (12/13), full 2026-03-19 changelog Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+69
-3
@@ -1,6 +1,6 @@
|
||||
# GallerySubscriber - Project Summary
|
||||
|
||||
**Updated:** 2026-02-04 (Pixiv OAuth in extension, improved failed/naming display)
|
||||
**Updated:** 2026-03-19 (Comprehensive review: bug fixes, superseded-at-write, error classification, code quality)
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -36,7 +36,7 @@ GallerySubscriber is a Docker-based web application that automates downloading c
|
||||
|-------|-------------|
|
||||
| Backend | Quart 0.19.4, SQLAlchemy 2.0.25, asyncpg, Celery 5.3.6, Hypercorn |
|
||||
| Frontend | Vue 3.4.15, Vuetify 3.5.1, Pinia 2.1.7, Vite 5.0.11 |
|
||||
| Extension | Firefox WebExtension (Manifest V2) |
|
||||
| Extension | Firefox WebExtension (Manifest V2), v1.1.1 |
|
||||
| Infrastructure | Docker, PostgreSQL 15, Redis 7 |
|
||||
| Download Engine | gallery-dl >=1.31.0 |
|
||||
|
||||
@@ -69,7 +69,8 @@ GallerySubscriber/
|
||||
│ ├── tasks/
|
||||
│ │ ├── celery_app.py # Celery configuration
|
||||
│ │ ├── downloads.py # Download tasks + scheduler
|
||||
│ │ └── maintenance.py # Cleanup jobs
|
||||
│ │ ├── maintenance.py # Cleanup jobs
|
||||
│ │ └── db.py # Shared DB session helpers (engine + session factory)
|
||||
│ └── utils/
|
||||
│ ├── encryption.py # Fernet crypto
|
||||
│ └── cookies.py # Netscape format handling
|
||||
@@ -114,6 +115,7 @@ GallerySubscriber/
|
||||
3. **downloads** - Job records
|
||||
- `source_id` (FK), `status` (queued|pending|running|completed|failed|skipped)
|
||||
- `file_count`, `total_size`, `error_type`, `error_message`
|
||||
- `superseded` (boolean, default false) — set to `true` on prior failures when a later run completes successfully
|
||||
|
||||
4. **credentials** - Encrypted platform authentication
|
||||
- `platform` (unique), `credential_type` (cookies|token), `data` (encrypted binary)
|
||||
@@ -198,6 +200,8 @@ Alternatively, you can manually obtain a token by running `gallery-dl oauth:pixi
|
||||
9. **Redis DB Mismatch Detection** - Workers log markers to detect configuration inconsistencies
|
||||
10. **Stale Job Recovery** - Automatic re-queuing of lost Celery tasks with duplicate prevention
|
||||
11. **Race Condition Guards** - Downloads check status before claiming to prevent duplicate processing
|
||||
12. **Superseded-at-Write** - When a download completes, all prior `FAILED` records for the same source are bulk-updated to `superseded=True`; stats/activity queries filter by column instead of correlated subquery
|
||||
13. **Session Safety** - All API handlers use `async with AsyncSession(...)` for guaranteed cleanup on all code paths
|
||||
|
||||
## Docker Services
|
||||
|
||||
@@ -269,6 +273,19 @@ docker compose logs -f app scheduler worker
|
||||
docker compose -f docker-compose.dev.yml up
|
||||
```
|
||||
|
||||
**Build Firefox extension:**
|
||||
```bash
|
||||
cd extension
|
||||
web-ext build
|
||||
# Output: web-ext-artifacts/gallerysubscriber-X.X.X.zip
|
||||
```
|
||||
|
||||
**Test extension locally:**
|
||||
```bash
|
||||
cd extension
|
||||
web-ext run
|
||||
```
|
||||
|
||||
## Extension Workflow
|
||||
|
||||
1. User installs Firefox extension
|
||||
@@ -329,6 +346,25 @@ Download workers do NOT perform these startup tasks to avoid duplicates.
|
||||
|
||||
**Note:** `sleep-request` (delay between HTTP requests during pagination) is automatically set to 1/4 of `rate_limit` (minimum 0.5s) to speed up archive checks while maintaining safe download pacing.
|
||||
|
||||
## Dashboard Features
|
||||
|
||||
The Dashboard view (`/`) provides at-a-glance monitoring:
|
||||
|
||||
**Stats Cards:** Total sources, completed downloads, failed downloads, active downloads (queued + running)
|
||||
|
||||
**Quick Actions:**
|
||||
- Check All Sources - triggers download check for all enabled sources
|
||||
- Retry All Failed - re-queues all failed downloads
|
||||
- Reset Stuck - resets jobs stuck in "running" status
|
||||
|
||||
**Active Downloads:** Shows currently queued and running downloads with `{subscription}:{platform}` labels
|
||||
|
||||
**Recent Activity:** Shows recent failures and downloads with new content, labeled as `{subscription}:{platform}`
|
||||
|
||||
**Sources Needing Attention:** Shows sources with **recent** failures (within last 7 days), not cumulative historical errors. Displays the actual error type from the last failure.
|
||||
|
||||
**Credential Status:** Shows which platforms have valid credentials configured
|
||||
|
||||
## Downloads Page Features
|
||||
|
||||
The Downloads view (`/downloads`) provides comprehensive download monitoring:
|
||||
@@ -422,3 +458,33 @@ The `analysis/` directory (gitignored) contains debugging and analysis artifacts
|
||||
- Analysis scripts for pattern debugging
|
||||
|
||||
These files are excluded from version control but preserved locally for ongoing analysis.
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-03-19 — Comprehensive Review
|
||||
|
||||
**Bug fixes:**
|
||||
- `process_download` retry task now passes `download_id` to `_download_source_async`, preventing the retry from being silently swallowed by the `source_already_active` guard
|
||||
- Orphaned-job reset message no longer says "None" — `started_at` is captured before being cleared
|
||||
- `asyncio.get_event_loop()` → `asyncio.get_running_loop()` in `gallery_dl.py` (safer in async context)
|
||||
- `datetime.utcnow()` (naive, deprecated) replaced with timezone-aware `utcnow()` from `app.models.base` in `gallery_dl.py`
|
||||
|
||||
**Superseded logic (write-time):**
|
||||
- Added `superseded` boolean column to `downloads` table (migration `003_add_superseded.py` with backfill and partial index)
|
||||
- When a download completes, all prior `FAILED` records for the same source are bulk-marked `superseded=True`
|
||||
- Removed `_not_superseded_condition()` correlated subquery; all three query sites now filter on `Download.superseded == False`
|
||||
|
||||
**Error classification:**
|
||||
- `_categorize_error()` now logs `logger.debug(...)` identifying which pattern triggered each classification
|
||||
- Pattern lists extracted to named class-level constants on `GalleryDLService`
|
||||
- Return code `1` with empty stdout no longer treated as `no_new_content` (was masking real failures)
|
||||
|
||||
**Code quality:**
|
||||
- `checkAllSources` in `Dashboard.vue` uses `Promise.allSettled()` instead of a sequential `for` loop
|
||||
- All API handlers across all blueprints (`downloads.py`, `credentials.py`, `settings.py`, `sources.py`, `subscriptions.py`) use `async with AsyncSession(...)` for guaranteed session cleanup
|
||||
- `_count_downloaded_files` now counts only lines starting with `/` (actual downloaded file paths) instead of using fragile heuristics
|
||||
- Duplicate `get_async_session` / `_cleanup_engine` helpers extracted to `backend/app/tasks/db.py`; both task files import from there
|
||||
|
||||
**Tooling:**
|
||||
- `.claude/` added to `.gitignore`; `settings.local.json` untracked
|
||||
- `.dockerignore` created — excludes docs, extension, scripts, analysis, node_modules, and other build-context noise
|
||||
|
||||
Reference in New Issue
Block a user