tuning error recognition

This commit is contained in:
Bryan Van Deusen
2026-01-30 22:04:07 -05:00
parent 097ab16c50
commit 3ee7de7ecd
7 changed files with 546 additions and 31 deletions
+42 -1
View File
@@ -1,6 +1,6 @@
# GallerySubscriber - Project Summary
**Updated:** 2026-01-29 (Redis DB config, mismatch detection, stale job recovery, Downloads UI improvements)
**Updated:** 2026-01-30 (Error recognition improvements, analysis tooling)
## Overview
@@ -368,3 +368,44 @@ Multiple layers prevent duplicate downloads for the same source:
3. **Scheduler Logic** (`downloads.py`):
- `scheduled_check` skips sources with existing QUEUED or RUNNING downloads
## Error Classification System
The `_categorize_error()` method in `gallery_dl.py` classifies download failures into actionable error types:
| Error Type | Meaning | User Action |
|------------|---------|-------------|
| `no_new_content` | All content already downloaded (success) | None needed |
| `auth_error` | Cookies expired or invalid | Re-export cookies from extension |
| `rate_limited` | Too many requests (429) | Increase rate_limit setting |
| `not_found` | Creator/content deleted or moved | Check URL, may need to remove |
| `access_denied` | Insufficient subscription tier | Upgrade pledge or remove source |
| `network_error` | Connection issues | Check network, will auto-retry |
| `timeout` | Download took too long | Will auto-retry |
| `unsupported_url` | URL not recognized by gallery-dl | Check URL format |
| `unknown_error` | Unclassified failure | Check logs for details |
**Classification Logic:**
1. **Skip Detection First** - Checks for `#` prefixed lines in stdout (gallery-dl's skip message format) and text patterns like "skipping", "already exists". If skips are detected with no actual errors, returns `no_new_content`.
2. **Actual Error Detection** - Looks for `][error]` log level, exceptions, and tracebacks before proceeding to pattern matching.
3. **Specific Patterns** - Uses targeted patterns (e.g., `"timed out"` not just `"timeout"`) to avoid false positives from config values in debug output.
## Backend Scripts
| Script | Purpose |
|--------|---------|
| `backend/scripts/export_failed_logs.py` | Export failed download logs to JSON for analysis |
## Analysis Directory
The `analysis/` directory (gitignored) contains debugging and analysis artifacts:
- `failed_logs_export_*.json` - Exported download logs for analysis
- `analysis_results.json` - Parsed analysis results
- `ERROR_RECOGNITION_ANALYSIS.md` - Detailed findings report
- Analysis scripts for pattern debugging
These files are excluded from version control but preserved locally for ongoing analysis.