tuning error recognition
This commit is contained in:
+42
-1
@@ -1,6 +1,6 @@
|
||||
# GallerySubscriber - Project Summary
|
||||
|
||||
**Updated:** 2026-01-29 (Redis DB config, mismatch detection, stale job recovery, Downloads UI improvements)
|
||||
**Updated:** 2026-01-30 (Error recognition improvements, analysis tooling)
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -368,3 +368,44 @@ Multiple layers prevent duplicate downloads for the same source:
|
||||
|
||||
3. **Scheduler Logic** (`downloads.py`):
|
||||
- `scheduled_check` skips sources with existing QUEUED or RUNNING downloads
|
||||
|
||||
## Error Classification System
|
||||
|
||||
The `_categorize_error()` method in `gallery_dl.py` classifies download failures into actionable error types:
|
||||
|
||||
| Error Type | Meaning | User Action |
|
||||
|------------|---------|-------------|
|
||||
| `no_new_content` | All content already downloaded (success) | None needed |
|
||||
| `auth_error` | Cookies expired or invalid | Re-export cookies from extension |
|
||||
| `rate_limited` | Too many requests (429) | Increase rate_limit setting |
|
||||
| `not_found` | Creator/content deleted or moved | Check URL, may need to remove |
|
||||
| `access_denied` | Insufficient subscription tier | Upgrade pledge or remove source |
|
||||
| `network_error` | Connection issues | Check network, will auto-retry |
|
||||
| `timeout` | Download took too long | Will auto-retry |
|
||||
| `unsupported_url` | URL not recognized by gallery-dl | Check URL format |
|
||||
| `unknown_error` | Unclassified failure | Check logs for details |
|
||||
|
||||
**Classification Logic:**
|
||||
|
||||
1. **Skip Detection First** - Checks for `#` prefixed lines in stdout (gallery-dl's skip message format) and text patterns like "skipping", "already exists". If skips are detected with no actual errors, returns `no_new_content`.
|
||||
|
||||
2. **Actual Error Detection** - Looks for `][error]` log level, exceptions, and tracebacks before proceeding to pattern matching.
|
||||
|
||||
3. **Specific Patterns** - Uses targeted patterns (e.g., `"timed out"` not just `"timeout"`) to avoid false positives from config values in debug output.
|
||||
|
||||
## Backend Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `backend/scripts/export_failed_logs.py` | Export failed download logs to JSON for analysis |
|
||||
|
||||
## Analysis Directory
|
||||
|
||||
The `analysis/` directory (gitignored) contains debugging and analysis artifacts:
|
||||
|
||||
- `failed_logs_export_*.json` - Exported download logs for analysis
|
||||
- `analysis_results.json` - Parsed analysis results
|
||||
- `ERROR_RECOGNITION_ANALYSIS.md` - Detailed findings report
|
||||
- Analysis scripts for pattern debugging
|
||||
|
||||
These files are excluded from version control but preserved locally for ongoing analysis.
|
||||
|
||||
Reference in New Issue
Block a user