Walks the downloads tree and reports files that fail magic-byte
validation — the "how many pre-existing files in my library are
silently truncated?" question, answered. Skips the _quarantine
subtree (already known-bad) and caps suspect_paths at 500 with a
truncated flag so the persisted report can't grow unbounded.
On-demand only: a full filesystem walk on a NAS-mounted library is
expensive and the user should choose when to pay for it. Pre-existing
files are reported, not quarantined — they may be the only copy and
the user decides what to do.
Adds GET /api/settings/library-validation (most recent report) and
POST /api/settings/library-validation/run (queue a fresh sweep).
Cached under cache.library_validation_report.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After gallery-dl returns, parse stdout for written file paths and run
the magic-byte validator on each. Files that fail are moved to
{download_path}/_quarantine/{subscription}/{platform}/ with a sidecar
JSON capturing the source URL, validation reason, and original path —
enough for an operator to redownload-from-source or delete.
Adds ErrorType.VALIDATION_FAILED. A successful gallery-dl run that
produced quarantined files is now a soft failure (success=False,
error_message names the dominant failure reason and count) so the
source's error_count ticks up and the dashboard surfaces it. Gated
by download.validate_files setting (default True).
files_quarantined and quarantined_paths are persisted into download
metadata for the UI/API to consume.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detects truncated/incomplete downloads by checking format-specific
head and tail bytes (JPEG SOI/EOI, PNG signature/IEND, GIF header/3B,
WEBP RIFF size). Catches the production failure mode where a file
landed missing its 2-byte JPEG EOI marker and PIL.ImageFile.load
raised "image file is truncated (6 bytes not processed)" downstream.
Reads only 16 bytes from each end — O(1) per file. Unknown extensions
(JSON sidecars, ugoira zips, etc.) pass through. Hooks into the
download flow in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a Patreon source loses tier access, gallery-dl emits "Not allowed to
view post N" warnings for every post AND yt-dlp tries to fetch HLS manifests
which also 403. The ytdl error text contained "HTTP Error 403: Forbidden",
which the classifier's ACCESS_DENIED_PATTERNS matched against the full
combined stdout+stderr — so the run was labeled ACCESS_DENIED instead of
TIER_LIMITED.
Two fixes in one edit, because they're interlocked:
- Compute per-item vs source-level error lines upfront. has_actual_error
now reflects only source-level errors (previously the per-item exclusion
was gated on skip evidence, which tier-limited runs don't produce since
no content was accessed).
- Strip per-item error lines from `combined` before downstream pattern
matching so noise from recovered per-item failures doesn't latch onto
ACCESS_DENIED / HTTP_ERROR / NOT_FOUND classifiers.
Regression test: tier-gated Patreon run with yt-dlp HLS 403s → TIER_LIMITED,
not ACCESS_DENIED.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Classify Patreon "Not allowed to view post" warnings as TIER_LIMITED so
subscription-gated runs are distinguished from genuine failures, and persist
error_type/message on completed runs so the distinction survives to the UI.
- Fold PLATFORM_DEFAULTS into _get_default_config so gallery-dl.conf is a
complete, editable document; _build_config_for_source now preserves the
user's conf and only re-seeds missing platform sections.
- Add Reset-to-Defaults action in Settings (vs. Revert Changes which just
reloads disk); show info banner when no gallery-dl.conf exists yet.
- Fix Discord embeds option: must be "all" string, not bool (gallery-dl
iterates the value).
- Install yt-dlp + ffmpeg in Docker images so Patreon/Mux HLS video posts
download instead of logging "Cannot import yt-dlp" and skipping.
- Recognize yt-dlp import failures as per-item errors so they don't mask
TIER_LIMITED/NO_NEW_CONTENT classification.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Dashboard's "Retry All Failed" button used to fetch every non-superseded
failure, then fire one retry HTTP call per row. Two missing guards:
- No per-source dedup — a source with N historical failures enqueued N jobs
for the same URL.
- No "recent success" check — a source that succeeded two hours ago could
still have stale un-superseded failures from last week that got re-queued.
Adds POST /api/downloads/retry-failed-bulk:
• Selects non-superseded failures, ordered (source_id ASC, created_at DESC)
• Per-source dedup keeps only the most recent failure per source
• Skips sources whose last_success is within the configurable recent window
(default 24h, via recent_window_hours body param)
• Resets + enqueues in a single DB transaction, single Celery dispatch loop
• Returns a per-reason skip breakdown: duplicate_source, recent_success,
no_source (orphaned rows)
The Dashboard button now calls the bulk endpoint and surfaces the full
skip breakdown in the toast, so the user knows exactly what was (and
wasn't) queued. Logic is extracted into a pure _plan_bulk_retry helper
and unit-tested against SimpleNamespace fixtures — 7 new tests covering
the dedup, window, and orphan paths.
Backend foundation for the upcoming Downloads modal refresh. Adds three
helpers on GalleryDLService:
_compute_run_stats exit_code + downloaded/skipped/per-item-failure/
warning counts, derived from stdout+stderr
_extract_errors_warnings filters stderr to just [error]/[warning] lines
so the modal can show a concise errors-only view
_truncate_log caps a log string (default 500KB) with head+tail+
elision marker, preventing verbose multi-hour runs
from bloating the download_sessions JSONB
Wires all three into download.metadata_ in the download task. No schema
change — everything nests into the existing metadata JSONB column.
Follow-up PR will consume these fields in the Downloads.vue modal.
gallery-dl logs `[download][error] Failed to download <file>.part` when a
single post attachment's media URL expires or is deleted, then recovers
and proceeds with the rest of the run. Our error categorizer was letting
a urllib3 debug line like `"HEAD /media-u/v3/<id> HTTP/1.1" 404 0` match
the NOT_FOUND_PATTERNS list, flagging the whole source as deleted even
though gallery-dl kept downloading subsequent items successfully.
Adds "failed to download" to the per-item allowlist so the mask flips
when skip activity is present, matching the shape of the campaign-ID
fix. Total-failure runs still classify as failures.
Also renames the Download Details modal's "Error Log" panel to
"Verbose Log (stderr)" — gallery-dl's `-v` flag sends all logging
(including [debug] lines) to stderr, so the original label was
misleading. Drops the blanket red text color for the same reason.
When gallery-dl logs "[patreon][error] Failed to extract campaign ID"
but then recovers via its /cw/<vanity> fallback and skips every file
(all already archived), _categorize_error was flagging the [error] line
as fatal and returning UNKNOWN_ERROR. Add "failed to extract campaign
id" to the per-item error allowlist alongside "not allowed to view" and
"unable to get post" so skip detection proceeds and the run is
classified NO_NEW_CONTENT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The anyio_backend fixture is only consumed by pytest-anyio, which is not a
dependency of this project (we use pytest-asyncio with asyncio_mode=auto).
Also drop the unused pytest import from test_sanity.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusen