After gallery-dl returns, parse stdout for written file paths and run
the magic-byte validator on each. Files that fail are moved to
{download_path}/_quarantine/{subscription}/{platform}/ with a sidecar
JSON capturing the source URL, validation reason, and original path —
enough for an operator to redownload-from-source or delete.
Adds ErrorType.VALIDATION_FAILED. A successful gallery-dl run that
produced quarantined files is now a soft failure (success=False,
error_message names the dominant failure reason and count) so the
source's error_count ticks up and the dashboard surfaces it. Gated
by download.validate_files setting (default True).
files_quarantined and quarantined_paths are persisted into download
metadata for the UI/API to consume.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detects truncated/incomplete downloads by checking format-specific
head and tail bytes (JPEG SOI/EOI, PNG signature/IEND, GIF header/3B,
WEBP RIFF size). Catches the production failure mode where a file
landed missing its 2-byte JPEG EOI marker and PIL.ImageFile.load
raised "image file is truncated (6 bytes not processed)" downstream.
Reads only 16 bytes from each end — O(1) per file. Unknown extensions
(JSON sidecars, ugoira zips, etc.) pass through. Hooks into the
download flow in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a Patreon source loses tier access, gallery-dl emits "Not allowed to
view post N" warnings for every post AND yt-dlp tries to fetch HLS manifests
which also 403. The ytdl error text contained "HTTP Error 403: Forbidden",
which the classifier's ACCESS_DENIED_PATTERNS matched against the full
combined stdout+stderr — so the run was labeled ACCESS_DENIED instead of
TIER_LIMITED.
Two fixes in one edit, because they're interlocked:
- Compute per-item vs source-level error lines upfront. has_actual_error
now reflects only source-level errors (previously the per-item exclusion
was gated on skip evidence, which tier-limited runs don't produce since
no content was accessed).
- Strip per-item error lines from `combined` before downstream pattern
matching so noise from recovered per-item failures doesn't latch onto
ACCESS_DENIED / HTTP_ERROR / NOT_FOUND classifiers.
Regression test: tier-gated Patreon run with yt-dlp HLS 403s → TIER_LIMITED,
not ACCESS_DENIED.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Classify Patreon "Not allowed to view post" warnings as TIER_LIMITED so
subscription-gated runs are distinguished from genuine failures, and persist
error_type/message on completed runs so the distinction survives to the UI.
- Fold PLATFORM_DEFAULTS into _get_default_config so gallery-dl.conf is a
complete, editable document; _build_config_for_source now preserves the
user's conf and only re-seeds missing platform sections.
- Add Reset-to-Defaults action in Settings (vs. Revert Changes which just
reloads disk); show info banner when no gallery-dl.conf exists yet.
- Fix Discord embeds option: must be "all" string, not bool (gallery-dl
iterates the value).
- Install yt-dlp + ffmpeg in Docker images so Patreon/Mux HLS video posts
download instead of logging "Cannot import yt-dlp" and skipping.
- Recognize yt-dlp import failures as per-item errors so they don't mask
TIER_LIMITED/NO_NEW_CONTENT classification.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backend foundation for the upcoming Downloads modal refresh. Adds three
helpers on GalleryDLService:
_compute_run_stats exit_code + downloaded/skipped/per-item-failure/
warning counts, derived from stdout+stderr
_extract_errors_warnings filters stderr to just [error]/[warning] lines
so the modal can show a concise errors-only view
_truncate_log caps a log string (default 500KB) with head+tail+
elision marker, preventing verbose multi-hour runs
from bloating the download_sessions JSONB
Wires all three into download.metadata_ in the download task. No schema
change — everything nests into the existing metadata JSONB column.
Follow-up PR will consume these fields in the Downloads.vue modal.
gallery-dl logs `[download][error] Failed to download <file>.part` when a
single post attachment's media URL expires or is deleted, then recovers
and proceeds with the rest of the run. Our error categorizer was letting
a urllib3 debug line like `"HEAD /media-u/v3/<id> HTTP/1.1" 404 0` match
the NOT_FOUND_PATTERNS list, flagging the whole source as deleted even
though gallery-dl kept downloading subsequent items successfully.
Adds "failed to download" to the per-item allowlist so the mask flips
when skip activity is present, matching the shape of the campaign-ID
fix. Total-failure runs still classify as failures.
Also renames the Download Details modal's "Error Log" panel to
"Verbose Log (stderr)" — gallery-dl's `-v` flag sends all logging
(including [debug] lines) to stderr, so the original label was
misleading. Drops the blanket red text color for the same reason.
When gallery-dl logs "[patreon][error] Failed to extract campaign ID"
but then recovers via its /cw/<vanity> fallback and skips every file
(all already archived), _categorize_error was flagging the [error] line
as fatal and returning UNKNOWN_ERROR. Add "failed to extract campaign
id" to the per-item error allowlist alongside "not allowed to view" and
"unable to get post" so skip detection proceeds and the run is
classified NO_NEW_CONTENT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Added missing 'from sqlalchemy.ext.asyncio import AsyncSession' import in downloads.py
This fixes the NameError at module load time that was breaking all download tasks
- Replaced regex patterns with plain-string equivalents in NOT_FOUND_PATTERNS:
* 'account.*deleted' → 'account deleted' (. and * are literal in substring search)
* 'profile.*not.*exist' → 'profile does not exist' (same issue)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bvandeusen
Bryan Van Deusen