Walks the downloads tree and reports files that fail magic-byte
validation — the "how many pre-existing files in my library are
silently truncated?" question, answered. Skips the _quarantine
subtree (already known-bad) and caps suspect_paths at 500 with a
truncated flag so the persisted report can't grow unbounded.
On-demand only: a full filesystem walk on a NAS-mounted library is
expensive and the user should choose when to pay for it. Pre-existing
files are reported, not quarantined — they may be the only copy and
the user decides what to do.
Adds GET /api/settings/library-validation (most recent report) and
POST /api/settings/library-validation/run (queue a fresh sweep).
Cached under cache.library_validation_report.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After gallery-dl returns, parse stdout for written file paths and run
the magic-byte validator on each. Files that fail are moved to
{download_path}/_quarantine/{subscription}/{platform}/ with a sidecar
JSON capturing the source URL, validation reason, and original path —
enough for an operator to redownload-from-source or delete.
Adds ErrorType.VALIDATION_FAILED. A successful gallery-dl run that
produced quarantined files is now a soft failure (success=False,
error_message names the dominant failure reason and count) so the
source's error_count ticks up and the dashboard surfaces it. Gated
by download.validate_files setting (default True).
files_quarantined and quarantined_paths are persisted into download
metadata for the UI/API to consume.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the four stat cards with a compact three-card strip: 7-day activity
with inline SVG sparkline (new ActivitySparkline), Running Now / Next Check
with per-source countdown list, and System with credential health + disk
usage bar.
- Add GET /downloads/activity-timeline — per-day completed/failed/files
counts, pre-filled with zero buckets so the sparkline always has N points.
- Report filesystem-level usage via shutil.disk_usage in storage rollup,
plus a live fallback in GET /settings so the capacity bar works before
the first Celery rollup runs and for cached rows that predate the field.
- Extract Download Details into a reusable DownloadDetailsModal component
and wire Recent Activity rows to open it (previously Downloads-page only).
- Compute next scheduled check per source from global schedule_interval;
surface credential expiration/missing alerts scoped to platforms that
actually have enabled sources.
- Two-speed polling (5s when downloads are active, 30s idle) with Page
Visibility awareness so background tabs don't churn the API.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Classify Patreon "Not allowed to view post" warnings as TIER_LIMITED so
subscription-gated runs are distinguished from genuine failures, and persist
error_type/message on completed runs so the distinction survives to the UI.
- Fold PLATFORM_DEFAULTS into _get_default_config so gallery-dl.conf is a
complete, editable document; _build_config_for_source now preserves the
user's conf and only re-seeds missing platform sections.
- Add Reset-to-Defaults action in Settings (vs. Revert Changes which just
reloads disk); show info banner when no gallery-dl.conf exists yet.
- Fix Discord embeds option: must be "all" string, not bool (gallery-dl
iterates the value).
- Install yt-dlp + ffmpeg in Docker images so Patreon/Mux HLS video posts
download instead of logging "Cannot import yt-dlp" and skipping.
- Recognize yt-dlp import failures as per-item errors so they don't mask
TIER_LIMITED/NO_NEW_CONTENT classification.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backend foundation for the upcoming Downloads modal refresh. Adds three
helpers on GalleryDLService:
_compute_run_stats exit_code + downloaded/skipped/per-item-failure/
warning counts, derived from stdout+stderr
_extract_errors_warnings filters stderr to just [error]/[warning] lines
so the modal can show a concise errors-only view
_truncate_log caps a log string (default 500KB) with head+tail+
elision marker, preventing verbose multi-hour runs
from bloating the download_sessions JSONB
Wires all three into download.metadata_ in the download task. No schema
change — everything nests into the existing metadata JSONB column.
Follow-up PR will consume these fields in the Downloads.vue modal.
Converts the Source Health last-check column and the Downloads created-at
column from locale date strings to human-relative times ("6 hours ago").
The absolute timestamp is preserved as a native browser tooltip so full
precision is still one hover away; detail modals keep the absolute form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Added missing 'from sqlalchemy.ext.asyncio import AsyncSession' import in downloads.py
This fixes the NameError at module load time that was breaking all download tasks
- Replaced regex patterns with plain-string equivalents in NOT_FOUND_PATTERNS:
* 'account.*deleted' → 'account deleted' (. and * are literal in substring search)
* 'profile.*not.*exist' → 'profile does not exist' (same issue)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract identical get_async_session() and cleanup_engine() functions
from downloads.py and maintenance.py into a shared db.py module.
This eliminates code duplication and improves maintainability.
db.py imports only from app.config and SQLAlchemy to avoid circular
dependencies with the task modules that import from it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bvandeusen
Bryan Van Deusen