bvandeusen 66ff671f09 fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos ...

Operator-flagged 2026-06-01 (DaferQ patreon, event #38919). Video posts
hosted on Mux carry a JWT that encodes a playback restriction policy
(`playback_restriction_id` in the token). The token signature alone is
not sufficient — Mux's CDN ALSO checks Referer/Origin on every fetch.

gallery-dl's HEAD probe to `stream.mux.com/<id>.m3u8?token=...` returns
200 (token is valid, HEAD sends no Referer that Mux's policy rejects),
but when yt-dlp follows up with the actual manifest GET it sends its own
default Referer and Mux 403s. yt-dlp retries 4 times, gives up, the
single video fails — every other image in the same post still downloads.

Static `downloader.ytdl.raw-options.http_headers` block now pins Referer
and Origin to `https://www.patreon.com` so yt-dlp's manifest fetch
clears the policy check.

Headers-only restrictions are now handled. Mux IP-range restrictions
(if a creator opts into them) remain unfixable from our worker — those
would need a Patreon-region IP. Per-video PARTIAL classifier (shipped
in plan #544 last commit) already handles the residual case: a run that
grabs all images and fails 1 video classifies as status=ok rather than
flipping the whole event red.

2026-06-01 19:00:50 -04:00

..

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

conftest.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_aliases.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_credentials.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

feat(extension): probe shows current state before click; v1.0.6

2026-06-01 00:41:17 -04:00

test_api_gallery.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_provenance.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_series.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings_downloader.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

feat(downloads): bulk retry respects cooldown; single-source RETRY overrides

2026-05-30 23:15:07 -04:00

test_api_sources.py

feat(download): tick/backfill modes + partial-success classifier (plan #544)

2026-06-01 18:23:28 -04:00

test_api_suggestions.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_system_activity.py

fix(tests): update scheduler-status shape pins for platform_cooldowns

2026-05-30 11:25:36 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_thumbnails.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_archive_extractor.py

feat(attachments): fail-soft archive_extractor (zip/cbz/rar/7z)

2026-05-19 11:11:37 -04:00

test_artist_directory_service.py

fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService

2026-05-21 22:14:15 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

feat(thumb-backfill): backfill_thumbnails planner task — keyset-paginates ImageRecord, NULLs bad thumb paths, enqueues generate_thumbnail for NULL/missing/corrupt

2026-05-26 10:36:21 -04:00

test_backup_service.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)

2026-05-25 01:28:54 -04:00

test_credential_crypto.py

fix(fc3b): ruff I001 (blank-line) + F841 (unused crypto binding)

2026-05-20 18:46:59 -04:00

test_credential_service.py

fix(hentaifoundry): inject host-only PHPSESSID/CSRF duplicates + extension preserves browser hostOnly

2026-05-27 19:12:51 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

feat(download): tick/backfill modes + partial-success classifier (plan #544)

2026-06-01 18:23:28 -04:00

test_download_source_task.py

fix(fc3c): explicit import of download task module in registration test

2026-05-20 21:20:30 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_gallery_dl_service.py

fix(downloads): forward Patreon Referer/Origin to yt-dlp for Mux videos

2026-06-01 19:00:50 -04:00

test_gallery_provenance_filter.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_gallery_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(provenance): nh3 HTML sanitizer for scraped post descriptions

2026-05-18 18:05:45 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

fix(test): archive members need structured imgs+threshold0; provenance payload gained attachments key

2026-05-19 11:22:17 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download'

2026-05-20 22:01:26 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_importer_upsert_helpers.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_maintenance.py

fix(scan): recovery sweep for stranded download events

2026-05-29 21:40:52 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(fc2b): schema migration 0003 — ML pipeline tables

2026-05-15 07:35:22 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0008.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

feat(fc2b): add AliasService — (name, category) -> canonical tag

2026-05-15 07:37:13 -04:00

test_ml_allowlist.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_ml_artist_retired.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

fix(test): unpin general-threshold test from old 0.95 default (alembic 0029)

2026-06-01 02:18:54 -04:00

test_ml_tagger.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_paths.py

fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig)

2026-05-14 14:24:25 -04:00

test_patreon_resolver.py

feat(fc3c): PatreonResolver — vanity → campaign_id via requests

2026-05-20 20:39:50 -04:00

test_phash_dedup.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_provenance_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_safe_probe.py

fix(import): archive probe → subprocess (was multiprocessing.Process)

2026-05-30 21:48:51 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_model.py

feat(series): SeriesPage model + 0005 migration (unique image, ordered pages)

2026-05-16 23:33:12 -04:00

test_series_service.py

feat(series): list_pages returns full image_url for the reader

2026-05-17 16:28:20 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_sidecar_util.py

fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms

2026-05-27 15:35:25 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

feat(download): tick/backfill modes + partial-success classifier (plan #544)

2026-06-01 18:23:28 -04:00

test_suggestions_bulk.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): TagService.merge repoints series_page.series_tag_id

2026-05-16 23:36:04 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(fc2a): align CI with FabledRulebook — lint + short unit tests only

2026-05-14 15:23:07 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)

2026-05-25 01:28:54 -04:00

test_tasks_backup.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_tasks_library_audit.py

fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:32:50 -04:00

test_tasks_ml.py

feat(fc2b): add allowlist-apply + centroid recompute tasks + beat

2026-05-15 07:43:39 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00