bvandeusen 87c7318125 feat(downloads): serialize same-platform downloads (Patreon concurrency cap) ...

Two concurrent Patreon walks could trip the server rate limit even with each
source pacing its own requests. The platform-cooldown handled the aftermath of
a 429; this adds the preventive half — a per-platform Redis lock so only one
Patreon walk runs at a time. Different platforms still run concurrently up to
the worker concurrency; only a second walk on the SAME serialized platform
waits.

download_source acquires fc:download_lock:<platform> (non-blocking) before the
run. On contention it re-enqueues itself with a short countdown (the pending
event stays — no new event, no log spam), bounded to ~15 min then runs uncapped
as a safety valve. The lock TTL sits just past the hard kill so a SIGKILL'd
worker auto-releases; a backfill chunk only holds it ~10 min, well under the
30-min DownloadEvent recovery sweep. A broker hiccup degrades to uncapped
(prior behaviour) rather than stalling downloads. SERIALIZED_PLATFORMS={patreon};
gallery-dl platforms are left uncapped (self-pacing subprocesses).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-06 21:22:32 -04:00

..

fixtures

feat(patreon): native JSON-API client — ingester build step 1 (plan #697)

2026-06-05 19:16:20 -04:00

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

conftest.py

perf(ci): reuse a session-scoped engine for the per-test DB reset

2026-06-04 07:35:28 -04:00

test_api_admin.py

feat(tags): 'Reset content tagging' admin action

2026-06-04 16:47:36 -04:00

test_api_aliases.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

fix(cleanup): library scans survive navigation, reconnect on return

2026-06-04 16:55:20 -04:00

test_api_credentials.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

feat(extension): probe shows current state before click; v1.0.6

2026-06-01 00:41:17 -04:00

test_api_gallery.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

fix(test): default suggestion_threshold_general now 0.70 (alembic 0033)

2026-06-02 18:49:24 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_provenance.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_series.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings_downloader.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

feat(downloads): bulk retry respects cooldown; single-source RETRY overrides

2026-05-30 23:15:07 -04:00

test_api_sources.py

feat(sources): pre-flight credential verify on backfill/recovery arm — #703 step 2

2026-06-05 23:21:05 -04:00

test_api_suggestions.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_system_activity.py

fix(tests): update scheduler-status shape pins for platform_cooldowns

2026-05-30 11:25:36 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

feat(gallery,tags): clear active filters

2026-06-03 23:26:04 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_api_thumbnails.py

fix(thumbnails): surface backfill results + tighten validity check

2026-06-01 21:57:29 -04:00

test_archive_extractor.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_artist_directory_service.py

fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService

2026-05-21 22:14:15 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_artist_visit.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

fix(test): include 'scanned' in all backfill result assertions

2026-06-01 22:09:02 -04:00

test_backup_service.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

feat(tags): 'Reset content tagging' admin action

2026-06-04 16:47:36 -04:00

test_credential_crypto.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_credential_service.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_download_backends.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708)

2026-06-06 11:34:37 -04:00

test_download_source_task.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

test_gallery_dl_service.py

feat(patreon): structured ingester results + quarantine surfacing — #704 step 1

2026-06-05 23:39:28 -04:00

test_gallery_facets.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_gallery_provenance_filter.py

feat(gallery): composable scroll filter (multi-tag AND, media, sort)

2026-06-03 23:50:19 -04:00

test_gallery_service.py

feat(gallery): composable scroll filter (multi-tag AND, media, sort)

2026-06-03 23:50:19 -04:00

test_gallery_similar.py

feat(gallery): visual 'more like this' search (Phase 3 backend)

2026-06-04 08:47:43 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(provenance): nh3 HTML sanitizer for scraped post descriptions

2026-05-18 18:05:45 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

fix(test): archive members need structured imgs+threshold0; provenance payload gained attachments key

2026-05-19 11:22:17 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount

2026-06-02 08:16:13 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_importer_upsert_helpers.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_maintenance.py

feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout

2026-06-04 00:49:39 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(fc2b): schema migration 0003 — ML pipeline tables

2026-05-15 07:35:22 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0008.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

feat(fc2b): add AliasService — (name, category) -> canonical tag

2026-05-15 07:37:13 -04:00

test_ml_allowlist.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_ml_artist_retired.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

feat(ml): normalize Camie suggestion names to human-readable

2026-06-03 13:00:08 -04:00

test_ml_tag_name.py

feat(ml): normalize Camie suggestion names to human-readable

2026-06-03 13:00:08 -04:00

test_ml_tagger.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_paths.py

fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig)

2026-05-14 14:24:25 -04:00

test_patreon_client.py

feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708)

2026-06-06 11:34:37 -04:00

test_patreon_contract.py

feat(patreon): drift detection + error categorization — build step 4 (plan #697)

2026-06-05 21:56:58 -04:00

test_patreon_downloader.py

test(patreon): fix _BoomSession stub for the B5 headers kwarg

2026-06-06 11:41:48 -04:00

test_patreon_ingester.py

feat(downloads): live per-file progress on running events — #709

2026-06-06 15:48:31 -04:00

test_patreon_resolver.py

fix(patreon): handle the /cw/ creator-URL prefix in vanity extraction

2026-06-06 20:34:15 -04:00

test_patreon_seen_media.py

feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)

2026-06-05 19:26:57 -04:00

test_phash_dedup.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platform_lock.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_provenance_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_reextract_archives.py

feat(maintenance): re-extract archive attachments + link to post — #713 part 2

2026-06-06 14:43:26 -04:00

test_safe_probe.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_model.py

feat(series): SeriesPage model + 0005 migration (unique image, ordered pages)

2026-05-16 23:33:12 -04:00

test_series_service.py

feat(series): list_pages returns full image_url for the reader

2026-05-17 16:28:20 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

test_sidecar_util.py

fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms

2026-05-27 15:35:25 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

feat(subscriptions): live posts-processed progress on backfill/recovery — #704 step 2

2026-06-05 23:43:23 -04:00

test_suggestions_bulk.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): TagService.merge repoints series_page.series_tag_id

2026-05-16 23:36:04 -04:00

test_tag_normalize.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)

2026-05-25 01:28:54 -04:00

test_tasks_backup.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_tasks_library_audit.py

fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:32:50 -04:00

test_tasks_ml.py

feat(fc2b): add allowlist-apply + centroid recompute tasks + beat

2026-05-15 07:43:39 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00