bvandeusen 77f7a23410 feat(scheduler): order due sources by last_checked_at — most overdue first ...

select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:

> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.

Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.

Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-30 12:08:43 -04:00

..

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

conftest.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_aliases.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_credentials.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_gallery.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(posts): in-context anchored feed with bidirectional infinite scroll

2026-05-28 21:12:18 -04:00

test_api_provenance.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_series.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings_downloader.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources.py

fix(tests): update scheduler-status shape pins for platform_cooldowns

2026-05-30 11:25:36 -04:00

test_api_suggestions.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_system_activity.py

fix(tests): update scheduler-status shape pins for platform_cooldowns

2026-05-30 11:25:36 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_thumbnails.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_archive_extractor.py

feat(attachments): fail-soft archive_extractor (zip/cbz/rar/7z)

2026-05-19 11:11:37 -04:00

test_artist_directory_service.py

fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService

2026-05-21 22:14:15 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

fix(test): artist_service _fixture sets artist_id (post-vii-c ArtistService reads it)

2026-05-18 21:47:45 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

feat(thumb-backfill): backfill_thumbnails planner task — keyset-paginates ImageRecord, NULLs bad thumb paths, enqueues generate_thumbnail for NULL/missing/corrupt

2026-05-26 10:36:21 -04:00

test_backup_service.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)

2026-05-25 01:28:54 -04:00

test_credential_crypto.py

fix(fc3b): ruff I001 (blank-line) + F841 (unused crypto binding)

2026-05-20 18:46:59 -04:00

test_credential_service.py

fix(hentaifoundry): inject host-only PHPSESSID/CSRF duplicates + extension preserves browser hostOnly

2026-05-27 19:12:51 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

fc3d: DownloadService finalize hook — Source.consecutive_failures/last_error/last_checked_at

2026-05-21 16:00:40 -04:00

test_download_source_task.py

fix(fc3c): explicit import of download task module in registration test

2026-05-20 21:20:30 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_gallery_dl_service.py

feat(fc3c): GalleryDLService — subprocess wrap + error classification + validate/quarantine

2026-05-20 20:41:42 -04:00

test_gallery_provenance_filter.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_gallery_service.py

feat(gallery): sort/group/jump by COALESCE(post.post_date, image_record.created_at) — surface migrated content at its original publish date, not FC scan date

2026-05-25 12:30:18 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(provenance): nh3 HTML sanitizer for scraped post descriptions

2026-05-18 18:05:45 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

fix(test): archive members need structured imgs+threshold0; provenance payload gained attachments key

2026-05-19 11:22:17 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download'

2026-05-20 22:01:26 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(test): drop unused Post binding in test_importer_provenance_race (ruff F841)

2026-05-26 14:39:57 -04:00

test_importer_upsert_helpers.py

fix(importer): Source = one per (artist, platform), not one per post — filesystem importer's sidecar paths now reuse the artist's existing subscription Source (or create one synthetic anchor with enabled=False) instead of fabricating a new Source per post URL. Alembic 0022 consolidates existing per-post Sources to canonical (prefers campaign URL; falls back to sidecar:<platform>:<slug>) and re-parents Posts + ImageProvenance, merging Post collisions.

2026-05-26 14:19:29 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_maintenance.py

fix(scan): recovery sweep for stranded download events

2026-05-29 21:40:52 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(fc2b): schema migration 0003 — ML pipeline tables

2026-05-15 07:35:22 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(post): add nullable description + attachment_count (migration 0007)

2026-05-18 09:38:43 -04:00

test_migration_0008.py

feat(provenance): image_record.artist_id + 0008 backfill/drop-artist-tags

2026-05-18 21:33:11 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

feat(fc2b): add AliasService — (name, category) -> canonical tag

2026-05-15 07:37:13 -04:00

test_ml_allowlist.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_ml_artist_retired.py

feat(provenance): ML stops surfacing the artist category

2026-05-18 21:36:43 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

feat(fc2b): add SuggestionService — alias-resolved, threshold-filtered, ranked

2026-05-15 07:38:33 -04:00

test_ml_tagger.py

fix(ml): align tagger + downloader with Camie v2 actual layout (model.onnx -> camie-tagger-v2.onnx + JSON metadata + ImageNet preprocessing + sigmoid on refined output)

2026-05-25 02:25:30 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_paths.py

fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig)

2026-05-14 14:24:25 -04:00

test_patreon_resolver.py

feat(fc3c): PatreonResolver — vanity → campaign_id via requests

2026-05-20 20:39:50 -04:00

test_phash_dedup.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

fc3e: PostFeedService — cursor-paginated unified posts feed mirroring GalleryService.scroll

2026-05-21 18:54:49 -04:00

test_provenance_service.py

fix(test): archive members need structured imgs+threshold0; provenance payload gained attachments key

2026-05-19 11:22:17 -04:00

test_safe_probe.py

feat(import-resilience L3): subprocess-isolated probes for video + archive

2026-05-28 00:01:32 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_model.py

feat(series): SeriesPage model + 0005 migration (unique image, ordered pages)

2026-05-16 23:33:12 -04:00

test_series_service.py

feat(series): list_pages returns full image_url for the reader

2026-05-17 16:28:20 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

fix(test): sidecar idempotency needs phash_threshold=0 (orthogonal splits collapse at default)

2026-05-18 15:13:48 -04:00

test_sidecar_util.py

fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms

2026-05-27 15:35:25 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

fix(fc3b): replace FC-3a KNOWN_PLATFORMS w/ registry; drop /api/sources/platforms

2026-05-20 18:37:17 -04:00

test_suggestions_bulk.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): TagService.merge repoints series_page.series_tag_id

2026-05-16 23:36:04 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(fc2a): align CI with FabledRulebook — lint + short unit tests only

2026-05-14 15:23:07 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)

2026-05-25 01:28:54 -04:00

test_tasks_backup.py

fix(fc3h): split semicolon-stacked statements (E702) and bridge v-dialog v-model to avoid prop write

2026-05-24 23:37:06 -04:00

test_tasks_library_audit.py

fix(fc-cleanup): test sha256 fixtures stay within varchar(64) + isort the registration import — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:32:50 -04:00

test_tasks_ml.py

feat(fc2b): add allowlist-apply + centroid recompute tasks + beat

2026-05-15 07:43:39 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00