bvandeusen 4272a19d40 fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883) ...

The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL
wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP
hosts (requests' timeout is the idle gap between bytes, never a total). So a
stalled HTTP connection tied up a download-worker slot AND the per-host
serialize lock for ~50 min before failing (operator-flagged 2026-06-17).

Split into two limits in external_fetch:
- read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets
  (connect, read); a stalled socket now fails in ~60s.
- total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline
  across chunks in _stream_to_file (HTTP has no total-download timeout), and
  passed as the subprocess total for mega.
fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive
(gdown) self-manages; the celery hard limit is the outer backstop.

Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it
can't wedge a host's links much past one fetch's budget.

Each external link is already one Celery task (sweep enqueues one
fetch_external_link.delay per link), so these budgets are per-link.

Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read);
mega gets the total. Worker fakes updated to **kwargs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-16 21:15:49 -04:00

..

fixtures

feat(patreon): native JSON-API client — ingester build step 1 (plan #697)

2026-06-05 19:16:20 -04:00

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

_prediction_helpers.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

conftest.py

perf(ci): reuse a session-scoped engine for the per-test DB reset

2026-06-04 07:35:28 -04:00

test_api_admin.py

feat(tags): 'Reset content tagging' admin action

2026-06-04 16:47:36 -04:00

test_api_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

fix(cleanup): library scans survive navigation, reconnect on return

2026-06-04 16:55:20 -04:00

test_api_credentials.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

feat(extension): probe shows current state before click; v1.0.6

2026-06-01 00:41:17 -04:00

test_api_gallery.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)

2026-06-16 11:07:00 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(browse): sticky tabs + per-tab search bar (server-side, scope-aware)

2026-06-12 00:04:06 -04:00

test_api_provenance.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_series.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_api_settings_downloader.py

feat(external): per-host enable toggles in Settings (Phase 4d)

2026-06-14 15:57:42 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

feat(downloads): bulk retry respects cooldown; single-source RETRY overrides

2026-05-30 23:15:07 -04:00

test_api_sources.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_api_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_system_activity.py

feat(activity): search/filter on both Activity-tab panes

2026-06-10 11:34:42 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

feat(gallery,tags): clear active filters

2026-06-03 23:26:04 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_api_thumbnails.py

fix(thumbnails): surface backfill results + tighten validity check

2026-06-01 21:57:29 -04:00

test_archive_extractor.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_artist_directory_service.py

test(artist-dir): deterministic sha256 in _seed_image (fix flaky uq collision)

2026-06-16 20:30:31 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_artist_visit.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

fix(test): include 'scanned' in all backfill result assertions

2026-06-01 22:09:02 -04:00

test_backup_service.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_routing.py

fix(activity): record external.* TaskRun.queue as download, not default (#883)

2026-06-16 21:05:26 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)

2026-06-16 15:20:35 -04:00

test_credential_crypto.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_credential_service.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_db_helpers.py

refactor(services): shared race-safe get_or_create helper (DRY backend sweep)

2026-06-09 23:46:01 -04:00

test_download_backends.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)

2026-06-14 23:27:38 -04:00

test_download_source_task.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_external_fetch.py

fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)

2026-06-16 21:15:49 -04:00

test_external_worker.py

fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883)

2026-06-16 21:15:49 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

test_gallery_dl_service.py

feat(patreon): structured ingester results + quarantine surfacing — #704 step 1

2026-06-05 23:39:28 -04:00

test_gallery_facets.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_gallery_provenance_filter.py

feat(gallery): composable scroll filter (multi-tag AND, media, sort)

2026-06-03 23:50:19 -04:00

test_gallery_service.py

fix(images): percent-encode original-image URLs ('#' in paths 404'd)

2026-06-12 00:36:44 -04:00

test_gallery_similar.py

feat(gallery): visual 'more like this' search (Phase 3 backend)

2026-06-04 08:47:43 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_image_prediction.py

feat(ml): image_prediction table + backfill + dual-write (#768 step 1)

2026-06-10 15:55:32 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

feat(import): recurse nested archives + precise "no images" reason (#718)

2026-06-15 21:18:02 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

feat(import): Tier-1 video near-dup by duration+aspect (#871)

2026-06-15 22:17:36 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_importer_upsert_helpers.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_link_extract.py

feat(posts): extract + record external file-host links (Phase 3)

2026-06-14 13:15:36 -04:00

test_maintenance.py

fix(maint): raise recovery-sweep threshold for fetch_external_link (#883)

2026-06-16 20:54:39 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0008.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_allowlist.py

feat(ml): clamp allowlist min_confidence to the tagger store floor

2026-06-10 13:52:20 -04:00

test_ml_artist_retired.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_tag_name.py

feat(ml): normalize Camie suggestion names to human-readable

2026-06-03 13:00:08 -04:00

test_ml_tagger.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_page_number_parser.py

feat(series): post→series flows + browse list — backend (FC-6.2)

2026-06-07 18:34:11 -04:00

test_paths.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_patreon_client.py

fix(ingest): skip tier-gated Patreon posts entirely (#874)

2026-06-16 14:43:00 -04:00

test_patreon_contract.py

fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)

2026-06-15 00:14:52 -04:00

test_patreon_downloader.py

test(downloader): fix test_sidecar_written_and_findable for post-first minimal sidecar (#856)

2026-06-15 10:15:04 -04:00

test_patreon_ingester.py

test(ingest): repair gated-post tests (#874 CI bounce)

2026-06-16 14:48:51 -04:00

test_patreon_resolver.py

feat(patreon): resolve the creator's campaign from a single-post URL

2026-06-06 21:32:15 -04:00

test_patreon_seen_media.py

feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)

2026-06-05 19:26:57 -04:00

test_phash_dedup.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platform_lock.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_prosemirror.py

fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)

2026-06-15 00:14:52 -04:00

test_provenance_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_reextract_archives.py

test(maintenance): patterned images for re-extract resume test

2026-06-07 12:08:02 -04:00

test_safe_probe.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_chapters.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_from_post.py

test(series): assert group start_page, not per-page stated_page

2026-06-11 23:17:14 -04:00

test_series_match_service.py

feat(series): pending staging for add-from-post (#789 Phase 2)

2026-06-11 21:47:58 -04:00

test_series_match.py

feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3)

2026-06-07 18:58:18 -04:00

test_series_model.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_pending.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_series_service.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_sidecar_util.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_suggestions_bulk.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_tag_normalize.py

fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)

2026-06-06 23:57:06 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_query.py

refactor(tags): shared tag_query for fandom self-join + serialization (DRY sweep)

2026-06-10 00:19:17 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

feat(maintenance): retroactive video-dedup action — preview + apply (#871)

2026-06-16 08:31:50 -04:00

test_tasks_backup.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_tasks_library_audit.py

fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)

2026-06-07 00:08:19 -04:00

test_tasks_ml.py

feat(ml): cadence-based video frame sampling + min-frame tag aggregation (#747)

2026-06-16 11:07:00 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00