bvandeusen 96e984cded feat(external): download worker for file-host links (Phase 4b) ...

tasks/external.py drives the external_link ledger:
- fetch_external_link(link_id): atomic claim (pending/failed→downloading, so a
  duplicate enqueue no-ops), per-host Redis serialize lock (#720 pattern;
  requeue-with-countdown if busy), fetch via external_fetch into the artist
  library tree, then route each file through importer.attach_in_place via a
  synthesized sidecar so it links to the SAME post (archive→ImageRecords,
  else→PostAttachment; on-disk original removed for captured files, art stays);
  thumbnail+ML enqueue for new images; status downloaded | failed | dead with
  attempts/last_error/completed_at/duration.
- sweep_external_links(): enqueue a bounded batch of actionable links.
- recover_external_links() + prune_external_links(): recovery + retention (#89).
- per-host enable read via getattr (forward-compatible; Settings UI adds the
  columns in 4d — defaults on, rule #26).

Wiring: celery include + route (download lane) + beat (sweep 10m, recover +
prune daily); download_service phase 3 enqueues a sweep after recording links.
Integration tests: download+attach, failure, dead-letter, non-claimable, sweep.

mega still needs the MEGAcmd binary in the runtime image (Phase 4c). Refs #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-14 13:44:07 -04:00

..

fixtures

feat(patreon): native JSON-API client — ingester build step 1 (plan #697)

2026-06-05 19:16:20 -04:00

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

_prediction_helpers.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

conftest.py

perf(ci): reuse a session-scoped engine for the per-test DB reset

2026-06-04 07:35:28 -04:00

test_api_admin.py

feat(tags): 'Reset content tagging' admin action

2026-06-04 16:47:36 -04:00

test_api_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

fix(cleanup): library scans survive navigation, reconnect on return

2026-06-04 16:55:20 -04:00

test_api_credentials.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

feat(extension): probe shows current state before click; v1.0.6

2026-06-01 00:41:17 -04:00

test_api_gallery.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(browse): sticky tabs + per-tab search bar (server-side, scope-aware)

2026-06-12 00:04:06 -04:00

test_api_provenance.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_series.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_api_settings_downloader.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

feat(downloads): bulk retry respects cooldown; single-source RETRY overrides

2026-05-30 23:15:07 -04:00

test_api_sources.py

feat(subs): kick off the first backfill walk immediately on source create

2026-06-06 21:22:32 -04:00

test_api_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_system_activity.py

feat(activity): search/filter on both Activity-tab panes

2026-06-10 11:34:42 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

feat(gallery,tags): clear active filters

2026-06-03 23:26:04 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_api_thumbnails.py

fix(thumbnails): surface backfill results + tighten validity check

2026-06-01 21:57:29 -04:00

test_archive_extractor.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_artist_directory_service.py

fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService

2026-05-21 22:14:15 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_artist_visit.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

fix(test): include 'scanned' in all backfill result assertions

2026-06-01 22:09:02 -04:00

test_backup_service.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_routing.py

feat(maintenance): dedicated maintenance_long lane for long one-shot tasks

2026-06-07 09:00:03 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_credential_crypto.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_credential_service.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_db_helpers.py

refactor(services): shared race-safe get_or_create helper (DRY backend sweep)

2026-06-09 23:46:01 -04:00

test_download_backends.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

feat(downloads): platform cooldown honors server Retry-After — B1 (plan #708)

2026-06-06 11:34:37 -04:00

test_download_source_task.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_external_fetch.py

feat(external): file-host fetcher subsystem (Phase 4a)

2026-06-14 13:29:20 -04:00

test_external_worker.py

feat(external): download worker for file-host links (Phase 4b)

2026-06-14 13:44:07 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

test_gallery_dl_service.py

feat(patreon): structured ingester results + quarantine surfacing — #704 step 1

2026-06-05 23:39:28 -04:00

test_gallery_facets.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_gallery_provenance_filter.py

feat(gallery): composable scroll filter (multi-tag AND, media, sort)

2026-06-03 23:50:19 -04:00

test_gallery_service.py

fix(images): percent-encode original-image URLs ('#' in paths 404'd)

2026-06-12 00:36:44 -04:00

test_gallery_similar.py

feat(gallery): visual 'more like this' search (Phase 3 backend)

2026-06-04 08:47:43 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(posts): faithful (semantic) HTML rendering of post bodies

2026-06-14 13:02:21 -04:00

test_image_prediction.py

feat(ml): image_prediction table + backfill + dual-write (#768 step 1)

2026-06-10 15:55:32 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

fix(test): archive members need structured imgs+threshold0; provenance payload gained attachments key

2026-05-19 11:22:17 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

fix(posts): link duplicate items to every post + prune bare shells

2026-06-08 19:28:33 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_importer_upsert_helpers.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_link_extract.py

feat(posts): extract + record external file-host links (Phase 3)

2026-06-14 13:15:36 -04:00

test_maintenance.py

fix(maintenance): download queue needs a sweep threshold above its 25-min time_limit

2026-06-10 11:57:31 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0008.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_allowlist.py

feat(ml): clamp allowlist min_confidence to the tagger store floor

2026-06-10 13:52:20 -04:00

test_ml_artist_retired.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_tag_name.py

feat(ml): normalize Camie suggestion names to human-readable

2026-06-03 13:00:08 -04:00

test_ml_tagger.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_page_number_parser.py

feat(series): post→series flows + browse list — backend (FC-6.2)

2026-06-07 18:34:11 -04:00

test_paths.py

fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig)

2026-05-14 14:24:25 -04:00

test_patreon_client.py

feat(patreon): capture media-less/text-only posts (post-only records)

2026-06-14 12:44:46 -04:00

test_patreon_contract.py

feat(patreon): drift detection + error categorization — build step 4 (plan #697)

2026-06-05 21:56:58 -04:00

test_patreon_downloader.py

feat(patreon): capture media-less/text-only posts (post-only records)

2026-06-14 12:44:46 -04:00

test_patreon_ingester.py

feat(patreon): capture media-less/text-only posts (post-only records)

2026-06-14 12:44:46 -04:00

test_patreon_resolver.py

feat(patreon): resolve the creator's campaign from a single-post URL

2026-06-06 21:32:15 -04:00

test_patreon_seen_media.py

feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)

2026-06-05 19:26:57 -04:00

test_phash_dedup.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platform_lock.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

feat(posts): extract + record external file-host links (Phase 3)

2026-06-14 13:15:36 -04:00

test_provenance_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_reextract_archives.py

test(maintenance): patterned images for re-extract resume test

2026-06-07 12:08:02 -04:00

test_safe_probe.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_chapters.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_from_post.py

test(series): assert group start_page, not per-page stated_page

2026-06-11 23:17:14 -04:00

test_series_match_service.py

feat(series): pending staging for add-from-post (#789 Phase 2)

2026-06-11 21:47:58 -04:00

test_series_match.py

feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3)

2026-06-07 18:58:18 -04:00

test_series_model.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_pending.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_series_service.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

feat(posts): extract + record external file-host links (Phase 3)

2026-06-14 13:15:36 -04:00

test_sidecar_util.py

fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms

2026-05-27 15:35:25 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

feat(subscriptions): live posts-processed progress on backfill/recovery — #704 step 2

2026-06-05 23:43:23 -04:00

test_suggestions_bulk.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_tag_normalize.py

fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)

2026-06-06 23:57:06 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_query.py

refactor(tags): shared tag_query for fandom self-join + serialization (DRY sweep)

2026-06-10 00:19:17 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_tasks_backup.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_tasks_library_audit.py

fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)

2026-06-07 00:08:19 -04:00

test_tasks_ml.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00