bvandeusen 8dee2f9628 feat(import): recurse nested archives + precise "no images" reason (#718) ...

Root cause (operator-confirmed via event metadata + lsar): a "High Resolution
files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase).
_import_archive only extracted one level — a nested-archive member failed
is_supported and was skipped, so the real pages were silently dropped and the post
showed "archive but no images". The disk scan found this pattern recurring across
the attachment store.

- Recurse into nested archives via _collect_archive_members: a member that is
  itself an archive is bomb-probed and extracted too, depth-capped at
  _ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so
  they link to the right Post. Each level is wrapped so one bad nested archive
  can't abort the import. The shared path means external (mega/gdrive) archives
  recurse too.
- Replace the catch-all "held no supported members" string with a per-outcome
  tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped
  case is now recognised as BENIGN — images already in the library, re-linked to
  this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no
  longer false-flags in event metadata.unextracted_archives. Genuine failures
  carry the precise breakdown.

Tests: nested zip-in-cbz imports both inner images + links them to the outer post;
all-deduped archive returns attached with error=None and links images to both posts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-15 21:18:02 -04:00

..

fixtures

feat(patreon): native JSON-API client — ingester build step 1 (plan #697)

2026-06-05 19:16:20 -04:00

__init__.py

feat: add Quart app factory, config loader, and /api/health endpoint

2026-05-14 07:32:17 -04:00

_prediction_helpers.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

conftest.py

perf(ci): reuse a session-scoped engine for the per-test DB reset

2026-06-04 07:35:28 -04:00

test_api_admin.py

feat(tags): 'Reset content tagging' admin action

2026-06-04 16:47:36 -04:00

test_api_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_allowlist.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_artist.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_artists_create.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_artists_directory.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_api_attachments.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_bulk_tags.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_cleanup.py

fix(cleanup): library scans survive navigation, reconnect on return

2026-06-04 16:55:20 -04:00

test_api_credentials.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_api_downloads.py

fix(downloads): rewire MaintenanceMenu to the downloads pipeline

2026-05-30 00:59:05 -04:00

test_api_extension_api_key.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_extension.py

feat(extension): probe shows current state before click; v1.0.6

2026-06-01 00:41:17 -04:00

test_api_gallery.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_api_import_admin.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_ml_admin.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

test_api_platforms.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_posts.py

feat(browse): sticky tabs + per-tab search bar (server-side, scope-aware)

2026-06-12 00:04:06 -04:00

test_api_provenance.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_api_series.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_api_settings_downloader.py

feat(external): per-host enable toggles in Settings (Phase 4d)

2026-06-14 15:57:42 -04:00

test_api_settings.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_api_showcase.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_sources_check.py

feat(downloads): bulk retry respects cooldown; single-source RETRY overrides

2026-05-30 23:15:07 -04:00

test_api_sources.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_api_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_api_system_activity.py

feat(activity): search/filter on both Activity-tab panes

2026-06-10 11:34:42 -04:00

test_api_system_backup.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tag_merge.py

feat(gallery,tags): clear active filters

2026-06-03 23:26:04 -04:00

test_api_tags_directory.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_api_tags.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_api_thumbnails.py

fix(thumbnails): surface backfill results + tighten validity check

2026-06-01 21:57:29 -04:00

test_archive_extractor.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_artist_directory_service.py

fc3f: ArtistDirectoryService — cursor-paginated artists directory mirroring TagDirectoryService

2026-05-21 22:14:15 -04:00

test_artist_service_find_or_create.py

feat(fc3a): ArtistService.find_or_create + autocomplete

2026-05-20 12:54:47 -04:00

test_artist_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_artist_visit.py

fix(artist): ruff UP017 + test_directory_card_shape pin

2026-06-03 15:45:59 -04:00

test_attachment_store.py

feat(attachments): sha-addressed AttachmentStore

2026-05-19 11:11:07 -04:00

test_audits_single_color.py

feat(fc-cleanup): audits/single_color.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:38 -04:00

test_audits_transparency.py

feat(fc-cleanup): audits/transparency.py + tests — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:18:07 -04:00

test_backfill_phash.py

feat(deep-scan): keyset-paginated backfill_phash task

2026-05-19 14:59:04 -04:00

test_backfill_thumbnails.py

fix(test): include 'scanned' in all backfill result assertions

2026-06-01 22:09:02 -04:00

test_backup_service.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_bulk_tag_service.py

feat(bulk): BulkTagService (common_tags/bulk_add/bulk_remove) + 3 image bulk-tag routes

2026-05-16 18:02:01 -04:00

test_celery_routing.py

feat(maintenance): dedicated maintenance_long lane for long one-shot tasks

2026-06-07 09:00:03 -04:00

test_celery_smoke.py

feat: configure Celery with per-feature queue lanes and a smoke task

2026-05-14 07:34:47 -04:00

test_cleanup_old_download_events.py

fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention)

2026-05-21 16:02:36 -04:00

test_cleanup_service_audit.py

feat(fc-cleanup): api/cleanup.py blueprint (9 endpoints) + register + delete-audit-<id> token (matches modal convention) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 08:13:53 -04:00

test_cleanup_service.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_credential_crypto.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_credential_service.py

fix(audit-g5c): refuse silent Fernet key regeneration on partial restore

2026-06-02 17:04:27 -04:00

test_db_helpers.py

refactor(services): shared race-safe get_or_create helper (DRY backend sweep)

2026-06-09 23:46:01 -04:00

test_download_backends.py

feat(downloads): native Patreon verify + uniform backend dispatch (plan #697)

2026-06-05 22:49:43 -04:00

test_download_models.py

fix(tests): update test_ensure_camie_skips_when_present to v2 filenames (camie-tagger-v2.onnx + camie-tagger-v2-metadata.json) — pinned-test bounce from 3b3e756

2026-05-25 11:12:55 -04:00

test_download_service.py

refactor(ingest): per-post handling into run stdout via a downloader outcome (#842)

2026-06-14 23:27:38 -04:00

test_download_source_task.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_extension_cors.py

fix(lint): S1 ruff fallout — collapse double blank after import block (I001) + strip W293 in test_suggestions_bulk

2026-05-28 12:43:08 -04:00

test_extensions_session.py

refactor(fc2c-i): single shared async engine with lifecycle dispose

2026-05-15 13:53:42 -04:00

test_external_fetch.py

feat(external): file-host fetcher subsystem (Phase 4a)

2026-06-14 13:29:20 -04:00

test_external_worker.py

fix(external): path-safe unlink + per-link staging + orphan repair (#859)

2026-06-15 01:48:38 -04:00

test_fe_be_contract.py

feat(I6): FE/BE enum contract test — pin JS mirrors to backend canon

2026-05-29 15:34:20 -04:00

test_file_validator.py

feat(fc3c): FileValidator — magic-byte head/tail check for fresh writes

2026-05-20 20:39:17 -04:00

test_gallery_artist_payload.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

test_gallery_dl_service.py

feat(patreon): structured ingester results + quarantine surfacing — #704 step 1

2026-06-05 23:39:28 -04:00

test_gallery_facets.py

feat(gallery): faceted filter params + /facets counts endpoint (Phase 2 backend)

2026-06-04 07:01:07 -04:00

test_gallery_provenance_filter.py

feat(gallery): composable scroll filter (multi-tag AND, media, sort)

2026-06-03 23:50:19 -04:00

test_gallery_service.py

fix(images): percent-encode original-image URLs ('#' in paths 404'd)

2026-06-12 00:36:44 -04:00

test_gallery_similar.py

feat(gallery): visual 'more like this' search (Phase 3 backend)

2026-06-04 08:47:43 -04:00

test_health.py

refactor(dry-S1): hoist app/client test fixtures into conftest

2026-05-28 11:33:05 -04:00

test_html_sanitize.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_image_prediction.py

feat(ml): image_prediction table + backfill + dual-write (#768 step 1)

2026-06-10 15:55:32 -04:00

test_import_file_attached.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_importer_archive.py

feat(import): recurse nested archives + precise "no images" reason (#718)

2026-06-15 21:18:02 -04:00

test_importer_artist_id.py

feat(provenance): importer sets image_record.artist_id, stops minting artist tags

2026-05-18 21:33:55 -04:00

test_importer_attach_in_place.py

refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)

2026-06-15 10:10:21 -04:00

test_importer_attachments.py

fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 20:33:44 -04:00

test_importer_deep.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_importer_provenance_race.py

fix(gallery+tests): alias Post inside artist EXISTS; tests stop asserting synthetic Source

2026-06-01 14:40:28 -04:00

test_importer_upsert_helpers.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_importer.py

fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever

2026-05-25 02:34:36 -04:00

test_link_extract.py

feat(posts): extract + record external file-host links (Phase 3)

2026-06-14 13:15:36 -04:00

test_maintenance.py

fix(maintenance): download queue needs a sweep threshold above its 25-min time_limit

2026-06-10 11:57:31 -04:00

test_migration_0002.py

fix(test): pin tag_kind enum test to the post-0023 set (meta + rating removed)

2026-05-26 21:23:12 -04:00

test_migration_0003.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_migration_0004.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

test_migration_0007.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0008.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_migration_0009.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

test_migration_0010.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

test_migration_0011.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

test_migration_0012.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

test_migration_0013.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

test_ml_aliases.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_allowlist.py

feat(ml): clamp allowlist min_confidence to the tagger store floor

2026-06-10 13:52:20 -04:00

test_ml_artist_retired.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

test_ml_centroids.py

feat(fc2b): add CentroidService — per-tag SigLIP centroids + similarity

2026-05-15 07:37:53 -04:00

test_ml_embedder.py

feat(fc2b): add SigLIP embedder wrapper

2026-05-15 07:36:17 -04:00

test_ml_suggestions.py

fix(aliases): store modal alias under raw model key + make aliases visible/manageable

2026-06-12 13:05:58 -04:00

test_ml_tag_name.py

feat(ml): normalize Camie suggestion names to human-readable

2026-06-03 13:00:08 -04:00

test_ml_tagger.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

test_models.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

test_page_number_parser.py

feat(series): post→series flows + browse list — backend (FC-6.2)

2026-06-07 18:34:11 -04:00

test_paths.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_patreon_client.py

fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)

2026-06-15 00:14:52 -04:00

test_patreon_contract.py

fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)

2026-06-15 00:14:52 -04:00

test_patreon_downloader.py

test(downloader): fix test_sidecar_written_and_findable for post-first minimal sidecar (#856)

2026-06-15 10:15:04 -04:00

test_patreon_ingester.py

feat(ingest): post-body schema-drift canary — fail a native walk red when zero bodies extracted (#862)

2026-06-15 08:56:11 -04:00

test_patreon_resolver.py

feat(patreon): resolve the creator's campaign from a single-post URL

2026-06-06 21:32:15 -04:00

test_patreon_seen_media.py

feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)

2026-06-05 19:26:57 -04:00

test_phash_dedup.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_phash_util.py

fix(test): phash 'far' case needs structured images — solid colors phash-collapse

2026-05-17 22:18:07 -04:00

test_platform_lock.py

feat(downloads): serialize same-platform downloads (Patreon concurrency cap)

2026-06-06 21:22:32 -04:00

test_platforms_registry.py

feat(fc3b): platforms.py — GS-6 registry (single source of truth)

2026-05-20 18:34:02 -04:00

test_post_feed_service.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_prosemirror.py

fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842)

2026-06-15 00:14:52 -04:00

test_provenance_service.py

feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics

2026-06-01 14:17:52 -04:00

test_reextract_archives.py

test(maintenance): patterned images for re-extract resume test

2026-06-07 12:08:02 -04:00

test_safe_probe.py

fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1

2026-06-06 14:33:24 -04:00

test_scan_deep_requeue.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

test_scan_enumerates_all.py

feat(attachments): scan enumerates all; import_file maps 'attached' + member ML

2026-05-19 11:14:49 -04:00

test_scheduler_service.py

feat(scheduler): order due sources by last_checked_at — most overdue first

2026-05-30 12:08:43 -04:00

test_series_chapters.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_from_post.py

test(series): assert group start_page, not per-page stated_page

2026-06-11 23:17:14 -04:00

test_series_match_service.py

feat(series): pending staging for add-from-post (#789 Phase 2)

2026-06-11 21:47:58 -04:00

test_series_match.py

feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3)

2026-06-07 18:58:18 -04:00

test_series_model.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_series_pending.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_series_service.py

feat(series): operator-set sparse page numbers + gap blocks (#789 tweak)

2026-06-11 23:10:30 -04:00

test_showcase_service.py

feat(fc2c-i): showcase random-sample service and endpoint

2026-05-15 15:49:25 -04:00

test_sidecar_import.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_sidecar_util.py

feat(ingest): localize inline post-body images to local copies (Phase 2)

2026-06-14 16:39:58 -04:00

test_slug.py

feat(fc2a): add slug and paths utilities

2026-05-14 12:04:05 -04:00

test_source_service.py

feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830)

2026-06-14 20:58:40 -04:00

test_suggestions_bulk.py

feat(ml): read suggestions + allowlist from image_prediction (#768 step 2)

2026-06-10 16:03:58 -04:00

test_tag_directory_service.py

feat(fc2c-i): tag directory service and /api/tags/directory

2026-05-15 15:50:41 -04:00

test_tag_merge.py

feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)

2026-06-11 21:30:01 -04:00

test_tag_normalize.py

fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)

2026-06-06 23:57:06 -04:00

test_tag_prefix.py

fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text.

2026-05-26 21:00:12 -04:00

test_tag_query.py

refactor(tags): shared tag_query for fandom self-join + serialization (DRY sweep)

2026-06-10 00:19:17 -04:00

test_tag_rename.py

feat(fc2b): add AllowlistService + TagService.rename

2026-05-15 07:39:33 -04:00

test_tag_service.py

fix(tags): preserve acronym casing in tag normalization (DC, NSFW)

2026-06-06 17:39:54 -04:00

test_task_run_signals.py

fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test as direct _finalize unit

2026-05-24 21:22:42 -04:00

test_tasks_admin.py

fix(external): path-safe unlink + per-link staging + orphan repair (#859)

2026-06-15 01:48:38 -04:00

test_tasks_backup.py

fix(backup): compressed -Fc dumps + pg_restore; reconcile subprocess timeouts (#739)

2026-06-10 14:29:17 -04:00

test_tasks_library_audit.py

fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)

2026-06-07 00:08:19 -04:00

test_tasks_ml.py

feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)

2026-06-11 18:52:33 -04:00

test_tasks_register.py

fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps

2026-05-27 22:45:11 -04:00

test_text_utils.py

fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps)

2026-05-21 18:53:35 -04:00

test_thumbnailer.py

feat(fc2a): add Thumbnailer service

2026-05-14 12:05:19 -04:00

test_tick_due_sources.py

fc3d: tick_due_sources Celery task on the scan queue

2026-05-21 16:01:59 -04:00

test_verify_integrity.py

fix(test): verify_integrity sha-mismatch row needs a distinct path (image_record.path is unique)

2026-05-19 18:15:59 -04:00