v26.05.28.0: downloads dashboard + task-resilience overhaul (timeouts, archive split, 3-layer poison-pill defense) #31

2026-05-28T00:44:54-04:00

bvandeusen commented

2026-05-28 00:44:54 -04:00

(Migrated from git.fabledsword.com)

Summary

Two themes: a Downloads-tab dashboard pass, and a deep task-resilience overhaul that eliminates the false-failure / poison-pill classes of backend-task bugs.

1. Downloads dashboard (b1b129c). Row restyle — colored status chip with label, 4px status-keyed left edge, color-coded PlatformChip, file-count chip (dim dot when 0), error as a red pill, per-row Retry/Details/Open-artist actions. Date-grouped collapsible sections (Today / Yesterday / Last 7 days / Earlier) with failed runs pinned to the top of each section + per-section count chips.

2. Task timeout fixes (407de18, a85880f, 57a22d6).

ml tag_and_embed: time limits bumped 5min/7min → 15min/20min for the video branch (10-frame sample × tagger+embedder ≈ 20 GPU ops). Recovery sweep made per-queue (ml=25min) so it stops marking in-flight video tagging "failed".
Archive imports split into their own import_archive_file task (soft=30min/hard=35min) — the per-member pipeline runs inline for every member, which blew the shared 300s media limit. New enqueue_import router (media vs archive) used by scan / retry / recovery. Both recovery sweeps made archive-aware (40-min threshold) so a legitimately-running archive isn't preempted.

3. Three-layer poison-pill defense (e77afe8, e3cdd0f, dcfe55d). Closes the "process won't stop failing" hole:

L1 — circuit breaker. alembic 0026 adds import_task.recovery_count + refetched. The stuck-task sweep caps re-queues: a row that keeps hard-crashing the worker (OOM/segfault, no terminal flip) is marked failed with a diagnostic after MAX_RECOVERY_ATTEMPTS instead of looping forever. Surfaces the offending file path in the failures list.
L3 — subprocess isolation. New safe_probe.py: probe_video (ffprobe, separate process — also captures video dims) and probe_archive (bomb-size guard + integrity test in a spawned child). A decoder crash / decompression-bomb OOM / native-lib segfault is contained to the child and returns a clean terminal failure. Wired into _import_media (video), _import_archive, and tag_and_embed. Images stay in-process by design.
L2 — one-shot re-download. refetch_service resolves a failed file back to an enabled real-URL Source via its sidecar, deletes the corrupt copy, and triggers one source re-check (skip_existing re-fetches it). Bounded by refetched. Manual button on failed import-task rows + env-gated auto path (FC_AUTO_REFETCH_CORRUPT=1, default off). Filesystem-only imports report no_source.

Migrations

alembic 0026_import_task_recovery_count_refetched — additive (two columns, defaults 0/false), no backfill.

Test plan

CI green
Deploy: :latest pulled, services force-recreated, alembic 0026 applied
Downloads tab: status chips, color platform chips, date sections collapse, failed runs pinned, per-row Retry works
A video ML job runs to completion without a spurious RecoverySweep failure at 5 min
A large archive imports without SoftTimeLimitExceeded at 5 min
Re-fetch button on a failed downloaded-file task → "Re-fetch queued"; on a filesystem-only task → "no re-fetchable source"
(Optional) set FC_AUTO_REFETCH_CORRUPT=1 and confirm a poison-pill failure with a resolvable source auto-triggers one re-fetch

## Summary Two themes: a Downloads-tab dashboard pass, and a deep task-resilience overhaul that eliminates the false-failure / poison-pill classes of backend-task bugs. **1. Downloads dashboard (`b1b129c`).** Row restyle — colored status chip with label, 4px status-keyed left edge, color-coded PlatformChip, file-count chip (dim dot when 0), error as a red pill, per-row Retry/Details/Open-artist actions. Date-grouped collapsible sections (Today / Yesterday / Last 7 days / Earlier) with failed runs pinned to the top of each section + per-section count chips. **2. Task timeout fixes (`407de18`, `a85880f`, `57a22d6`).** - ml `tag_and_embed`: time limits bumped 5min/7min → 15min/20min for the video branch (10-frame sample × tagger+embedder ≈ 20 GPU ops). Recovery sweep made per-queue (ml=25min) so it stops marking in-flight video tagging "failed". - Archive imports split into their own `import_archive_file` task (soft=30min/hard=35min) — the per-member pipeline runs inline for every member, which blew the shared 300s media limit. New `enqueue_import` router (media vs archive) used by scan / retry / recovery. Both recovery sweeps made archive-aware (40-min threshold) so a legitimately-running archive isn't preempted. **3. Three-layer poison-pill defense (`e77afe8`, `e3cdd0f`, `dcfe55d`).** Closes the "process won't stop failing" hole: - **L1 — circuit breaker.** alembic 0026 adds `import_task.recovery_count` + `refetched`. The stuck-task sweep caps re-queues: a row that keeps hard-crashing the worker (OOM/segfault, no terminal flip) is marked `failed` with a diagnostic after `MAX_RECOVERY_ATTEMPTS` instead of looping forever. Surfaces the offending file path in the failures list. - **L3 — subprocess isolation.** New `safe_probe.py`: `probe_video` (ffprobe, separate process — also captures video dims) and `probe_archive` (bomb-size guard + integrity test in a spawned child). A decoder crash / decompression-bomb OOM / native-lib segfault is contained to the child and returns a clean terminal failure. Wired into `_import_media` (video), `_import_archive`, and `tag_and_embed`. Images stay in-process by design. - **L2 — one-shot re-download.** `refetch_service` resolves a failed file back to an enabled real-URL Source via its sidecar, deletes the corrupt copy, and triggers one source re-check (skip_existing re-fetches it). Bounded by `refetched`. Manual button on failed import-task rows + env-gated auto path (`FC_AUTO_REFETCH_CORRUPT=1`, default off). Filesystem-only imports report `no_source`. ## Migrations `alembic 0026_import_task_recovery_count_refetched` — additive (two columns, defaults 0/false), no backfill. ## Test plan - [ ] CI green - [ ] Deploy: `:latest` pulled, services force-recreated, alembic 0026 applied - [ ] Downloads tab: status chips, color platform chips, date sections collapse, failed runs pinned, per-row Retry works - [ ] A video ML job runs to completion without a spurious RecoverySweep failure at 5 min - [ ] A large archive imports without SoftTimeLimitExceeded at 5 min - [ ] Re-fetch button on a failed downloaded-file task → "Re-fetch queued"; on a filesystem-only task → "no re-fetchable source" - [ ] (Optional) set `FC_AUTO_REFETCH_CORRUPT=1` and confirm a poison-pill failure with a resolvable source auto-triggers one re-fetch

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#31