v26.05.24.3: FC-3i System Activity dashboard + migration backup-gate retired + modal Escape #9

2026-05-24T21:47:42-04:00

bvandeusen commented

2026-05-24 21:47:42 -04:00

(Migrated from git.fabledsword.com)

Big release: FC-3i System Activity Overview ships (IR-style operator visibility into the Celery pipeline), plus two dogfood fixes that unblock the IR migration UI.

Headline — FC-3i: System Activity Overview

Operator visibility into every Celery lane: queue depths, worker assignments, recent activity, recent failures. Lives at Settings → Activity with a quick-summary card on Settings → Overview.

New task_run table (alembic 0016): id, queue, task_name, target_id, started_at/finished_at, duration_ms, status (running/ok/error/timeout/retry), error_type, error_message, retry_count, worker_hostname, args_summary. Composite indexes on (queue, started_at desc), (status, started_at desc), (task_name, started_at desc).
Celery signal handlers in backend/app/celery_signals.py populate task_run automatically for every task — drop-in for every existing and future Celery task; no per-task instrumentation. task_prerun inserts a running row; task_postrun marks ok; task_failure marks error (or timeout for SoftTimeLimitExceeded); task_retry marks retry with the count. Wrapped in try/except + structured logging so a broken signal handler never breaks the task itself.
Recovery sweep recover_stalled_task_runs (Beat every 5 min): flips task_run rows stuck in running past STUCK_THRESHOLD_MINUTES to error with error_type='RecoverySweep'.
Retention sweep prune_task_runs (Beat daily): deletes ok rows older than 24h, error/timeout/retry rows older than 7 days. running rows never deleted by retention — that's the recovery sweep's job.
/api/system/activity blueprint with four endpoints: /queues (Redis LLEN per queue, cached 2s), /workers (celery inspect, cached 5s), /runs (paginated task_run history, queue/status filters, keyset cursor), /failures (24h-window error+timeout rows + count_by_type aggregation).
Frontend: new systemActivity Pinia store + QueuesTable.vue (shared) + SystemActivitySummary.vue (5s poll on Overview) + SystemActivityTab.vue (3s+15s polls on Activity tab with three panes: queues+workers, recent failures with clickable error-type pills, all-activity table with queue/status filters).
32 new integration tests + 6 new maintenance task tests.

Migration UX unblock — backup-gate retired

The _APPLY_KINDS check in /api/migrate/<kind> was blocking IR/GS ingest from the UI when there was no recent pre_migration backup — but backup on a large NFS-backed image library hits the 30-min timeout and fails reliably. The gate has been retired (_APPLY_KINDS = frozenset()); the UI ingest path now just works. FC-3h will rewrite backup as a first-class feature with proper splitting (DB vs images, independent timeouts/schedules) and recovery.

Modal Escape via document-level listener

The @keydown.esc handler on the viewer modal's root div didn't fire when a <video> element had focus (video controls capture keystrokes and don't bubble reliably). Moved to document.addEventListener('keydown', ..., capture=true) so Escape closes the modal whether you're on an image or a video, focused on the modal or on the video itself.

Commits

37fcc74 fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test
6a532c1 fc3i(ui): Settings → Activity tab + Overview summary card
3686130 fc3i(ui): SystemActivityTab.vue — queues + failures + all-activity panes
ddbb84d fc3i(ui): SystemActivitySummary.vue — Overview-tab quick summary card (5s poll)
9b25294 fc3i(ui): QueuesTable.vue — shared queue/worker/recent table
89d0cb2 fc3i(ui): systemActivity Pinia store
36611cb fc3i: tests for recover_stalled_task_runs + prune_task_runs
48ef224 fc3i: integration tests for /api/system/activity/* endpoints
e523d0a fc3i: integration tests for Celery signal → task_run lifecycle
541e2bf fc3i: /api/system/activity blueprint
7782672 fc3i: recover_stalled_task_runs + prune_task_runs + Beat entries
d12b51f fc3i: Celery signal handlers
0cda46f fc3i: alembic 0016 — task_run table
79fee98 fc3i: TaskRun model
b1d6892 fix(tests): drop test_post_apply_without_backup_rejected
5535677 fix: drop migration backup-gate + modal Escape via document-level listener

Big release: FC-3i System Activity Overview ships (IR-style operator visibility into the Celery pipeline), plus two dogfood fixes that unblock the IR migration UI. ## Headline — FC-3i: System Activity Overview Operator visibility into every Celery lane: queue depths, worker assignments, recent activity, recent failures. Lives at `Settings → Activity` with a quick-summary card on `Settings → Overview`. - **New `task_run` table** (alembic 0016): id, queue, task_name, target_id, started_at/finished_at, duration_ms, status (running/ok/error/timeout/retry), error_type, error_message, retry_count, worker_hostname, args_summary. Composite indexes on (queue, started_at desc), (status, started_at desc), (task_name, started_at desc). - **Celery signal handlers** in `backend/app/celery_signals.py` populate `task_run` automatically for every task — drop-in for every existing and future Celery task; no per-task instrumentation. `task_prerun` inserts a `running` row; `task_postrun` marks `ok`; `task_failure` marks `error` (or `timeout` for `SoftTimeLimitExceeded`); `task_retry` marks `retry` with the count. Wrapped in try/except + structured logging so a broken signal handler never breaks the task itself. - **Recovery sweep** `recover_stalled_task_runs` (Beat every 5 min): flips `task_run` rows stuck in `running` past `STUCK_THRESHOLD_MINUTES` to `error` with `error_type='RecoverySweep'`. - **Retention sweep** `prune_task_runs` (Beat daily): deletes `ok` rows older than 24h, `error`/`timeout`/`retry` rows older than 7 days. `running` rows never deleted by retention — that's the recovery sweep's job. - **`/api/system/activity` blueprint** with four endpoints: `/queues` (Redis LLEN per queue, cached 2s), `/workers` (celery inspect, cached 5s), `/runs` (paginated task_run history, queue/status filters, keyset cursor), `/failures` (24h-window error+timeout rows + count_by_type aggregation). - **Frontend**: new `systemActivity` Pinia store + `QueuesTable.vue` (shared) + `SystemActivitySummary.vue` (5s poll on Overview) + `SystemActivityTab.vue` (3s+15s polls on Activity tab with three panes: queues+workers, recent failures with clickable error-type pills, all-activity table with queue/status filters). - 32 new integration tests + 6 new maintenance task tests. ## Migration UX unblock — backup-gate retired The `_APPLY_KINDS` check in `/api/migrate/<kind>` was blocking IR/GS ingest from the UI when there was no recent `pre_migration` backup — but backup on a large NFS-backed image library hits the 30-min timeout and fails reliably. The gate has been retired (`_APPLY_KINDS = frozenset()`); the UI ingest path now just works. FC-3h will rewrite backup as a first-class feature with proper splitting (DB vs images, independent timeouts/schedules) and recovery. ## Modal Escape via document-level listener The `@keydown.esc` handler on the viewer modal's root div didn't fire when a `<video>` element had focus (video controls capture keystrokes and don't bubble reliably). Moved to `document.addEventListener('keydown', ..., capture=true)` so Escape closes the modal whether you're on an image or a video, focused on the modal or on the video itself. ## Commits ``` 37fcc74 fix(fc3i): single-line celery.signals import + INT32 bounds on target_id + dict.fromkeys + rewrite retry test 6a532c1 fc3i(ui): Settings → Activity tab + Overview summary card 3686130 fc3i(ui): SystemActivityTab.vue — queues + failures + all-activity panes ddbb84d fc3i(ui): SystemActivitySummary.vue — Overview-tab quick summary card (5s poll) 9b25294 fc3i(ui): QueuesTable.vue — shared queue/worker/recent table 89d0cb2 fc3i(ui): systemActivity Pinia store 36611cb fc3i: tests for recover_stalled_task_runs + prune_task_runs 48ef224 fc3i: integration tests for /api/system/activity/* endpoints e523d0a fc3i: integration tests for Celery signal → task_run lifecycle 541e2bf fc3i: /api/system/activity blueprint 7782672 fc3i: recover_stalled_task_runs + prune_task_runs + Beat entries d12b51f fc3i: Celery signal handlers 0cda46f fc3i: alembic 0016 — task_run table 79fee98 fc3i: TaskRun model b1d6892 fix(tests): drop test_post_apply_without_backup_rejected 5535677 fix: drop migration backup-gate + modal Escape via document-level listener ```

bvandeusen referenced this issue from a commit

2026-06-06 00:24:25 -04:00

refactor(ingest): extract platform-agnostic Ingester core — roadmap #9 (plan #706)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#9