Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.
- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
(VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
autocommit connection.
- GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.
Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.
- download_service: replace hardcoded ('discord','pixiv') tuple with
auth_type_for(platform) == 'token'. A 7th token-platform now picks
up the right credential path without touching this site.
- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
merge so the target's centroid reflects its new image set
immediately. Daily list_drifted catches it within 24h, but eager
recompute closes the suggestion-quality dip in the meantime.
- backfill_thumbnails added to beat_schedule (daily). The task
docstring claimed periodic Beat but the entry was never registered,
so the library got no self-healing thumbnail repair; only the
manual admin-UI button fired it.
- modal.createAndAdd pushes a kind='fandom' tag into
tagsStore.fandomCache so FandomPicker sees the new fandom on next
open. Was: cache-gated load (length===0) skipped refetch, new
fandom invisible until full page reload.
- cleanup cluster:
- Drop .webp from cleanup_service.unlink — thumbnailer only writes
.jpg/.png; the third tuple member was dead code.
- Drop effective_date from /api/gallery/scroll response — no FE
consumer reads it. Service still computes the attribute for
timeline ordering; this just trims the JSON.
- Rename store.recentMinute → store.recentRuns across the
systemActivity store + three consumers (SystemActivitySummary,
QueuesTable, SystemActivityTab). The data is the last 200 runs
(not actually "last minute"), so the name lied.
NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
The scan tick (scan.py:_tick_due_sources_async) inserts
DownloadEvent(status='pending') and fires download_source.delay(). If the
task dies before finalizing the event — worker OOM/SIGKILL, lost task, or
a gallery-dl that didn't unwind on the 1200s hard time_limit — the event
stays in-flight forever. Every later tick then skips the source via the
in-flight guard (scan.py:168), so Source.last_checked_at is never written
and the operator sees "last check never" in the Subscriptions health
column, permanently.
cleanup_old_download_events only prunes terminal events (by design); no
existing sweep covered the pending/running case. Operator confirmed
2026-05-29 with a diagnostic query: all 43 "never checked" sources were
stranded behind stale in-flight events (eligible_stuck_inflight = 43,
every other bucket zero).
New recover_stalled_download_events task (Beat every 5 min):
- Flips DownloadEvent rows pending/running > 30 min (10 min past the
download_source 1200s hard kill, so legitimately-running tasks are
never touched) to status='error' with a sentinel message.
- Bumps each affected Source's consecutive_failures ONCE per source —
backoff is 2^N on that counter so per-event bumps would needlessly
inflate the next interval — sets last_error, stamps last_checked_at.
UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that
would hit the psycopg 65535-param ceiling on a large strand pile.
Net: the 43 currently-stranded sources unstick on the first sweep after
deploy, their health dots flip amber instead of unchecked, and the next
scan tick re-queues them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The celery_app.py beat-schedule edit failed with a stale-read error in
the Task 9 commit, so the ML daily jobs weren't registered. Adds
ml-backfill-daily, recompute-centroids-daily, apply-allowlist-sweep-daily.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tag_and_embed: Camie + SigLIP on one image (video → 10-frame sample,
max-pool tags, mean-pool embeddings), stores predictions/embedding with
model versions, then enqueues per-image allowlist apply. backfill:
keyset-paginated discovery of images missing predictions/embeddings for
the current model versions (restart-safe). apply_allowlist_tags stub
included so .delay() resolves between commits (filled in Task 9).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
recover_interrupted_tasks runs every 5 minutes, finds ImportTask rows
stuck in 'processing' for >30 minutes (well above any legitimate import
duration), and re-queues them. cleanup_old_tasks runs daily and deletes
finished tasks older than 7 days so the task table stays an operational
view rather than an archive.
Both thresholds match ImageRepo's precedent. The 30-min stuck threshold
is documented inline so a future reader can adjust it intentionally
rather than mistaking it for a 'magic number'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scan_directory walks ImportSettings.import_scan_path, creates an
ImportBatch, enumerates supported files into ImportTasks, and enqueues
import_media_file per task. import_media_file moves the task through
its state machine (pending → queued → processing → complete/skipped/failed),
updates ImportBatch counters atomically (UPDATE ... SET col = col + 1),
enqueues a thumbnail task on success, and marks the batch complete when
the last task drains.
generate_thumbnail runs on its own queue (thumbnail) so big imports
don't starve thumbnail throughput; failure here is logged and does not
fail the import.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Routes are pre-declared for FC-2/FC-3 task modules (import, ml, thumbnail,
download, scan, maintenance). Queue lanes match the ImageRepo pattern where
beat+maintenance run on a separate worker so long imports don't starve
periodic tasks. Smoke ping task confirms the wiring in eager mode for CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen