bvandeusenandClaude Opus 5
45bb7044f7
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 31s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m44s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"
There was a reason and it had expired, and underneath it there was plain
waste.
The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.
The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.
So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.
Consequences, stated rather than hidden:
- The live figures are up to one sweep old. measured_at travels with each
lane and the page says how old, because a stale number presented as
current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
staleness thresholds are in a relationship. 60s against a 90s stale
threshold left one missed tick between normal and all-yellow — the
shape of lesson #4355 — so the period is 30s, named once in
worker_lanes, and system_health asserts its headroom at import with a
test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
the direction asked for: "idle instances quiet down when not running".
Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
364 lines
19 KiB
Python
364 lines
19 KiB
Python
"""Celery configuration with separate queue lanes.
|
||
|
||
Queues:
|
||
import — filesystem import path (FC-2)
|
||
ml — WD14 + SigLIP inference (FC-2; runs in the ml-worker image)
|
||
thumbnail — image and video thumbnail generation (FC-2)
|
||
download — gallery-dl tasks (FC-3)
|
||
scan — periodic source checks (FC-3) — kept separate so long imports
|
||
don't starve the scheduler
|
||
maintenance — recovery sweeps, pHash backfill, GPU-queue coordination, etc.
|
||
default — anything not explicitly routed
|
||
"""
|
||
|
||
from celery import Celery
|
||
|
||
from .config import get_config
|
||
from .services.worker_lanes import SWEEP_PERIOD_SECONDS
|
||
|
||
|
||
def make_celery() -> Celery:
|
||
cfg = get_config()
|
||
app = Celery(
|
||
"fabledcurator",
|
||
broker=cfg.celery_broker_url,
|
||
backend=cfg.celery_result_backend,
|
||
include=[
|
||
"backend.app.tasks.smoke",
|
||
"backend.app.tasks.scan",
|
||
"backend.app.tasks.import_file",
|
||
"backend.app.tasks.thumbnail",
|
||
"backend.app.tasks.maintenance",
|
||
"backend.app.tasks.ml",
|
||
"backend.app.tasks.gpu_queue",
|
||
"backend.app.tasks.download",
|
||
"backend.app.tasks.external",
|
||
"backend.app.tasks.backup",
|
||
"backend.app.tasks.admin",
|
||
"backend.app.tasks.library_audit",
|
||
"backend.app.tasks.translation",
|
||
],
|
||
)
|
||
app.conf.update(
|
||
task_default_queue="default",
|
||
task_routes={
|
||
"backend.app.tasks.import_file.*": {"queue": "import"},
|
||
"backend.app.tasks.ml.*": {"queue": "ml"},
|
||
# GPU-queue coordination (backfill enqueues, orphan recovery,
|
||
# reprocess) is pure DB work — it rides the maintenance quick lane
|
||
# so the GPU agent pipeline works even on stacks that drop the
|
||
# (now-optional, B3) ml-worker container entirely.
|
||
"backend.app.tasks.gpu_queue.*": {"queue": "maintenance"},
|
||
"backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
|
||
"backend.app.tasks.download.*": {"queue": "download"},
|
||
# External file-host fetches are downloads — same lane (they can run
|
||
# long, but the download worker already tolerates long backfills).
|
||
"backend.app.tasks.external.*": {"queue": "download"},
|
||
"backend.app.tasks.scan.*": {"queue": "scan"},
|
||
# `maintenance` is the QUICK lane — recovery sweeps, vacuum, cleanup
|
||
# (concurrency-1 on the scheduler). The long one-shots (DB backups,
|
||
# library audits, admin maintenance: normalize/re-extract/cascade-
|
||
# delete) run on a SEPARATE `maintenance_long` lane + worker so they
|
||
# can never starve the quick self-healing sweeps (operator-flagged
|
||
# 2026-06-07: a 2h audit blocked vacuum/backup/normalize for hours).
|
||
"backend.app.tasks.maintenance.*": {"queue": "maintenance"},
|
||
"backend.app.tasks.backup.*": {"queue": "maintenance_long"},
|
||
"backend.app.tasks.admin.*": {"queue": "maintenance_long"},
|
||
"backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
|
||
# Translation backfill hits the LLM (~1–6s/item) → the long lane so it
|
||
# never starves the quick self-healing sweeps (#143).
|
||
"backend.app.tasks.translation.*": {"queue": "maintenance_long"},
|
||
},
|
||
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
|
||
task_acks_late=True,
|
||
# Deploy graceful-shutdown safety: with acks_late, a task killed because
|
||
# it outran the container's stop-grace window (SIGKILL) is re-queued
|
||
# rather than silently lost. Safe because our long tasks are idempotent +
|
||
# chunked (translation per-post commit, downloads terminal-status, audits
|
||
# chunk) and the 5-min recovery sweeps re-drive anything left non-terminal
|
||
# — a re-run resumes cleanly and never corrupts. No redeliver-loop risk:
|
||
# heavy GPU work is tombstoned via gpu_queue, not run inline in a worker.
|
||
task_reject_on_worker_lost=True,
|
||
worker_prefetch_multiplier=1,
|
||
# Broker resilience (2026-06-24): a swarm overlay-network blip after a
|
||
# redeploy left Redis healthy but transiently unreachable, and a worker
|
||
# starting in that window crash-looped on the initial broker connect
|
||
# (kombu OperationalError) instead of waiting it out — needing a manual
|
||
# Redis reset to recover. Retry the broker FOREVER (None) on startup and
|
||
# at runtime so a transient outage self-heals when routing returns,
|
||
# rather than the worker exiting.
|
||
broker_connection_retry_on_startup=True,
|
||
broker_connection_retry=True,
|
||
broker_connection_max_retries=None,
|
||
# Redis-transport socket options (apply to the BROKER connection): a
|
||
# short connect timeout + TCP keepalive so a dead/blocked socket is
|
||
# noticed and retried, and a periodic health check that proactively
|
||
# reconnects a live worker through a network hiccup.
|
||
broker_transport_options={
|
||
"socket_connect_timeout": 5,
|
||
"socket_timeout": 30,
|
||
"socket_keepalive": True,
|
||
"retry_on_timeout": True,
|
||
"health_check_interval": 30,
|
||
},
|
||
# Same hardening for the Redis RESULT backend (separate connection pool).
|
||
redis_socket_connect_timeout=5,
|
||
redis_socket_timeout=30,
|
||
redis_socket_keepalive=True,
|
||
redis_retry_on_timeout=True,
|
||
redis_backend_health_check_interval=30,
|
||
beat_schedule={
|
||
"recover-interrupted-tasks": {
|
||
"task": "backend.app.tasks.maintenance.recover_interrupted_tasks",
|
||
"schedule": 300.0, # every 5 minutes
|
||
},
|
||
"size-worker-lanes": {
|
||
"task": "backend.app.tasks.maintenance.size_worker_lanes",
|
||
"schedule": SWEEP_PERIOD_SECONDS,
|
||
#
|
||
# The number lives in `services/worker_lanes` because three
|
||
# places must agree on it: this schedule, the freshness of the
|
||
# sample the System tab reads, and the roster staleness
|
||
# thresholds in `api/system_health` — which now depend on this
|
||
# sweep rather than on a browser being open, and assert their
|
||
# headroom over it at import.
|
||
#
|
||
# ONE entry, replacing `autoscale-worker-lanes` (60s) and
|
||
# `reconcile-worker-lanes` (300s) on 2026-09-23. They were two
|
||
# sweeps over one number and most of the autoscaler's design
|
||
# existed to stop the reconcile undoing its work; with the
|
||
# stored `slots` gone there is nothing to disagree about.
|
||
#
|
||
# Fast enough to react to a BACKLOG — a five-minute reaction to
|
||
# a queue filling up is no reaction. It also carries what the
|
||
# reconcile was for: a worker restarted at its ENV concurrency
|
||
# is corrected on the next tick rather than after five.
|
||
#
|
||
# Cheap when settled: one inspect plus one LLEN sweep, and no
|
||
# control messages at all once every lane matches. It is also
|
||
# now the ONLY thing that inspects — nothing on a request path
|
||
# does — so this is the whole broker cost of the System tab,
|
||
# whether nobody or ten tabs are watching.
|
||
},
|
||
"cleanup-old-tasks": {
|
||
"task": "backend.app.tasks.maintenance.cleanup_old_tasks",
|
||
"schedule": 86400.0, # daily
|
||
},
|
||
"cleanup-orphaned-temp-files": {
|
||
"task": "backend.app.tasks.maintenance.cleanup_orphaned_temp_files",
|
||
"schedule": 86400.0, # daily — sweep .part/.partial left by a
|
||
# download/import killed mid-write (graceful-shutdown fallout)
|
||
},
|
||
"backfill-phash-daily": {
|
||
"task": "backend.app.tasks.maintenance.backfill_phash",
|
||
"schedule": 86400.0, # daily — NULL-only, so a no-op once the
|
||
# library is hashed. This is what makes migration 0098's
|
||
# re-hash happen on its own: 0098 NULLs every phash, and
|
||
# without a scheduled refill the library would sit
|
||
# dedup-disabled until someone ran a deep scan (#4223).
|
||
},
|
||
"train-heads-nightly": {
|
||
"task": "backend.app.tasks.ml.scheduled_train_heads",
|
||
"schedule": 86400.0, # passive cadence; manual retrain stays available
|
||
},
|
||
"refresh-character-prototypes": {
|
||
"task": "backend.app.tasks.ml.refresh_character_prototypes",
|
||
"schedule": 900.0, # ~15 min; cheap global-gate no-op when idle (#1317)
|
||
},
|
||
"reconcile-character-prototypes-nightly": {
|
||
"task": "backend.app.tasks.ml.refresh_character_prototypes",
|
||
"schedule": 86400.0, # nightly FULL reconcile (belt-and-suspenders)
|
||
"args": (True,), # full=True
|
||
},
|
||
"apply-head-tags-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_apply_head_tags",
|
||
"schedule": 86400.0, # no-op unless head_auto_apply_enabled
|
||
},
|
||
"recover-orphaned-gpu-jobs": {
|
||
"task": "backend.app.tasks.gpu_queue.recover_orphaned_gpu_jobs",
|
||
"schedule": 60.0, # quick pickup of work a dead agent orphaned
|
||
},
|
||
"triage-gpu-errors": {
|
||
"task": "backend.app.tasks.maintenance.triage_gpu_errors",
|
||
"schedule": 900.0, # probe errored jobs' files → defect/file_ok
|
||
},
|
||
"enqueue-ccip-backfill-hourly": {
|
||
"task": "backend.app.tasks.gpu_queue.enqueue_gpu_backfill",
|
||
"schedule": 3600.0, # auto-feed NEW images; errored are
|
||
"args": ("ccip",), # tombstoned — retry is the button only
|
||
},
|
||
"enqueue-siglip-backfill-daily": {
|
||
"task": "backend.app.tasks.gpu_queue.enqueue_gpu_backfill",
|
||
"schedule": 86400.0, # drain the concept-crop back-catalogue
|
||
"args": ("siglip",), # (errored are tombstoned, not retried)
|
||
},
|
||
"enqueue-embed-backfill-daily": {
|
||
"task": "backend.app.tasks.gpu_queue.enqueue_gpu_backfill",
|
||
"schedule": 86400.0, # whole-image re-embed under the current
|
||
"args": ("embed",), # model (an operator swap) drains via agent
|
||
},
|
||
"ccip-auto-apply-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_ccip_auto_apply",
|
||
"schedule": 86400.0, # no-op unless ccip_auto_apply_enabled
|
||
},
|
||
"retract-auto-tags-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_retract_auto_tags",
|
||
"schedule": 86400.0, # soft auto-apply: drop auto-tags now below
|
||
# their threshold (m139); no-op unless the auto-apply switch is on
|
||
},
|
||
"presentation-auto-apply-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_presentation_auto_apply",
|
||
"schedule": 86400.0, # auto-hide banner chrome (#141);
|
||
# no-op unless presentation_auto_apply_enabled
|
||
},
|
||
"process-auto-apply-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_process_auto_apply",
|
||
"schedule": 86400.0, # auto-tag wip/editor process art (#1464);
|
||
# no-op unless process_auto_apply_enabled (opt-in)
|
||
},
|
||
"soft-wip-conflict-audit-daily": {
|
||
"task": "backend.app.tasks.ml.scheduled_soft_wip_conflict_audit",
|
||
"schedule": 86400.0, # flag ring-loud soft-WIP (sketch/doodle) tags
|
||
# for review (#1474); no-op with no content heads
|
||
},
|
||
"prune-presentation-reviews-daily": {
|
||
"task": "backend.app.tasks.ml.prune_presentation_reviews",
|
||
"schedule": 86400.0, # retention: drop resolved review flags >30d
|
||
},
|
||
"translate-posts-8h": {
|
||
"task": "backend.app.tasks.translation.translate_posts",
|
||
"schedule": 28800.0, # every 8h: steady-state cadence for the
|
||
# trickle of newly-imported posts (no-op unless translation
|
||
# configured + healthy). One bounded 300-chunk per fire — the
|
||
# one-time backlog drains via the "Translate now" button
|
||
# (drain=True, run-until-done), not this sweep.
|
||
},
|
||
"snapshot-head-metrics-daily": {
|
||
"task": "backend.app.tasks.maintenance.snapshot_head_metrics",
|
||
"schedule": 86400.0,
|
||
},
|
||
"group-discord-drops-hourly": {
|
||
"task": "backend.app.tasks.maintenance.group_discord_drops",
|
||
"schedule": 3600.0, # hourly. Not daily: the grouping signal is
|
||
# the SigLIP embedding, which lands asynchronously AFTER import
|
||
# (#388 E2), so this sweep is what picks up a drop once its
|
||
# vectors have caught up. No-op unless discord_grouping_enabled.
|
||
},
|
||
"match-post-associations-hourly": {
|
||
"task": "backend.app.tasks.maintenance.match_post_associations",
|
||
"schedule": 3600.0, # hourly, and AFTER the grouper's own cadence
|
||
# by construction: a pair cannot be proposed until the drop it
|
||
# points at exists as a grouping (#388 E5). No-op unless
|
||
# discord_link_enabled.
|
||
},
|
||
"sync-memberships-daily": {
|
||
"task": "backend.app.tasks.maintenance.sync_memberships",
|
||
"schedule": 86400.0, # daily — memberships change on a BILLING
|
||
# cycle, not a download cadence (#387 C3). No-op per platform
|
||
# when the client lacks the seam or no credential exists.
|
||
},
|
||
"integrity-verify-weekly": {
|
||
"task": "backend.app.tasks.maintenance.verify_integrity",
|
||
"schedule": 604800.0, # weekly
|
||
},
|
||
"fc3d-tick-due-sources": {
|
||
"task": "backend.app.tasks.scan.tick_due_sources",
|
||
"schedule": 60.0, # every minute
|
||
},
|
||
"fc3d-cleanup-download-events": {
|
||
"task": "backend.app.tasks.maintenance.cleanup_old_download_events",
|
||
"schedule": 86400.0, # daily
|
||
},
|
||
"recover-stalled-download-events": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_download_events",
|
||
"schedule": 300.0, # every 5 min, matches recover-interrupted-tasks
|
||
},
|
||
"recover-stalled-task-runs": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_task_runs",
|
||
"schedule": 300.0, # every 5 min, matches recover-interrupted-tasks
|
||
},
|
||
"prune-task-runs": {
|
||
"task": "backend.app.tasks.maintenance.prune_task_runs",
|
||
"schedule": 86400.0, # daily
|
||
},
|
||
"vacuum-analyze": {
|
||
"task": "backend.app.tasks.maintenance.vacuum_analyze",
|
||
"schedule": 604800.0, # weekly — reclaim dead-tuple bloat + refresh stats
|
||
},
|
||
"fc3h-backup-db-nightly": {
|
||
"task": "backend.app.tasks.backup.backup_db_nightly",
|
||
"schedule": 3600.0, # hourly tick; task self-gates on configured UTC hour
|
||
},
|
||
"fc3h-prune-backups": {
|
||
"task": "backend.app.tasks.backup.prune_backups",
|
||
"schedule": 86400.0, # daily
|
||
},
|
||
# Audit 2026-06-02 — three new per-entity recovery sweeps.
|
||
# Each runs every 5 min like the other recover_stalled_*
|
||
# sweeps; each is a no-op when nothing is stuck.
|
||
"recover-stalled-backup-runs": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_backup_runs",
|
||
"schedule": 300.0,
|
||
},
|
||
"recover-stalled-library-audit-runs": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
|
||
"schedule": 300.0,
|
||
},
|
||
"recover-stalled-head-training-runs": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_head_training_runs",
|
||
"schedule": 300.0,
|
||
},
|
||
"recover-stalled-head-auto-apply-runs": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs",
|
||
"schedule": 300.0,
|
||
},
|
||
"recover-stalled-import-batches": {
|
||
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
|
||
"schedule": 300.0,
|
||
},
|
||
# Audit 2026-06-02 — daily retention for two entities
|
||
# whose terminal rows otherwise accumulate forever.
|
||
"prune-library-audit-runs": {
|
||
"task": "backend.app.tasks.maintenance.prune_library_audit_runs",
|
||
"schedule": 86400.0,
|
||
},
|
||
"prune-import-batches": {
|
||
"task": "backend.app.tasks.maintenance.prune_import_batches",
|
||
"schedule": 86400.0,
|
||
},
|
||
# Audit 2026-06-02 — backfill_thumbnails's docstring claimed
|
||
# "periodic Beat" but the entry was never registered, so the
|
||
# library got no self-healing thumbnail repair; only the
|
||
# manual admin-UI button fired it. Daily cadence is gentle
|
||
# (the task is idempotent and only enqueues regen for rows
|
||
# whose stored thumbnails are missing or corrupt).
|
||
"backfill-thumbnails-daily": {
|
||
"task": "backend.app.tasks.thumbnail.backfill_thumbnails",
|
||
"schedule": 86400.0,
|
||
},
|
||
# External file-host downloads (#830): a steady sweep catches links
|
||
# the post-download hook missed (worker down, etc.); recovery re-tries
|
||
# dead links daily; retention prunes long-dead rows.
|
||
"extdl-sweep": {
|
||
"task": "backend.app.tasks.external.sweep_external_links",
|
||
"schedule": 600.0, # every 10 min
|
||
},
|
||
"extdl-recover-daily": {
|
||
"task": "backend.app.tasks.external.recover_external_links",
|
||
"schedule": 86400.0,
|
||
},
|
||
"extdl-prune-daily": {
|
||
"task": "backend.app.tasks.external.prune_external_links",
|
||
"schedule": 86400.0,
|
||
},
|
||
},
|
||
timezone="UTC",
|
||
)
|
||
# FC-3i: register task_run signal handlers (side-effect import).
|
||
from . import celery_signals # noqa: F401
|
||
|
||
return app
|
||
|
||
|
||
celery = make_celery()
|