feat(beat): self-gating schedules for ML backfill, centroids, auto-accept
Promotes three previously-manual maintenance tasks to Celery Beat schedules
so the user doesn't have to remember to run them:
- ml.backfill daily
- apply_auto_accept_predictions daily
- recompute_all_centroids weekly
Cadences are env-overridable (ML_BACKFILL_EVERY_SECONDS,
AUTO_ACCEPT_EVERY_SECONDS, CENTROIDS_EVERY_SECONDS).
Each task self-gates so a scheduled run is a no-op when there's nothing
to do:
- ml.backfill: already self-gating — its first paginated query returns
zero rows when no image is missing predictions/embeddings, the loop
breaks, and the task returns. No code change.
- apply_auto_accept_predictions: adds a fast-path NOT EXISTS query that
returns immediately when no WD14 prediction at/above the threshold
exists for an unattached, non-rejected (image, tag) pair. The full
walk only fires when fresh predictions have landed since the last run.
Returns {'skipped_no_candidates': True} on the no-op path.
- recompute_all_centroids: tightens the aggregate query to LEFT JOIN
tag_reference_embedding and skip tags whose stored reference_count
already matches current image_tags membership count. Without this gate
the daily-scheduled sweep would re-enqueue a recompute for every
eligible tag every run, contending with tag_and_embed on the ml queue.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -120,6 +120,24 @@ def make_celery(app=None):
|
||||
'task': 'app.tasks.scan.update_system_stats',
|
||||
'schedule': 21600, # Every 6 hours
|
||||
},
|
||||
|
||||
# ML-pipeline self-maintenance. Each is internally self-gating:
|
||||
# backfill returns immediately when nothing's missing,
|
||||
# recompute_all_centroids only enqueues for tags whose member
|
||||
# count changed, and apply_auto_accept_predictions short-circuits
|
||||
# when no above-threshold predictions are unattached.
|
||||
'ml-backfill-sweep': {
|
||||
'task': 'app.tasks.ml.backfill',
|
||||
'schedule': int(os.environ.get('ML_BACKFILL_EVERY_SECONDS', '86400')), # daily
|
||||
},
|
||||
'apply-auto-accept-sweep': {
|
||||
'task': 'app.tasks.maintenance.apply_auto_accept_predictions',
|
||||
'schedule': int(os.environ.get('AUTO_ACCEPT_EVERY_SECONDS', '86400')), # daily
|
||||
},
|
||||
'recompute-centroids-sweep': {
|
||||
'task': 'app.tasks.ml.recompute_all_centroids',
|
||||
'schedule': int(os.environ.get('CENTROIDS_EVERY_SECONDS', '604800')), # weekly
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user