feat(beat): self-gating schedules for ML backfill, centroids, auto-accept

Promotes three previously-manual maintenance tasks to Celery Beat schedules
so the user doesn't have to remember to run them:

- ml.backfill                              daily
- apply_auto_accept_predictions            daily
- recompute_all_centroids                  weekly

Cadences are env-overridable (ML_BACKFILL_EVERY_SECONDS,
AUTO_ACCEPT_EVERY_SECONDS, CENTROIDS_EVERY_SECONDS).

Each task self-gates so a scheduled run is a no-op when there's nothing
to do:

- ml.backfill: already self-gating — its first paginated query returns
  zero rows when no image is missing predictions/embeddings, the loop
  breaks, and the task returns. No code change.

- apply_auto_accept_predictions: adds a fast-path NOT EXISTS query that
  returns immediately when no WD14 prediction at/above the threshold
  exists for an unattached, non-rejected (image, tag) pair. The full
  walk only fires when fresh predictions have landed since the last run.
  Returns {'skipped_no_candidates': True} on the no-op path.

- recompute_all_centroids: tightens the aggregate query to LEFT JOIN
  tag_reference_embedding and skip tags whose stored reference_count
  already matches current image_tags membership count. Without this gate
  the daily-scheduled sweep would re-enqueue a recompute for every
  eligible tag every run, contending with tag_and_embed on the ml queue.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-04-28 00:06:09 -04:00
parent 6488dfff1a
commit b7d4998cdb
3 changed files with 107 additions and 10 deletions
+18
View File
@@ -120,6 +120,24 @@ def make_celery(app=None):
'task': 'app.tasks.scan.update_system_stats',
'schedule': 21600, # Every 6 hours
},
# ML-pipeline self-maintenance. Each is internally self-gating:
# backfill returns immediately when nothing's missing,
# recompute_all_centroids only enqueues for tags whose member
# count changed, and apply_auto_accept_predictions short-circuits
# when no above-threshold predictions are unattached.
'ml-backfill-sweep': {
'task': 'app.tasks.ml.backfill',
'schedule': int(os.environ.get('ML_BACKFILL_EVERY_SECONDS', '86400')), # daily
},
'apply-auto-accept-sweep': {
'task': 'app.tasks.maintenance.apply_auto_accept_predictions',
'schedule': int(os.environ.get('AUTO_ACCEPT_EVERY_SECONDS', '86400')), # daily
},
'recompute-centroids-sweep': {
'task': 'app.tasks.ml.recompute_all_centroids',
'schedule': int(os.environ.get('CENTROIDS_EVERY_SECONDS', '604800')), # weekly
},
},
)