feat(beat): self-gating schedules for ML backfill, centroids, auto-accept

Promotes three previously-manual maintenance tasks to Celery Beat schedules
so the user doesn't have to remember to run them:

- ml.backfill                              daily
- apply_auto_accept_predictions            daily
- recompute_all_centroids                  weekly

Cadences are env-overridable (ML_BACKFILL_EVERY_SECONDS,
AUTO_ACCEPT_EVERY_SECONDS, CENTROIDS_EVERY_SECONDS).

Each task self-gates so a scheduled run is a no-op when there's nothing
to do:

- ml.backfill: already self-gating — its first paginated query returns
  zero rows when no image is missing predictions/embeddings, the loop
  breaks, and the task returns. No code change.

- apply_auto_accept_predictions: adds a fast-path NOT EXISTS query that
  returns immediately when no WD14 prediction at/above the threshold
  exists for an unattached, non-rejected (image, tag) pair. The full
  walk only fires when fresh predictions have landed since the last run.
  Returns {'skipped_no_candidates': True} on the no-op path.

- recompute_all_centroids: tightens the aggregate query to LEFT JOIN
  tag_reference_embedding and skip tags whose stored reference_count
  already matches current image_tags membership count. Without this gate
  the daily-scheduled sweep would re-enqueue a recompute for every
  eligible tag every run, contending with tag_and_embed on the ml queue.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-04-28 00:06:09 -04:00
parent 6488dfff1a
commit b7d4998cdb
3 changed files with 107 additions and 10 deletions
+47 -1
View File
@@ -363,12 +363,58 @@ def apply_auto_accept_predictions(batch_size: int = 100) -> dict:
# Lazy-import the suggestion service so the maintenance worker doesn't
# pay its overhead on unrelated tasks.
from app.services.tag_suggestions import (
_config, _existing_tag_names, get_suggestions,
_DEFAULTS, _config, _existing_tag_names, get_suggestions,
)
from app.ml.wd14 import MODEL_VERSION as WD14_VER
cfg = _config()
existing_all = _existing_tag_names()
# Fast-path: if no WD14 prediction at/above the threshold exists for
# an image-tag combo that isn't already attached and isn't user-rejected,
# there's nothing to do. Daily-scheduled runs will hit this branch on
# most days once the library has settled, so the full walk only runs
# after fresh predictions land.
try:
threshold = float(cfg.get('auto_accept_general_threshold',
_DEFAULTS['auto_accept_general_threshold']))
except (TypeError, ValueError):
threshold = float(_DEFAULTS['auto_accept_general_threshold'])
pending_exists = db.session.execute(text("""
SELECT 1
FROM image_tag_prediction p
WHERE p.confidence >= :thr
AND p.tag_category = 'general'
AND p.model_version = :wd14_ver
AND NOT EXISTS (
SELECT 1 FROM image_tags it
JOIN tag t ON t.id = it.tag_id
WHERE it.image_id = p.image_id
AND t.name = p.tag_name
AND t.kind = 'user'
)
AND NOT EXISTS (
SELECT 1 FROM suggestion_feedback sf
WHERE sf.image_id = p.image_id
AND sf.tag_name = p.tag_name
AND sf.decision = 'rejected'
)
LIMIT 1
"""), {'thr': threshold, 'wd14_ver': WD14_VER}).first()
if pending_exists is None:
log.info(
"apply_auto_accept_predictions: no candidates above threshold=%.3f; skipping walk",
threshold,
)
return {
'scanned': 0,
'images_with_applies': 0,
'tags_applied': 0,
'skipped_no_candidates': True,
}
scanned = 0
images_with_applies = 0
tags_applied = 0