feat(beat): self-gating schedules for ML backfill, centroids, auto-accept
Promotes three previously-manual maintenance tasks to Celery Beat schedules
so the user doesn't have to remember to run them:
- ml.backfill daily
- apply_auto_accept_predictions daily
- recompute_all_centroids weekly
Cadences are env-overridable (ML_BACKFILL_EVERY_SECONDS,
AUTO_ACCEPT_EVERY_SECONDS, CENTROIDS_EVERY_SECONDS).
Each task self-gates so a scheduled run is a no-op when there's nothing
to do:
- ml.backfill: already self-gating — its first paginated query returns
zero rows when no image is missing predictions/embeddings, the loop
breaks, and the task returns. No code change.
- apply_auto_accept_predictions: adds a fast-path NOT EXISTS query that
returns immediately when no WD14 prediction at/above the threshold
exists for an unattached, non-rejected (image, tag) pair. The full
walk only fires when fresh predictions have landed since the last run.
Returns {'skipped_no_candidates': True} on the no-op path.
- recompute_all_centroids: tightens the aggregate query to LEFT JOIN
tag_reference_embedding and skip tags whose stored reference_count
already matches current image_tags membership count. Without this gate
the daily-scheduled sweep would re-enqueue a recompute for every
eligible tag every run, contending with tag_and_embed on the ml queue.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -363,12 +363,58 @@ def apply_auto_accept_predictions(batch_size: int = 100) -> dict:
|
||||
# Lazy-import the suggestion service so the maintenance worker doesn't
|
||||
# pay its overhead on unrelated tasks.
|
||||
from app.services.tag_suggestions import (
|
||||
_config, _existing_tag_names, get_suggestions,
|
||||
_DEFAULTS, _config, _existing_tag_names, get_suggestions,
|
||||
)
|
||||
from app.ml.wd14 import MODEL_VERSION as WD14_VER
|
||||
|
||||
cfg = _config()
|
||||
existing_all = _existing_tag_names()
|
||||
|
||||
# Fast-path: if no WD14 prediction at/above the threshold exists for
|
||||
# an image-tag combo that isn't already attached and isn't user-rejected,
|
||||
# there's nothing to do. Daily-scheduled runs will hit this branch on
|
||||
# most days once the library has settled, so the full walk only runs
|
||||
# after fresh predictions land.
|
||||
try:
|
||||
threshold = float(cfg.get('auto_accept_general_threshold',
|
||||
_DEFAULTS['auto_accept_general_threshold']))
|
||||
except (TypeError, ValueError):
|
||||
threshold = float(_DEFAULTS['auto_accept_general_threshold'])
|
||||
|
||||
pending_exists = db.session.execute(text("""
|
||||
SELECT 1
|
||||
FROM image_tag_prediction p
|
||||
WHERE p.confidence >= :thr
|
||||
AND p.tag_category = 'general'
|
||||
AND p.model_version = :wd14_ver
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM image_tags it
|
||||
JOIN tag t ON t.id = it.tag_id
|
||||
WHERE it.image_id = p.image_id
|
||||
AND t.name = p.tag_name
|
||||
AND t.kind = 'user'
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM suggestion_feedback sf
|
||||
WHERE sf.image_id = p.image_id
|
||||
AND sf.tag_name = p.tag_name
|
||||
AND sf.decision = 'rejected'
|
||||
)
|
||||
LIMIT 1
|
||||
"""), {'thr': threshold, 'wd14_ver': WD14_VER}).first()
|
||||
|
||||
if pending_exists is None:
|
||||
log.info(
|
||||
"apply_auto_accept_predictions: no candidates above threshold=%.3f; skipping walk",
|
||||
threshold,
|
||||
)
|
||||
return {
|
||||
'scanned': 0,
|
||||
'images_with_applies': 0,
|
||||
'tags_applied': 0,
|
||||
'skipped_no_candidates': True,
|
||||
}
|
||||
|
||||
scanned = 0
|
||||
images_with_applies = 0
|
||||
tags_applied = 0
|
||||
|
||||
Reference in New Issue
Block a user