bvandeusen
5a0e1bbd03
Item 1 of #3072. Both sweeps issued a single-row pg_insert(image_tag) from inside their per-image loop. Steady state that is nothing; a first sweep over a back-catalogue is one round-trip per applied tag, tens of thousands of them. Each chunk now collects its rows and writes them in one statement. The ticket suggested one insert per chunk PER TAG. A single multi-row VALUES carries every tag at once, so it is one statement per chunk full stop — and the sweeps already accumulate across all heads before they commit, so nothing had to be restructured to allow it. Not a new helper: wip_title.apply_wip_image_tags was already doing the chunked ON CONFLICT DO NOTHING insert, so that shape is extracted to services/image_tag_apply.insert_image_tags and all three writers share it. The extraction deliberately leaves wip_title's pre-SELECT behind rather than pulling it into the shared function — the sweeps don't need it (their `skip` sets already exclude applied and rejected images) and it exists only to produce an accurate count, which the sweeps also compute themselves. So the shared primitive returns nothing: psycopg reports rowcount -1 for a multi-row ON CONFLICT DO NOTHING insert, and a count taken from the statement would be a lie rather than an approximation. Ordering note for the system-tag sweep: tag rows are now written after that chunk's PresentationReview rows rather than interleaved before them. Safe — PresentationReview FKs to image_record and tag, not to image_tag. Chunk size stays 5000: 5000 rows x 3 bound params = 15000, inside Postgres' 65535-parameter ceiling with room to spare. tests/test_image_tag_apply.py covers the primitive directly, since it is now the single place three writers can be wrong at once — most importantly that a re-run never restamps a hand-applied tag's source, which would silently poison head training (it excludes the auto sources). Left alone: _insert_presentation_review is still per-row, and the retract path still deletes per-row. Both operate on sets that are small by construction, unlike the apply path. Refs #3072
122 lines
5.4 KiB
Python
122 lines
5.4 KiB
Python
"""Title-based WIP auto-tagging (task #1458).
|
|
|
|
Deterministic heuristic: when a post's TITLE explicitly declares work-in-progress
|
|
(the artist's own "WIP" / "work in progress" label), the ``wip`` system tag is
|
|
applied to that post's images — a cheap, high-precision complement to the
|
|
image-based ML ``wip`` head. WIP images are excluded from the Explore/gallery
|
|
browse (see gallery_service ``excluded_system_tags``), so honouring the artist's
|
|
own label keeps unfinished pieces out of the main browse right at import.
|
|
|
|
Precision over recall — a false WIP tag HIDES a finished post — so matching is
|
|
token-anchored: ``swipe`` / ``wiped`` / ``wiping`` never trip it (a letter on the
|
|
boundary blocks the match).
|
|
|
|
Sync-only: both consumers (the importer and the backfill Celery task) run on a
|
|
sync Session. Application is idempotent-additive (ON CONFLICT DO NOTHING) and
|
|
stamps a distinct ``image_tag.source`` so a later pass can tell where a wip tag
|
|
came from — the "manual" / "head_auto" / "ccip_auto" / "ml_accepted" provenance
|
|
family gains one member.
|
|
"""
|
|
import re
|
|
|
|
from sqlalchemy import select
|
|
from sqlalchemy.orm import Session
|
|
|
|
from ..models.tag import WIP_SYSTEM_TAG, Tag, image_tag
|
|
from .image_tag_apply import insert_image_tags
|
|
|
|
# image_tag.source stamped on title-heuristic WIP tags — distinct from the other
|
|
# apply sources so provenance stays legible and a future undo can target only these.
|
|
# HARD tier ("WIP"/"work in progress") is high-precision → trains the wip head.
|
|
WIP_TITLE_SOURCE = "wip_title"
|
|
# SOFT tier (sketch/doodle/scribble, #1474) is LOWER-precision — a finished "sketch"
|
|
# is often not WIP. This source is PROVISIONAL (in training_data._AUTO_SOURCES) so it
|
|
# NEVER trains the wip head; a soft-tagged image that also looks like real content is
|
|
# surfaced by the ring-loud audit for review.
|
|
WIP_TITLE_SOFT_SOURCE = "wip_title_soft"
|
|
|
|
# A standalone "WIP" / "W.I.P" token, or the phrase "work in progress"
|
|
# (space/underscore/hyphen separated). The letter-boundary lookarounds are what
|
|
# make this precision-first: `s|wip|e`, `|wip|ed`, `|wip|ing` all have a letter
|
|
# abutting the token, so they're rejected. A trailing digit is allowed so
|
|
# "WIP2" (= WIP part 2) still matches.
|
|
_WIP_RE = re.compile(
|
|
r"(?<![A-Za-z])(?:w\.?i\.?p\.?|work[\s_-]+in[\s_-]+progress)(?![A-Za-z])",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
# Soft tier: sketch / doodle / scribble (+ plurals), letter-boundary anchored so
|
|
# "sketchbook" / "kadoodle" don't trip it. Deliberately conservative — recall is
|
|
# secondary because the soft source doesn't train the head and the ring-loud audit
|
|
# catches false positives.
|
|
_SOFT_WIP_RE = re.compile(
|
|
r"(?<![A-Za-z])(?:sketch|sketches|doodle|doodles|scribble|scribbles)(?![A-Za-z])",
|
|
re.IGNORECASE,
|
|
)
|
|
|
|
# Coarse SQL prefilters for the backfill sweep — narrow the post scan to rows that
|
|
# COULD match before the precise regex confirms. Case-insensitive ILIKE patterns.
|
|
# Each MUST stay a SUPERSET of its regex or the sweep would silently miss posts.
|
|
WIP_TITLE_SQL_PREFILTER = ("%wip%", "%work%progress%")
|
|
SOFT_WIP_TITLE_SQL_PREFILTER = ("%sketch%", "%doodle%", "%scribble%")
|
|
|
|
# Chunk bulk inserts so a large sweep can't blow past psycopg's 65535-parameter
|
|
# ceiling (3 params/row → ~21k rows max; 5k stays comfortably under).
|
|
_INSERT_CHUNK = 5000
|
|
|
|
|
|
def matches_wip_title(title: str | None) -> bool:
|
|
"""True when a post title explicitly marks it work-in-progress (HARD tier)."""
|
|
if not title:
|
|
return False
|
|
return _WIP_RE.search(title) is not None
|
|
|
|
|
|
def matches_soft_wip_title(title: str | None) -> bool:
|
|
"""True when a title carries a SOFT WIP cue (sketch/doodle/scribble, #1474)."""
|
|
if not title:
|
|
return False
|
|
return _SOFT_WIP_RE.search(title) is not None
|
|
|
|
|
|
def resolve_wip_tag_id(session: Session) -> int | None:
|
|
"""The seeded ``wip`` system tag's id (migration 0075), or None if absent."""
|
|
return session.execute(
|
|
select(Tag.id).where(Tag.name == WIP_SYSTEM_TAG, Tag.is_system.is_(True))
|
|
).scalar_one_or_none()
|
|
|
|
|
|
def apply_wip_image_tags(
|
|
session: Session, image_ids, tag_id: int, *, source: str = WIP_TITLE_SOURCE
|
|
) -> int:
|
|
"""Attach ``tag_id`` (stamped with ``source``) to each image id, idempotently —
|
|
never disturbs an existing tag or its source. Returns the number of image_tag
|
|
rows newly inserted. Does NOT commit.
|
|
|
|
The insert count is computed from a pre-SELECT of already-tagged ids rather
|
|
than the statement's ``rowcount``: psycopg reports -1 for a multi-row
|
|
ON CONFLICT DO NOTHING insert (it runs via an executemany path), so rowcount
|
|
is unusable here. The SELECT is accurate within this single transaction (no
|
|
concurrent writer touches these (image, wip) rows); ON CONFLICT DO NOTHING
|
|
stays as a race-safety belt so a rare concurrent insert can't error."""
|
|
ids = list({int(i) for i in image_ids})
|
|
if not ids:
|
|
return 0
|
|
inserted = 0
|
|
for start in range(0, len(ids), _INSERT_CHUNK):
|
|
chunk = ids[start:start + _INSERT_CHUNK]
|
|
already = set(session.execute(
|
|
select(image_tag.c.image_record_id)
|
|
.where(image_tag.c.tag_id == tag_id)
|
|
.where(image_tag.c.image_record_id.in_(chunk))
|
|
).scalars())
|
|
to_insert = [iid for iid in chunk if iid not in already]
|
|
if not to_insert:
|
|
continue
|
|
insert_image_tags(session, [
|
|
{"image_record_id": iid, "tag_id": tag_id, "source": source}
|
|
for iid in to_insert
|
|
])
|
|
inserted += len(to_insert)
|
|
return inserted
|