fix(ml): preserve digit-only tag names in normalize (year tags)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 14s
CI / frontend-build (push) Successful in 21s
CI / intimp (push) Successful in 3m45s
CI / intapi (push) Successful in 7m42s
CI / intcore (push) Successful in 8m12s

Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
This commit is contained in:
2026-06-03 13:09:35 -04:00
parent a6e8d4b52e
commit e450145304
+4 -3
View File
@@ -18,7 +18,8 @@ Rules (operator-approved 2026-06-03):
6. Preserve hyphens (booru hyphens often carry meaning) 6. Preserve hyphens (booru hyphens often carry meaning)
7. Title-case each space-separated word (first character only — 7. Title-case each space-separated word (first character only —
apostrophes, digits, hyphens stay) apostrophes, digits, hyphens stay)
8. If no letters remain, return None (drop emoticons like `:/`) 8. If no letters AND no digits remain, return None (drops emoticons
like `:/` or `^_^`; preserves bare digit tags like `2005`)
9. No surname/givenname swap — no reliable signal in the vocab 9. No surname/givenname swap — no reliable signal in the vocab
""" """
@@ -28,7 +29,7 @@ _LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$") _TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
_MULTISPACE = re.compile(r"\s+") _MULTISPACE = re.compile(r"\s+")
_COLON_NOSPACE = re.compile(r":(?=\S)") _COLON_NOSPACE = re.compile(r":(?=\S)")
_HAS_LETTER = re.compile(r"[A-Za-z]") _HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
def _strip_wrapping_quotes(s: str) -> str: def _strip_wrapping_quotes(s: str) -> str:
@@ -56,6 +57,6 @@ def normalize(raw: str) -> str | None:
s = s.replace("_", " ") s = s.replace("_", " ")
s = _COLON_NOSPACE.sub(": ", s) s = _COLON_NOSPACE.sub(": ", s)
s = _MULTISPACE.sub(" ", s).strip() s = _MULTISPACE.sub(" ", s).strip()
if not s or not _HAS_LETTER.search(s): if not s or not _HAS_ALPHANUMERIC.search(s):
return None return None
return " ".join(_title_word(w) for w in s.split(" ")) return " ".join(_title_word(w) for w in s.split(" "))