fix(ml): preserve digit-only tag names in normalize (year tags)
Rule 8 'no letters -> drop' was over-eager: bare digit tags like '2005'
returned None even though they're legitimate (booru year-tag shape).
Widen the keep-condition to any alphanumeric. Emoticons (':/', '^_^',
'+_+') still drop since they contain neither letters nor digits.
This commit is contained in:
@@ -18,7 +18,8 @@ Rules (operator-approved 2026-06-03):
|
|||||||
6. Preserve hyphens (booru hyphens often carry meaning)
|
6. Preserve hyphens (booru hyphens often carry meaning)
|
||||||
7. Title-case each space-separated word (first character only —
|
7. Title-case each space-separated word (first character only —
|
||||||
apostrophes, digits, hyphens stay)
|
apostrophes, digits, hyphens stay)
|
||||||
8. If no letters remain, return None (drop emoticons like `:/`)
|
8. If no letters AND no digits remain, return None (drops emoticons
|
||||||
|
like `:/` or `^_^`; preserves bare digit tags like `2005`)
|
||||||
9. No surname/givenname swap — no reliable signal in the vocab
|
9. No surname/givenname swap — no reliable signal in the vocab
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -28,7 +29,7 @@ _LEADING_JUNK = re.compile(r"^[#.+;~_\s]+")
|
|||||||
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
|
_TRAILING_DISAMBIG = re.compile(r"_\([^)]*\)\s*$")
|
||||||
_MULTISPACE = re.compile(r"\s+")
|
_MULTISPACE = re.compile(r"\s+")
|
||||||
_COLON_NOSPACE = re.compile(r":(?=\S)")
|
_COLON_NOSPACE = re.compile(r":(?=\S)")
|
||||||
_HAS_LETTER = re.compile(r"[A-Za-z]")
|
_HAS_ALPHANUMERIC = re.compile(r"[A-Za-z0-9]")
|
||||||
|
|
||||||
|
|
||||||
def _strip_wrapping_quotes(s: str) -> str:
|
def _strip_wrapping_quotes(s: str) -> str:
|
||||||
@@ -56,6 +57,6 @@ def normalize(raw: str) -> str | None:
|
|||||||
s = s.replace("_", " ")
|
s = s.replace("_", " ")
|
||||||
s = _COLON_NOSPACE.sub(": ", s)
|
s = _COLON_NOSPACE.sub(": ", s)
|
||||||
s = _MULTISPACE.sub(" ", s).strip()
|
s = _MULTISPACE.sub(" ", s).strip()
|
||||||
if not s or not _HAS_LETTER.search(s):
|
if not s or not _HAS_ALPHANUMERIC.search(s):
|
||||||
return None
|
return None
|
||||||
return " ".join(_title_word(w) for w in s.split(" "))
|
return " ".join(_title_word(w) for w in s.split(" "))
|
||||||
|
|||||||
Reference in New Issue
Block a user