fix: count the POSTS a working name spans, not the files (4392)
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / integration (push) Successful in 2m19s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Counting files punished a piece for the one thing a working name is
guaranteed to do — have several exports. Measured across the operator's four
dual-platform artists: knuxy carries `p217` on four files spanning exactly two
posts, the Patreon post and the Discord drop, and ~200 comic-page tokens have
that shape. Every one scored half strength. Counting posts scores them 1.00
and still catches the habits, which span many posts rather than many files:
tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4. Knuxy's
ambiguous band drops from 268 tokens to 55.
Three false positives found the same way — by running the module over the real
library rather than reading it:
* The screenshot guard matches from the start of the stem, so it never fired
on the legacy `<post id>_media_<media id>_` era. `Screenshot 2025-07-27
182450ab` sailed through and contributed `2025-07-27` — the same-day date
collision this module's docstring exists to refuse.
* tamadaheijun names screenshots in Japanese. A guard that knows only the
English word is a guard for one artist.
* `timeline 3-0002` spans three unrelated posts. Keeping hyphens inside
tokens for `0-k`'s sake let the number survive whole, so an identity token
now has to contain a letter — which is the property behind the old
bare-year and all-digits rules anyway, and leaves `0-k`, `680lc` and `p59`
untouched.
Also stopwords for `the`, `gif`, `main`, `patreon`, `capture`, `timeline`,
each measured carrying a false match inside the admitted band.
IDENTITY_FLOOR names what a shared name must reach to propose a link with no
corroboration. At 0.75, 13 of artist 8's 15 name-sharing pairs clear it,
including the operator's own example. The two that do not are real pairs this
signal will not carry alone — the stated cost of refusing the four-post band,
where conto's `illustration9` and `maid` also sit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -67,7 +67,7 @@ from pathlib import PurePosixPath
|
||||
# spelling — sailed straight past the guard while the space-separated Patreon
|
||||
# spelling was caught. Found by running this against the live library rather
|
||||
# than by reading it. Assert the next character is not a letter instead.
|
||||
_SCREENSHOT = re.compile(r"^screen[ _-]?shot(?![a-z])", re.I)
|
||||
_SCREENSHOT = re.compile(r"^(?:screen[ _-]?shot(?![a-z])|\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8|\u30b9\u30af\u30b7\u30e7)", re.I)
|
||||
|
||||
# The importer's per-post media index: `01_`, `02_`. Not part of any name.
|
||||
_MEDIA_INDEX = re.compile(r"^\d{1,3}_")
|
||||
@@ -76,6 +76,15 @@ _MEDIA_INDEX = re.compile(r"^\d{1,3}_")
|
||||
# `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}`.
|
||||
_DISCORD_PREFIX = re.compile(r"^\d{8}_\d{6,}_\d{1,3}_")
|
||||
|
||||
# The legacy era (#4002): images sit FLAT at the artist root as
|
||||
# `<post id>_media_<media id>_<name>`. Stripping it is not cosmetic — the
|
||||
# SCREENSHOT guard below matches from the start of the stem, so while this
|
||||
# prefix was left on, a legacy screenshot never looked like one. Measured on
|
||||
# tamadaheijun: `109078417_media_334848471_Screenshot 2025-07-27 182450ab`
|
||||
# sailed through and contributed `2025-07-27`, which is precisely the
|
||||
# same-day date collision this module was built to refuse.
|
||||
_LEGACY_PREFIX = re.compile(r"^\d+_media_\d+_")
|
||||
|
||||
# The importer's content-hash suffix, `__<10 hex>`, sometimes doubled on files
|
||||
# that went through an older import era.
|
||||
_HASH_SUFFIX = re.compile(r"(?:__[0-9a-f]{10})+$")
|
||||
@@ -93,31 +102,70 @@ _DECORATION = re.compile(
|
||||
_STOPWORDS = frozenset({
|
||||
"img", "image", "untitled", "new", "test", "page", "final", "copy",
|
||||
"post", "media", "file", "avatar", "cover", "banner", "icon", "splash",
|
||||
# Each of these was MEASURED carrying a false match in the 3..6 frequency
|
||||
# band, where the rarity gate still admits a token: `capture` across six
|
||||
# unrelated knuxy posts, `the` and `patreon` out of legacy title-derived
|
||||
# names, `main` out of `Anya Main CST`, `timeline` and `gif` out of
|
||||
# tamadaheijun's exports.
|
||||
"the", "gif", "main", "patreon", "capture", "timeline", "screenshot",
|
||||
# Literally the string "None": issue #3999's Discord naming rendered
|
||||
# `{user[name]}` as it for ~1,600 files, so it is the single most common
|
||||
# "name" in the library and identifies nothing.
|
||||
"none",
|
||||
})
|
||||
|
||||
# A bare year, or a date fragment. These are what made the screenshot collision
|
||||
# look like a match, and they are worthless as identity even in a real name.
|
||||
# A bare year: still needed for the TEXT signal, where words and numbers are
|
||||
# tokenised separately.
|
||||
_YEAR = re.compile(r"^(?:19|20)\d{2}$")
|
||||
_ALL_DIGITS = re.compile(r"^\d+$")
|
||||
|
||||
# An identity token must contain a LETTER. This replaces separate "all digits"
|
||||
# and "bare year" rules with the property behind both, and it is the rule that
|
||||
# holds once hyphens are kept inside tokens for `0-k`'s sake: without it
|
||||
# `2025-07-27` and `3-0002` survive as single tokens, and both were measured
|
||||
# linking unrelated posts — the second across three of them, out of
|
||||
# tamadaheijun's `timeline 3-0002` exports.
|
||||
#
|
||||
# `0-k`, `680lc`, `cnni18x` and `p59` all keep a letter and are unaffected.
|
||||
_HAS_LETTER = re.compile(r"[A-Za-z]")
|
||||
|
||||
MIN_TOKEN_LEN = 3
|
||||
|
||||
# A token shared by more than this many of ONE ARTIST's images is a habit, not
|
||||
# an identity — a character name, a series tag, a recurring export preset. The
|
||||
# bar is deliberately low: a working name identifies one piece, so it should
|
||||
# appear on that piece's handful of exports (base, wips, the teaser crop) and
|
||||
# nowhere else. Raising this trades false positives for reach, which is the
|
||||
# wrong direction here — a wrong link asserts two different pieces are one.
|
||||
MAX_TOKEN_FREQUENCY = 6
|
||||
# A token appearing in more than this many of ONE ARTIST's POSTS is a habit,
|
||||
# not an identity — a character name, a series tag, a recurring export preset.
|
||||
#
|
||||
# POSTS, not files, and the difference is not bookkeeping. Counting files
|
||||
# punishes a piece for having many exports, which is the one thing a working
|
||||
# name is GUARANTEED to do. Measured across the operator's four dual-platform
|
||||
# artists: knuxy's comic pages carry `p217` on four files spread over exactly
|
||||
# two posts — the Patreon post and the Discord drop — and file-counting scored
|
||||
# every one of ~200 such tokens at half strength for it. Counting posts scores
|
||||
# them 1.00 while still catching the real habits, which span many posts:
|
||||
# tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4.
|
||||
#
|
||||
# Six rather than two, although two posts IS the shape of a teaser and its
|
||||
# drop, because a creator legitimately revisits one working name: a wip post,
|
||||
# then an alt, then the release. Measured on artist 8, `cnni18x` and `680lc`
|
||||
# each span four posts and both are genuine. IDENTITY_FLOOR below is what
|
||||
# decides how much span a link may carry on its own.
|
||||
MAX_TOKEN_POSTS = 6
|
||||
|
||||
# A shared name at or above this strength is enough to propose a link with NO
|
||||
# corroboration — it is identity evidence, and the whole reason this module
|
||||
# exists is that identity survives where circumstance does not.
|
||||
#
|
||||
# 0.75 is a token spanning three posts or fewer. Measured on artist 8: of the
|
||||
# 15 same-artist pairs that share a name, 13 clear this bar, including the
|
||||
# operator's own example (`0-k`, three posts, 1.3h apart). The two that do not
|
||||
# — `680lc` and `cnni18x`, four posts each, 21h apart — are real pairs this
|
||||
# signal will not carry alone; they are the measured cost of not admitting the
|
||||
# four-post band, where conto's `illustration9` and `maid` also sit.
|
||||
IDENTITY_FLOOR = 0.75
|
||||
|
||||
|
||||
def _strip_prefixes(stem: str) -> str:
|
||||
"""Remove the framing each platform's importer adds around the real name."""
|
||||
stem = _DISCORD_PREFIX.sub("", stem)
|
||||
stem = _LEGACY_PREFIX.sub("", stem)
|
||||
stem = _MEDIA_INDEX.sub("", stem)
|
||||
return _HASH_SUFFIX.sub("", stem)
|
||||
|
||||
@@ -142,22 +190,27 @@ def working_name_tokens(path: str) -> set[str]:
|
||||
tok = _DECORATION.sub("", raw).strip("-")
|
||||
if len(tok) < MIN_TOKEN_LEN:
|
||||
continue
|
||||
if tok in _STOPWORDS or _YEAR.match(tok) or _ALL_DIGITS.match(tok):
|
||||
if tok in _STOPWORDS or not _HAS_LETTER.search(tok):
|
||||
continue
|
||||
out.add(tok)
|
||||
return out
|
||||
|
||||
|
||||
def token_frequencies(paths: Iterable[str]) -> Counter[str]:
|
||||
"""How often each working-name token appears across one artist's images.
|
||||
def token_frequencies(posts: Iterable[Iterable[str]]) -> Counter[str]:
|
||||
"""How many of ONE ARTIST's POSTS each working-name token appears in.
|
||||
|
||||
Takes posts — each an iterable of that post's image paths — rather than a
|
||||
flat list of paths, because the unit of the count is the post. See
|
||||
MAX_TOKEN_POSTS for what that buys; the short version is that a piece with
|
||||
six exports in one post has used its name once.
|
||||
|
||||
Scoped to the ARTIST, not the library: a working name belongs to the person
|
||||
who chose it, and the same string can be one creator's piece and another's
|
||||
boilerplate. Built once per artist per sweep, not per candidate pair.
|
||||
"""
|
||||
counts: Counter[str] = Counter()
|
||||
for p in paths:
|
||||
counts.update(working_name_tokens(p))
|
||||
for paths in posts:
|
||||
counts.update({t for path in paths for t in working_name_tokens(path)})
|
||||
return counts
|
||||
|
||||
|
||||
@@ -187,7 +240,7 @@ def shared_identity(
|
||||
right: Iterable[str],
|
||||
frequencies: Counter[str],
|
||||
*,
|
||||
max_frequency: int = MAX_TOKEN_FREQUENCY,
|
||||
max_frequency: int = MAX_TOKEN_POSTS,
|
||||
) -> tuple[float, str | None]:
|
||||
"""Strength in [0, 1] that two sets of filenames name the SAME piece.
|
||||
|
||||
@@ -197,16 +250,17 @@ def shared_identity(
|
||||
|
||||
Strength is a function of the winning token's rarity within the artist's
|
||||
own library, not of how many tokens matched. One decisive token beats three
|
||||
vague ones, and a token that appears on forty of this artist's images is a
|
||||
vague ones, and a token appearing across forty of this artist's posts is a
|
||||
habit rather than an identity however exactly it matches.
|
||||
|
||||
`frequencies` must be the POST counts from `token_frequencies`.
|
||||
"""
|
||||
shared = {t for t in set(left) & set(right) if frequencies.get(t, 0) <= max_frequency}
|
||||
if not shared:
|
||||
return 0.0, None
|
||||
|
||||
# The rarest shared token decides — one decisive token beats three vague
|
||||
# ones. `frequencies` counts IMAGES here, and a real working name lands on
|
||||
# a few of them: the base, its wips, the teaser crop.
|
||||
# ones.
|
||||
token = min(shared, key=lambda t: (frequencies.get(t, 0), -len(t), t))
|
||||
strength = round(_rarity(max(frequencies.get(token, 1), 1), max_frequency), 4)
|
||||
# A token sitting exactly ON the cap decays to zero, and naming it anyway
|
||||
@@ -248,11 +302,16 @@ _COMMON_TEXT = frozenset({
|
||||
})
|
||||
|
||||
# A marker in more than this many of ONE ARTIST's posts is a signature, not a
|
||||
# tie-back. Tighter than MAX_TOKEN_FREQUENCY because the units differ and so
|
||||
# does the evidence: that one counts a working name across a piece's handful of
|
||||
# EXPORTS, where this counts a public decoration across POSTS. A marker that
|
||||
# ties an announcement to its drop lands on two posts — the two.
|
||||
MAX_MARKER_FREQUENCY = 4
|
||||
# tie-back. A marker tying an announcement to its drop lands on two posts —
|
||||
# the two.
|
||||
#
|
||||
# Tighter than MAX_TOKEN_POSTS. Both count posts, so the numbers are directly
|
||||
# comparable and the gap between them is the claim being made: a working name
|
||||
# is the creator's private label for one piece and may honestly recur as they
|
||||
# revisit it, while a marker is public decoration and stops being evidence the
|
||||
# moment it is reused. Measured on artist 8: 💦 spans 13 posts, the word
|
||||
# "like" 43, and 🌗 — a real tie-back — exactly 2.
|
||||
MAX_MARKER_POSTS = 4
|
||||
|
||||
|
||||
def text_markers(text: str | None) -> set[str]:
|
||||
@@ -291,7 +350,7 @@ def marker_overlap(
|
||||
right: str | None,
|
||||
frequencies: Counter[str],
|
||||
*,
|
||||
max_frequency: int = MAX_MARKER_FREQUENCY,
|
||||
max_frequency: int = MAX_MARKER_POSTS,
|
||||
) -> float:
|
||||
"""Strength in [0, 1] that two texts share a DELIBERATE marker.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user