Near-duplicate dedup rebuilt on three gates, backup credential leak closed, library consolidation started #255

Merged

bvandeusen merged 8 commits from dev into main 2026-09-21 11:11:39 -04:00

Conversation 0 Commits 8 Files Changed 22

+1348 -53

bvandeusen commented 2026-09-21 11:11:33 -04:00

Owner

Copy Link

Copy Source

Eight commits, all from 2026-09-21. dev tip 9728407 is green on both
workflows (runs 7168/7169).

1. Variant artwork stopped being deleted as a near-duplicate (#4223)

A 15-image variant pack was landing as 3 records, and variants were still
being dropped with phash_threshold at 0 — the floor of the dial. No
setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse
light/dark layout, so two variants sharing a composition produce the SAME
bits. Distance 0 meant "identical hash", never "identical image".

The hash no longer decides a merge alone. find_similar runs three gates:
the threshold PROPOSES candidates, aspect ratio rejects crops and
re-canvases, and a pixel-level confirm ACCEPTS. Every gate fails closed —
too strict keeps a redundant copy the operator can see, too loose deletes
artwork only a source re-walk returns.

Measured on the operator's real library (InCase, "Office Girlfriend"):
rescales of one file land at mean 0.11-0.15 against a limit of 6.0 and merge;
a true 9A/9C variant pair sits at 26.2 and is kept. Roughly a 100x gap, with
the limit nearer the rescale side.

2. The images backup carried the key to the accounts it backs up (#4234)

backup_service.backup_images excluded only _backups and _quarantine, so
every images tarball also contained secrets/credential_key.b64 — the key
that decrypts the stored Patreon/SubscribeStar cookies — and cookies/
itself. Found by listing an old tarball while investigating #4233. Both are
now excluded; a restore no longer re-establishes credentials, which is the
correct behaviour for a media archive.

3. Library consolidation, steps 1-2 of milestone #421

The images tree has 57 directory families for what the database says are
single artists. There was never a duplicate Artist row — _copy_to_library
named the destination after the IMPORT folder while the downloader wrote
under the slug.

• Step 1 — library paths now derive from the artist's slug. This is the
  half that stops it re-growing and had to land before anything moves files.
• Step 2 — library_layout.py plus GET /api/cleanup/layout, read-only,
  reporting which rows sit outside their artist's directory. The predicate it
  uses is the one step 3's apply will spread, not restate (rule 93).

No files are moved by this PR. Step 3 (the move tool + Maintenance UI) is
still to come.

What happens on deploy — read this part

Migration 0098 runs and NULLs every image_record.phash. That is
deliberate: the stored values are 64-bit hashes of images the app now hashes
at 256 bits, and the two cannot be compared. backfill_phash is NULL-only
and is now on the daily beat, so the library re-hashes itself — but until it
finishes, image dedup degrades to sha256 only and duplicates may land.
That is the safe direction and the only one available; the alternative
compares hashes of different widths and drops artwork.

0098 also resets phash_threshold to the new default of 24. The unit
changed (bits out of 64 -> 256), so the old number would have kept its value
while meaning something four times tighter. There is no honest carry-over.

Order of operations afterwards: deploy -> 0098 -> let the backfill finish ->
only THEN run a per-source "Recover dropped near-duplicates" walk, since a
recovery that outruns the backfill re-imports against NULL hashes.

Separately: those tarballs carried credential_key.b64 for four months. They
have been deleted (#4233 reclaimed 4.2 TB), but rotating that key is the
conservative follow-up — it invalidates stored cookies and forces a re-auth
per source, so it is the operator's call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

Eight commits, all from 2026-09-21. `dev` tip `9728407` is green on both workflows (runs 7168/7169). ## 1. Variant artwork stopped being deleted as a near-duplicate (#4223) A 15-image variant pack was landing as 3 records, and variants were still being dropped with `phash_threshold` at **0** — the floor of the dial. No setting could have fixed it: at `hash_size=8` a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", never "identical image". The hash no longer decides a merge alone. `find_similar` runs three gates: the threshold PROPOSES candidates, aspect ratio rejects crops and re-canvases, and a pixel-level confirm ACCEPTS. Every gate fails closed — too strict keeps a redundant copy the operator can see, too loose deletes artwork only a source re-walk returns. Measured on the operator's real library (InCase, "Office Girlfriend"): rescales of one file land at mean 0.11-0.15 against a limit of 6.0 and merge; a true 9A/9C variant pair sits at 26.2 and is kept. Roughly a 100x gap, with the limit nearer the rescale side. ## 2. The images backup carried the key to the accounts it backs up (#4234) `backup_service.backup_images` excluded only `_backups` and `_quarantine`, so every images tarball also contained `secrets/credential_key.b64` — the key that decrypts the stored Patreon/SubscribeStar cookies — and `cookies/` itself. Found by listing an old tarball while investigating #4233. Both are now excluded; a restore no longer re-establishes credentials, which is the correct behaviour for a media archive. ## 3. Library consolidation, steps 1-2 of milestone #421 The images tree has 57 directory families for what the database says are single artists. There was never a duplicate Artist row — `_copy_to_library` named the destination after the IMPORT folder while the downloader wrote under the slug. - **Step 1** — library paths now derive from the artist's slug. This is the half that stops it re-growing and had to land before anything moves files. - **Step 2** — `library_layout.py` plus `GET /api/cleanup/layout`, read-only, reporting which rows sit outside their artist's directory. The predicate it uses is the one step 3's apply will spread, not restate (rule 93). No files are moved by this PR. Step 3 (the move tool + Maintenance UI) is still to come. ## What happens on deploy — read this part **Migration 0098 runs and NULLs every `image_record.phash`.** That is deliberate: the stored values are 64-bit hashes of images the app now hashes at 256 bits, and the two cannot be compared. `backfill_phash` is NULL-only and is now on the daily beat, so the library re-hashes itself — but until it finishes, **image dedup degrades to sha256 only** and duplicates may land. That is the safe direction and the only one available; the alternative compares hashes of different widths and drops artwork. 0098 also resets `phash_threshold` to the new default of 24. The unit changed (bits out of 64 -> 256), so the old number would have kept its value while meaning something four times tighter. There is no honest carry-over. Order of operations afterwards: deploy -> 0098 -> let the backfill finish -> only THEN run a per-source "Recover dropped near-duplicates" walk, since a recovery that outruns the backfill re-imports against NULL hashes. Separately: those tarballs carried `credential_key.b64` for four months. They have been deleted (#4233 reclaimed 4.2 TB), but rotating that key is the conservative follow-up — it invalidates stored cookies and forces a re-auth per source, so it is the operator's call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

bvandeusen added 8 commits 2026-09-21 11:11:33 -04:00

fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223) ... a05cb66635

The operator reported a 15-image variant pack landing as 3 records, then
reported variants STILL being dropped with phash_threshold at 0 — the floor
of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64
bits of coarse light/dark layout, so two variants sharing a composition
produce the SAME bits. Distance 0 meant "identical hash", not "identical
image", and the dial was simultaneously too coarse to keep variants and too
tight to catch a re-encoded rescale.

The hash no longer decides a merge on its own. find_similar now runs three
gates, cheapest first: the threshold proposes candidates, aspect ratio
(ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases,
and a pixel-level confirm on the two files accepts. Every gate fails closed —
unknown dimensions, an unreadable candidate, a hash of the wrong width all
mean "not a duplicate", because too strict keeps a redundant copy the
operator can see while too loose deletes artwork only a source re-walk
returns.

- utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used);
  aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only,
  mean drift + changed-pixel fraction), find_similar gains `confirm`.
- importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and
  cached, so a non-matching import costs no extra I/O.
- 0098: widens image_record.phash to 64 chars and NULLs every value — a
  stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash
  is NULL-only, keyset-paginated and now on the daily beat, so the library
  re-hashes itself. Dedup degrades to sha256 until it finishes.
- phash_threshold counts bits and the denominator went 64 -> 256, so the
  setting is reset to the new default of 24 (there is no honest carry-over)
  and the slider is rescaled to 0-64.
- gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so
  the Explore rail keeps the variance the operator tuned in on 2026-07-01.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

fix: ruff UP037 — unquote the fingerprint_path return annotation ... 84f873eb1b

PIL's Image is imported at module scope, so the annotation never needed to
be a string.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

feat: a report that shows what the near-dup gates decide about real artwork (4223) ... 3313c3b10a

The three pixel constants added with the #4223 fix were chosen without ever
measuring real files — CI only has synthetic split/solid fixtures, and FC
verifies nowhere else. This prints the measurements they should have been
chosen from: per pair, the hash distance, the mean drift, the changed-pixel
fraction, the verdict, and which gate produced it.

It drives the real find_similar with the real confirm rather than restating
the decision, so it cannot drift from what the importer does. Read-only:
opens files, touches no database.

Also splits fingerprint_diff out of fingerprints_match — same computation,
now returning the numbers instead of only the boolean, so the report can show
how far a pair sat from a limit rather than which side of it it fell on.

Runs inside the published :dev image (PIL + imagehash already there, no local
env needed) with the art folder mounted read-only — rule 147's channel, so
nothing has to reach main to be tried.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

fix: the images backup carried the key to the accounts it backs up (4234) ... 6915cbbbe1

Listing a 2026-05 tarball while investigating the 4.3T `_backups` pile showed
its second and third entries:

    images/secrets/
    images/secrets/credential_key.b64

That is the key that decrypts the stored Patreon/SubscribeStar session
credentials, and `cookies/` sat beside it — both unexcluded, so this was true
of every images backup taken today, not just the old ones. An images tarball
is supposed to be a media archive; one that carries the operator's account
keys is a credential leak wearing a backup's name, in a single file that is
easy to copy to another disk or restore somewhere less protected. Encryption
at rest buys nothing when the key travels in the same archive.

`secrets` and `cookies` join `_backups` and `_quarantine` in one named tuple,
each with its reason recorded — the recursion that produced 4.3T of nested
tarballs is the cautionary tale for why the list is worth explaining rather
than just listing.

A restore no longer re-establishes credentials. You sign in again, which is
the correct outcome for a media backup.

Tests cover both new names and that every exclude stays root-relative — a bare
`secrets` would also match an artist folder of that name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

fix: library paths follow the artist's slug, not the import folder's name (4244) ... 30337a6c11

Step 1 of milestone #421. The images tree has 57 directory families for what
the database says are single artists — `Conto`/`conto`, `InCaseArt`/`incaseart`,
`StickySpoodge`/`Stickyspoodge`/`stickyspoodge`, and so on down to a four-way
split for Pocket Ace Games.

There was never a duplicate Artist row. `/api/artists/names` returns exactly
one per artist. The files simply get written to two places for one row:
`_copy_to_library` built its destination from `derive_subdir`, which mirrors
the IMPORT tree's folder name verbatim, while the download path leaves files
where the ingester wrote them — under the slug. Two writers, two conventions,
one artist.

This is the half that stops it re-growing, and it has to land before anything
moves existing files: consolidate first and the next filesystem import out of
a capitalised folder re-creates the directory that was just emptied.

`canonical_subdir` replaces the top-level segment with the artist's slug and
leaves everything below it alone — the post hierarchy is the downloader's
business. Two deliberate pass-throughs: no resolved artist (nothing
authoritative to canonicalise against) and an empty subdir (a file at the
images root, whose fate is task #4247, not a side effect of this helper).

`_supersede` resolves the KEPT row's artist for the same reason — a supersede
rewrites `existing.path`, so writing it anywhere else would move a row back
out of the tree being consolidated. ImageRecord carries `artist_id` with no
relationship attribute, so that is a session lookup rather than an attribute.

test_import_one_happy_path pinned the old `Alice/` destination and now pins
`alice/` (rule 90).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

fix: ruff I001 — canonical_subdir sorts before derive_subdir ... 01c906e6ad

Rule 102's import-order trap, hit by inserting into the block rather than
re-sorting it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

feat: survey which image rows sit outside their artist's canonical directory (4245) ... fc982f74b9

Step 2 of milestone #421. The disk survey counted FOLDERS; this counts ROWS,
which is the number that matters — every move in step 3 is a row update, and
`ImageRecord.path` is the only pointer at the bytes.

`library_layout.py` holds the decision in two shared pieces, and both halves
of the consolidation spread them rather than restating them (rule 93, the
_x_conditions shape from snippet #3087):

- `_misplaced_conditions(root, artist_id, slug)` — rows of one artist whose
  file is not under that artist's directory. The prefix carries a trailing
  separator deliberately: without it `ara` matches everything under
  `arbuzbudesh/`, and one artist reads as fully placed while another's rows
  are silently skipped. Both are real artists here, hence the test.
- `destination_for(path, root, slug)` — where a row's file belongs, or None
  when it must not be moved: outside the images root, or under one of the
  reserved stores (`thumbs`, `attachments`, `cookies`, `secrets`, `_backups`,
  `_quarantine`). Relocating those would move the thumbnail cache or the
  credential key into an artist folder.

`destination_for` diverges from `canonical_subdir` in exactly one case, and
the docstring says why: a file at the images ROOT with a known artist moves
under that artist here, where the import-time helper leaves it alone. The two
answer different questions — an empty subdir at import means no artist was
resolved, while a row that already carries an artist_id is an anomaly with a
known correct home. The 660 unattributed files have no artist_id at all, so
no predicate reaches them; they are counted and left for task #4247.

`GET /api/cleanup/layout` exposes it. `?check_disk=1` additionally stats every
destination for collisions and missing sources — the conditions the apply
refuses on — but it is off by default so the count-only pass answers "how big
is this" in seconds instead of timing the request out on NFS.

Nothing here writes; a test asserts that against both the row and the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

docs: record why the misplaced-rows LIKE needs no escaping (4245) ... 9728407539

`startswith` compiles to LIKE, where `_` and `%` are wildcards, and the call
does not escape them. That is safe only because `slugify` reduces a slug to
[a-z0-9-] — an invariant living in a different module, which is exactly the
kind of thing that gets widened later without anyone connecting the two.

Worth naming because `poch4n_art` is a real directory here: if slugs ever
carried underscores, that prefix would start matching `poch4nXart` and the
sweep would quietly mis-file one artist's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

bvandeusen merged commit e2246e861e into main 2026-09-21 11:11:39 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#255