Near-duplicate dedup rebuilt on three gates, backup credential leak closed, library consolidation started #255

Merged

bvandeusen merged 8 commits from dev into main 2026-09-21 11:11:39 -04:00

Conversation 0 Commits 8 Files Changed 22

+1348 -53

8 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	9728407539	docs: record why the misplaced-rows LIKE needs no escaping (4245) ... `startswith` compiles to LIKE, where `_` and `%` are wildcards, and the call does not escape them. That is safe only because `slugify` reduces a slug to [a-z0-9-] — an invariant living in a different module, which is exactly the kind of thing that gets widened later without anyone connecting the two. Worth naming because `poch4n_art` is a real directory here: if slugs ever carried underscores, that prefix would start matching `poch4nXart` and the sweep would quietly mis-file one artist's rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 11:05:37 -04:00
bvandeusenandClaude Opus 5	fc982f74b9	feat: survey which image rows sit outside their artist's canonical directory (4245) ... Step 2 of milestone #421. The disk survey counted FOLDERS; this counts ROWS, which is the number that matters — every move in step 3 is a row update, and `ImageRecord.path` is the only pointer at the bytes. `library_layout.py` holds the decision in two shared pieces, and both halves of the consolidation spread them rather than restating them (rule 93, the _x_conditions shape from snippet #3087): - `_misplaced_conditions(root, artist_id, slug)` — rows of one artist whose file is not under that artist's directory. The prefix carries a trailing separator deliberately: without it `ara` matches everything under `arbuzbudesh/`, and one artist reads as fully placed while another's rows are silently skipped. Both are real artists here, hence the test. - `destination_for(path, root, slug)` — where a row's file belongs, or None when it must not be moved: outside the images root, or under one of the reserved stores (`thumbs`, `attachments`, `cookies`, `secrets`, `_backups`, `_quarantine`). Relocating those would move the thumbnail cache or the credential key into an artist folder. `destination_for` diverges from `canonical_subdir` in exactly one case, and the docstring says why: a file at the images ROOT with a known artist moves under that artist here, where the import-time helper leaves it alone. The two answer different questions — an empty subdir at import means no artist was resolved, while a row that already carries an artist_id is an anomaly with a known correct home. The 660 unattributed files have no artist_id at all, so no predicate reaches them; they are counted and left for task #4247. `GET /api/cleanup/layout` exposes it. `?check_disk=1` additionally stats every destination for collisions and missing sources — the conditions the apply refuses on — but it is off by default so the count-only pass answers "how big is this" in seconds instead of timing the request out on NFS. Nothing here writes; a test asserts that against both the row and the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 11:03:03 -04:00
bvandeusenandClaude Opus 5	01c906e6ad	fix: ruff I001 — canonical_subdir sorts before derive_subdir ... Rule 102's import-order trap, hit by inserting into the block rather than re-sorting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:49:38 -04:00
bvandeusenandClaude Opus 5	30337a6c11	fix: library paths follow the artist's slug, not the import folder's name (4244) ... Step 1 of milestone #421. The images tree has 57 directory families for what the database says are single artists — `Conto`/`conto`, `InCaseArt`/`incaseart`, `StickySpoodge`/`Stickyspoodge`/`stickyspoodge`, and so on down to a four-way split for Pocket Ace Games. There was never a duplicate Artist row. `/api/artists/names` returns exactly one per artist. The files simply get written to two places for one row: `_copy_to_library` built its destination from `derive_subdir`, which mirrors the IMPORT tree's folder name verbatim, while the download path leaves files where the ingester wrote them — under the slug. Two writers, two conventions, one artist. This is the half that stops it re-growing, and it has to land before anything moves existing files: consolidate first and the next filesystem import out of a capitalised folder re-creates the directory that was just emptied. `canonical_subdir` replaces the top-level segment with the artist's slug and leaves everything below it alone — the post hierarchy is the downloader's business. Two deliberate pass-throughs: no resolved artist (nothing authoritative to canonicalise against) and an empty subdir (a file at the images root, whose fate is task #4247, not a side effect of this helper). `_supersede` resolves the KEPT row's artist for the same reason — a supersede rewrites `existing.path`, so writing it anywhere else would move a row back out of the tree being consolidated. ImageRecord carries `artist_id` with no relationship attribute, so that is a session lookup rather than an attribute. test_import_one_happy_path pinned the old `Alice/` destination and now pins `alice/` (rule 90). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:48:46 -04:00
bvandeusenandClaude Opus 5	6915cbbbe1	fix: the images backup carried the key to the accounts it backs up (4234) ... Listing a 2026-05 tarball while investigating the 4.3T `_backups` pile showed its second and third entries: images/secrets/ images/secrets/credential_key.b64 That is the key that decrypts the stored Patreon/SubscribeStar session credentials, and `cookies/` sat beside it — both unexcluded, so this was true of every images backup taken today, not just the old ones. An images tarball is supposed to be a media archive; one that carries the operator's account keys is a credential leak wearing a backup's name, in a single file that is easy to copy to another disk or restore somewhere less protected. Encryption at rest buys nothing when the key travels in the same archive. `secrets` and `cookies` join `_backups` and `_quarantine` in one named tuple, each with its reason recorded — the recursion that produced 4.3T of nested tarballs is the cautionary tale for why the list is worth explaining rather than just listing. A restore no longer re-establishes credentials. You sign in again, which is the correct outcome for a media backup. Tests cover both new names and that every exclude stays root-relative — a bare `secrets` would also match an artist folder of that name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:35:35 -04:00
bvandeusenandClaude Opus 5	3313c3b10a	feat: a report that shows what the near-dup gates decide about real artwork (4223) ... The three pixel constants added with the #4223 fix were chosen without ever measuring real files — CI only has synthetic split/solid fixtures, and FC verifies nowhere else. This prints the measurements they should have been chosen from: per pair, the hash distance, the mean drift, the changed-pixel fraction, the verdict, and which gate produced it. It drives the real find_similar with the real confirm rather than restating the decision, so it cannot drift from what the importer does. Read-only: opens files, touches no database. Also splits fingerprint_diff out of fingerprints_match — same computation, now returning the numbers instead of only the boolean, so the report can show how far a pair sat from a limit rather than which side of it it fell on. Runs inside the published :dev image (PIL + imagehash already there, no local env needed) with the art folder mounted read-only — rule 147's channel, so nothing has to reach main to be tried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 06:30:54 -04:00
bvandeusenandClaude Opus 5	84f873eb1b	fix: ruff UP037 — unquote the fingerprint_path return annotation ... PIL's Image is imported at module scope, so the annotation never needed to be a string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:17:15 -04:00
bvandeusenandClaude Opus 5	a05cb66635	fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223) ... The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:16:14 -04:00