2e0f8f8c611c28eed4265b884c63443eac9954c5
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5
|
2e0f8f8c61 |
feat(cleanup): reclaim orphaned attachments — rows and store blobs (#3068)
PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or artist left the row behind rather than taking it. Nothing ever pruned those rows, and nothing in the repo had ever unlinked a file under the attachment store — so both rows and bytes accumulated permanently, invisible to every existing diagnostic. Why a disk->DB reconciliation rather than a row sweep: the store is sha-addressed and idempotent, so ONE blob backs MANY rows. Deleting a row does not free its blob, and since the artist cascade (#3066) now deletes its attachment rows outright, a freed blob has no DB pointer left to find it by. Walking the store and asking "does any row still reference this sha?" catches orphans from every cause, including ones no future delete path will think to report. Preview and apply share `_orphan_attachment_conditions` (rule 93). The dry-run derives its surviving-sha set by NEGATING that same predicate, so it is honest about blobs the delete would free rather than counting them as still-referenced — the one place this was easy to get backwards, so it has its own parity test. Guards, each with a reason: - A blob is written before its row commits, so a just-stored file legitimately has no referencing row. Files under 6h are never judged — same guard and reasoning as ORPHAN_TEMP_MIN_AGE_HOURS. - `.partial` staging files belong to cleanup_orphaned_temp_files; skipped rather than raced. - The sha is parsed as the first 64 chars, not via Path.stem: store() takes the extension from the source filename, and a URL-encoded basename yields a multi-dot suffix that would make stem eat part of the sha. - A 900s walk budget reports partial=True instead of running to the task's hard limit (rule 89). - TASK_STUCK_THRESHOLD_MINUTES override at 30 (= time_limit 25 + 5). Without it a healthy 20-minute walk is phantom-flagged 'RecoverySweep' at the bare 5-min default — the #883 failure class; its invariant test is mirrored here. Defaults to the safe preview at both the task and the route, unlike the other maintenance triggers: this apply unlinks files. Operator-triggered only, never on a beat. Ships with its UI (rule 27): AttachmentReclaimCard in Cleanup → Duplicates & leftovers, built on the existing useMaintenanceTask/MaintenanceTile shapes, so a run survives navigating away. Surfaces files_failed and partial explicitly, since both change what the numbers mean. Also promotes humanBytes to utils/bytes.js — it was byte-identical in VideoDedupCard and GatedPurgeCard and this card would have been the third copy. The three divergent `formatBytes` helpers are deliberately left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5
|
2ce467e347 |
fix(cleanup): artist cascade preview counts posts and attachments (#3067)
`project_artist_cascade` is documented as "a read-only projection of what delete_artist_cascade would touch" and drives the Tier-C confirm dialog, but it counted only images, sources, thumbs, import_tasks and bytes. It never counted posts or attachments, both of which the apply destroys. That is silent in the worst case. `Post.artist_id` is ondelete=CASCADE, so every post goes whether or not it carried an image — and FC has a large body-only post population (#1288 measured 694 pixiv posts with text and no images). Such an artist previewed as `images: 0`, reading as "empty, safe to remove", while the apply destroyed every captured body, description, external-link set and raw_metadata snapshot. The danger-zone card already promised "every image, source, post, and attachment" — the copy was honest and the numbers were not. Root of the drift: the preview re-derived its own predicates instead of sharing the apply's, the same shape as the 2026-06-08 fandom-tag deletion that rule 93 exists for. So rather than bolt on two counts, both halves now build from shared `_artist_{images,posts,attachments}_conditions` helpers, following the `_unused_tag_conditions` / `_bare_post_conditions` style already in the file. Sources keep no helper — the apply doesn't query them either, it gets them from the Artist.sources ORM cascade. The apply also now reports `posts_deleted` (counted before the delete, since the CASCADE leaves nothing to count after). Rule 93's second half asks for the apply to be tested, and parity is only assertable if both halves state the number. Adds a preview/apply parity test that runs both against one artist and asserts the three pairs agree AND that the rows actually went, plus a body-only-artist test covering the case that motivated this. The confirm dialog's counts grid renders every key, so posts and attachments surface there automatically; the prose summary line names posts explicitly, since that is the number that changes how an artist reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5
|
39cf81aea6 |
fix(cleanup): clear an artist's attachments before the cascade delete (#3066)
`delete_artist_cascade` could abort partway through, and it aborted after the irreversible half. Deleting an artist CASCADEs to Post (post.artist_id is ondelete=CASCADE), which SET NULLs post_attachment.post_id — and `uq_post_attachment_null_post_sha` is a partial UNIQUE on sha256 ALONE WHERE post_id IS NULL. So any two of that artist's attachments sharing a sha collapse onto one another and raise. That shape is ordinary, not corrupt: `_capture_attachment` deliberately writes one row per post over a single sha-addressed blob, so a creator who attaches the same pdf to two posts already has two such rows. A pre-existing filesystem-import row (post_id NULL) with the same sha collides on its own. The images and their on-disk files are deleted and committed in 500-row batches BEFORE the artist row is touched, so the failure landed after them: images gone, artist and posts alive, files unrecoverable. Fix: delete the artist's post_attachment rows explicitly first, matched by artist_id OR by the owning post's artist (artist_id is nullable, so neither arm alone covers every row). `_repoint_post_links` already guards the identical collision class in the reconcile path; this is its artist-cascade counterpart. Migration 0043 reasoned only about upgrade-time safety and never about this later SET NULL. The sha-addressed blobs are deliberately left on disk: one blob backs many rows, so unlinking needs a refcount pass, and this Tier-C op must not delete bytes its own preview never disclosed. Adds `attachments_deleted` to the summary, and two regression tests — the same sha on two posts, and an unrelated NULL-post row that must survive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Fable 5
|
65bd1c22c3 |
test: whole-table tag counts become non-system counts
The four remaining run-1895 failures were stale expectations, not predicate bugs — prune/reset returned the right counts, but these tests verified no-deletion by counting the ENTIRE tag table (or asserting the full kind set), which now includes the three seeded hygiene tags that survive prunes and resets by design. Filter is_system=false with a pointer to #128 so future system tags cannot re-break them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM |
||
|
bvandeusenandClaude Opus 4.8
|
10434509d3 |
fix(tags): fandom views aggregate images via their characters
A fandom owns characters via Tag.fandom_id, but every image<->tag query went purely through direct image_tag rows, so a fandom only surfaced images literally tagged with it — images carrying one of its characters were invisible to its browse count, previews, and gallery filter. Derive membership at query time instead of materializing fandom rows (which would drift on every reassign/merge/remove). Add one shared predicate in tag_query.py — image_in_tag_scope / image_in_any_tag_scope: an image belongs to a tag if tagged with it directly OR (when the tag is a fandom) carrying a character whose fandom_id is that tag. The character leg is empty for non-fandom tags, so it applies uniformly with no kind branching. Route all read sites through it: - gallery _apply_scope: include, OR-groups, and symmetric exclude - directory image_count: correlated COUNT(DISTINCT) scalar subquery - directory previews: UNION direct + via-character, then ROW_NUMBER<=3 - cleanup count_tag_associations: Tier-B delete prompt now reports a fandom's true blast radius (was 0 for fandoms with no direct rows) find_unused_tags already protected fandoms via used_via_fandom; left as is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
77d02f57ae |
fix(reconcile): preserve from_attachment_id when merging duplicate posts (#73/#87)
Milestone #73 (reconcile duplicate gallery-dl/native post rows) shipped in eff6427; closing it out after today's #87 work, which added a seam it didn't account for. _repoint_post_links drops a loser post's ImageProvenance row on the (image, post) uniqueness collision — and that row may now carry from_attachment_id (which archive the file was extracted from). For the exact gallery-dl->native case this targets, the keeper is the native stub (no archive) and the loser is the gallery-dl row that extracted the member, so a blind delete silently lost the containing-archive linkage. Carry from_attachment_id onto the keeper's surviving row (when NULL) before dropping the collision. The rarer PostAttachment-collision case (both dup posts captured the same archive blob) doesn't arise in the targeted scenario — the archive lives only on the gallery-dl post, so it re-points straight to the keeper and the FK stays valid. Test: collision merge preserves the loser's from_attachment_id on the keeper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
eff64275fc |
feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.8
|
540151290b |
feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
A one-shot Maintenance action to remove the blurred locked-preview images the ingester downloaded from tier-gated Patreon posts before #874. current_user_can_view was never persisted, so the cleanup re-walks each enabled Patreon source (read-only) to re-derive which posts are gated now and the blurred filehashes Patreon serves for them, then matches by CONTENT HASH against stored source_filehash. Because the hash is content-addressed, a real file downloaded when access existed has a different hash and can never match — regained-then-lost-access content is provably spared (operator's hard requirement). NULL source_filehash => unverifiable, kept + reported. On apply: delete matched ImageRecords + files (provenance cascades), clear seen/dead-letter ledger rows for those hashes so the real media re-ingests if access returns, and delete gated posts left bare. Shares one match predicate between preview and apply (rule 93). - cleanup_service: collect_gated_previews + purge_gated_previews - tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed) - api.admin: POST /maintenance/purge-gated-previews - GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply) - tests: collect predicate, hash-match delete/spare/unverifiable, ledger clear, bare-post removal, no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
41652db20f |
feat(maintenance): retroactive video-dedup action — preview + apply (#871)
Phase 2 of #871: clean up the duplicate videos already in the library (the #859 "same video from multiple sources" clutter). Import-time dedup (Phase 1) only prevents NEW dups; this is the operator-triggered cleanup of existing ones. cleanup_service.dedup_videos(dry_run): - backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the existing library participates; idempotent (only NULL rows), writes a negative sentinel for un-probeable files so they're neither re-probed forever nor matched. - find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect, anchored per cluster to bound the span (no chain drift); keeper = highest pixel area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances. - apply: re-point each loser's post links to the keeper (so no post loses the video) THEN delete the redundant records + files via delete_images (cascade). dry_run shares the same discovery predicate and returns the projection only (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated). - dedup_videos_task (maintenance queue; summary → task_run.metadata). - POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so the card shows the dry-run projection before the destructive apply. - VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind a confirm dialog. Mounted in the Maintenance panel. Tests: dedup collapses + re-links the loser's post to the keeper + removes the file; dry-run deletes nothing; distinct durations aren't grouped; task registered. (Migration 0052 for duration_seconds already shipped with Phase 1.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
59746d213d |
feat(series): flat series sequence + cosmetic chapter dividers (#789 Phase 1)
Reframe a series from "ordered chapters that own pages" to ONE flat, series-global ordered run of pages with optional cosmetic chapter DIVIDERS over it. A chapter no longer wraps content — it's a labeled divider anchored to the page that begins it; a page's chapter is derived as the nearest preceding divider. This is what lets installments assembled from multiple sources sit in one continuous, correctly-numbered sequence (operator's Goblin Juice case). - migration 0047: flatten each series to a series-global page_number (preserving today's reading order); convert each existing chapter to a divider anchored at its first page (keeping title/stated_part); drop series_page.chapter_id; reshape series_chapter (anchor_page_id UNIQUE FK, drop chapter_number/is_placeholder/stated_page_start/end). Loss-safe for content; drops empty placeholder chapters + a redundant page-1 divider. - series_page: page_number is now the series-global order; no chapter_id. - series_chapter: anchored divider (anchor_page_id, title, stated_part). - series_service: flat list_pages (one run + derived dividers + per-page source_post + part_gaps), series-wide reorder/renumber, divider CRUD (create/update/move/delete); retired per-chapter reorder/merge/placement. - api/tags: drop chapter_id from add; /chapters endpoints are divider create/update/delete (removed chapter reorder/merge/page-reorder). - series_match_service: series "end" reads max(series_page.stated_page); accept appends via add_post. tag_service series-merge appends src's pages after tgt's max so the merged series stays one clean run. - frontend: seriesManage store + SeriesManageView → one continuous drag-reorder grid with inline divider bars + series-global page numbers; reader walks the flat run, headings from dividers; PostSeriesMenu copy. - tests reworked across the series suite for the divider model. Phase 2 (pending staging for add-from-post) is separate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
a8f624a0f1 |
fix(posts): link duplicate items to every post + prune bare shells
The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
df76bc0f58 |
test(cleanup): fix prune-spares-fandom fixture — used character keeps fandom alive
The character pointing at the fandom had no image associations, so it was itself unused and inflated the dry-run count to 2. Tag it on a real image so it is used (the real-world shape) — the fandom survives via a live character. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
de4ef6ae74 |
fix(cleanup): live prune uses the same predicate as the preview (data loss)
The fandom/chapter exclusions added in
|
||
|
bvandeusenandClaude Opus 4.8
|
fe0ed52595 |
test: drop unused binding in find_unused_tags test (ruff F841)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
fb05c5eef7 |
fix(cleanup): don't flag a character's fandom (or a chaptered series) as unused
find_unused_tags only excluded tags with image_tag or series_page references, so it flagged every fandom as 'unused' — fandoms are NEVER applied to images (a character carries its fandom via tag.fandom_id), and the FK is ondelete=SET NULL, so deleting one silently strips the fandom off all its characters (operator-flagged 2026-06-08: artist-OC fandoms showing as unused). Exclude tags referenced as a character's fandom_id, and (same class of gap) tags referenced by a series_chapter (an all-placeholder series has chapters but no pages yet). A genuinely orphaned fandom with no characters is still swept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
1804a2c622 |
feat(series): chapter layer over series_page — backend (FC-6.1)
Adds an ordered chapter layer to series. Reading order becomes (series_chapter.chapter_number, series_page.page_number); a chapter may be a placeholder reserving a slot, and carries an optional parsed stated-page range used to flag missing-page gaps. An image still lives in at most one series ⇒ one chapter (image_id stays UNIQUE). - models: series_chapter; series_page gains chapter_id (NOT NULL, cascade) + stated_page. Migration 0040 backfills every existing series into one auto-chapter holding its current flat pages — no data loss. - SeriesService: chapter CRUD (create/update/reorder/delete/merge), page→chapter assignment, reorder_pages, chapter-aware set_cover; list_pages now returns chapters[] + gaps[] alongside a back-compat flat pages[]. Legacy series-wide reorder operates on the single default chapter and rejects multi-chapter series. - API: chapter endpoints under /api/series/<tag>/chapters; POST pages accepts an optional chapter_id. - TagService.merge now repoints series_chapter too, so a merged series' chapters (and their pages) survive the source tag's deletion instead of cascading away. - Tests: new chapter suite; updated the 4 direct SeriesPage(...) constructions to supply chapter_id. Frontend (chapter-aware manage view + reader) lands next; until then the existing UI keeps working via the flat pages[] + single default chapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
91b0145bc8 |
feat(tags): 'Reset content tagging' admin action
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 4.7
|
832345a245 |
fix(fc3k): add origin=imported_filesystem to test ImageRecord ctors (second NOT NULL column after mime)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
a0136fa30d |
fix(fc3k): add mime=image/jpeg to test ImageRecord ctors (NOT NULL) + reorder admin import after stdlib/3rd-party (ruff I001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.7
|
f6aa805725 |
fc3k: cleanup_service unit tests — projections + mutations + file unlinks against real Postgres
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |