57 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	ca802bd885	fix: a non-image file lands on the source it was downloaded for, and the undated shells it left are folded into their real posts (#4435) ... The attachment path looked the post's source up by (artist, platform), taking the artist's lowest-id source. A Discord artist has one source per channel, so every archive or pdf from a later channel became an undated second post on the first channel's source. _post_for_sidecar now takes the downloading source, as upsert_post_record and _apply_sidecar already did. Migration 0114 re-points each shell's attachments to its dated twin and deletes the shell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 10:56:08 -04:00
bvandeusenandClaude Opus 5.5	efcb548ebf	fix: natively downloaded images take their post's date, not their download time (#4431) ... The native ingesters (Discord, Patreon, SubscribeStar) import a post's media before its record. Only the record (`_post.json`) carries the date: the Discord message timestamp, or the Patreon/SubscribeStar published_at. So each image was linked to a post with no date yet, and it kept its download time in both gallery date columns. The post itself was dated correctly once the record landed, but nothing went back to update its images. - upsert_post_record now re-dates the images already linked to the post. `effective_date` becomes the primary post's date, and `earliest_post_date` the earliest dated post the image is in, which are the same rules _attach_provenance applies. - Migration 0113 repairs the library from the posts. It writes only rows that differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 09:30:17 -04:00
bvandeusenandClaude Opus 5.5	7b1570f2a5	feat: retire the sketch/doodle WIP title tier, and the review strip asks "Is this a WIP?" (milestone 430) ... The soft tier (#1474) tagged `wip` on any post titled sketch, doodle or scribble: 6,096 of the library's 8,876 wip tags. Its conflict audit flagged most of them, because finished art scores >= 0.5 on some content head, which filled the Gallery strip with 2,086 cards. A "sketch" is usually finished work, so the operator retired the tier. The artist's own "WIP" / "work in progress" title rule and human wip tags stay. - Removed: the soft matcher, source and prefilter; the importer and backfill soft branches; soft_wip_conflict_audit with its task and beat; the wip_soft_title_tagging_enabled setting, API field and toggle; and wip_title_soft from _AUTO_SOURCES. - Migration 0112: a soft tag the operator confirmed, or kept from the strip, becomes `manual`. Every other soft tag is deleted, along with the open review cards whose tag is gone. The column is dropped. - Review strip (#4424): each card asks "Is this a WIP?" (or banner, and so on), and the buttons read "Is a WIP" / "Is not a WIP". The content tag it also scored on is shown as the reason it was flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-25 08:20:28 -04:00
bvandeusenandClaude Opus 5	01c906e6ad	fix: ruff I001 — canonical_subdir sorts before derive_subdir ... Rule 102's import-order trap, hit by inserting into the block rather than re-sorting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:49:38 -04:00
bvandeusenandClaude Opus 5	30337a6c11	fix: library paths follow the artist's slug, not the import folder's name (4244) ... Step 1 of milestone #421. The images tree has 57 directory families for what the database says are single artists — `Conto`/`conto`, `InCaseArt`/`incaseart`, `StickySpoodge`/`Stickyspoodge`/`stickyspoodge`, and so on down to a four-way split for Pocket Ace Games. There was never a duplicate Artist row. `/api/artists/names` returns exactly one per artist. The files simply get written to two places for one row: `_copy_to_library` built its destination from `derive_subdir`, which mirrors the IMPORT tree's folder name verbatim, while the download path leaves files where the ingester wrote them — under the slug. Two writers, two conventions, one artist. This is the half that stops it re-growing, and it has to land before anything moves existing files: consolidate first and the next filesystem import out of a capitalised folder re-creates the directory that was just emptied. `canonical_subdir` replaces the top-level segment with the artist's slug and leaves everything below it alone — the post hierarchy is the downloader's business. Two deliberate pass-throughs: no resolved artist (nothing authoritative to canonicalise against) and an empty subdir (a file at the images root, whose fate is task #4247, not a side effect of this helper). `_supersede` resolves the KEPT row's artist for the same reason — a supersede rewrites `existing.path`, so writing it anywhere else would move a row back out of the tree being consolidated. ImageRecord carries `artist_id` with no relationship attribute, so that is a session lookup rather than an attribute. test_import_one_happy_path pinned the old `Alice/` destination and now pins `alice/` (rule 90). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 08:48:46 -04:00
bvandeusenandClaude Opus 5	a05cb66635	fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223) ... The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:16:14 -04:00
bvandeusenandClaude Opus 4.8	af0d39ed52	feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474) ... Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 10:14:38 -04:00
bvandeusenandClaude Opus 4.8	503c8854bc	style(importer): sort wip_title import into the local-import block ... Ruff I001 fixup for 5719387 — the new .wip_title import belongs after .thumbnailer (alphabetical), not after .archive_extractor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:16:11 -04:00
bvandeusenandClaude Opus 4.8	571938781a	feat(tagging): title-based WIP auto-tagging (#1458) ... Auto-apply the `wip` system tag to posts whose TITLE explicitly declares work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision complement to the image-based ML `wip` head. WIP images are excluded from the Explore/gallery browse, so honouring the artist's own label keeps unfinished pieces out of the main browse. - services/wip_title.py: precision-first token-anchored matcher (swipe/wiped never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO NOTHING, chunked under the psycopg param ceiling). - importer: live hook on FRESH import only (never on deep-scan/supersede), so a manually-removed WIP tag is never re-applied by a routine re-scan. - maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep (coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a beat — a periodic re-run would silently undo manual removals. - ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating the live hook; GET/PATCH + POST /settings/wip-title/scan. - Settings UI: toggle + "Scan existing posts" button. - Tests: pure matcher unit tests + integration (apply idempotency, backfill precision). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:12:19 -04:00
bvandeusenandClaude Opus 4.8	8838b325fb	fix(recapture): link on-disk images to their post (#1288) ... Recapture disk-skips already-downloaded media, and upsert_post_record only writes Post fields — so a pre-existing image (e.g. one pulled under the old gallery-dl path, imported bare with no post) stays orphaned even after its post record is (re)written. Confirmed on the operator's instance: 329 pixiv images with primary_post_id NULL, 694 pixiv posts with content but no linked images, 0 duplicate posts. Fix: the recapture relink channel now carries the media's post_id (2- → 3-tuple path/url/post_id), and phase 3 calls importer.link_existing_image_to_post — match the on-disk image by path, find its Post by (source, external_post_id), upsert image_provenance + primary_post_id. Factored the provenance-linking out of _apply_sidecar into a shared _attach_provenance so the fresh-import and recapture-backlink paths can't diverge. Idempotent; generic across native platforms (no-op for already-linked Patreon/SubscribeStar). Re-running recapture now repairs orphaned images; future walks never orphan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-04 20:13:53 -04:00
bvandeusenandClaude Fable 5	5b34c9221c	feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home ... Settings IA per the approved A3 design (the old layout was the two-app merge fossilized): - Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual /import scans stay API-level; imports arrive via downloads/extension, heal via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm moves to Maintenance → 'Ingestion & filters' and loads its own settings; the import store shrinks to settings-only (no remaining consumers of the scan/task-list machinery). Overview's pending banner now points at Activity. - Maintenance regrouped: Ingestion & filters / GPU agent & embeddings (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders, Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive re-extract demoted last) / Storage. - One extension home: BrowserExtensionCard moves from Settings → Overview to Subscriptions → Settings, above the API key bar it authenticates. - Single-color import filter WIRED: skip_single_color/threshold existed since FC-2 but nothing read them (the audit module's docstring said as much) — now enforced on both import paths via the audit's canonical predicate (tolerance 30, matching the Cleanup card default; animated images exempt like the transparency check). Default stays off; test added. - Dead weight: PlaceholderView (zero refs) and the permanently-disabled 'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed (celery queue docstring, threshold comment citing retired tasks, ml package docstring, HeadsCard 'replaces Camie' blurb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:37:21 -04:00
bvandeusenandClaude Opus 4.8	c22f37d64d	feat(gallery): sort by earliest post date across all posts (new default) ... The gallery's newest/oldest sort keys off image_record.effective_date = COALESCE(primary post's post_date, created_at). The primary post is often the repost/download the file came from, so the grid led with download dates rather than when content was first posted (operator-flagged). Add a second materialized sort key, earliest_post_date = MIN(post_date) across ALL of an image's provenance posts (every post it appears in), else created_at — the original publish date. Mirrors the effective_date pattern so the sort stays a forward index scan. - alembic 0071: add earliest_post_date + index (DESC, id DESC); backfill created_at baseline then MIN over image_provenance ⋈ post. - importer: recompute earliest_post_date whenever a dated post is linked (MIN over the image's provenance, which now includes the just-added row). - gallery_service: new sorts posted_new / posted_old key off earliest_post_date; cursor + year/month grouping follow the active column transparently. - api: accept posted_new|posted_old; DEFAULT is now posted_new so the grid leads with original publish date. newest/oldest (effective_date) still available. - frontend: sort dropdown gains "Newest/Oldest post date" (default Newest post date); existing effective-date sorts relabelled "Newest/Oldest added". - tests: service test asserts posted_new/posted_old key off earliest_post_date; frontend default-sort omission test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-07-01 10:46:09 -04:00
bvandeusenandClaude Opus 4.8	bc6d43d3f2	refactor(ml): drop dead tagger/suggestion settings + columns (#1199) ... Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:41:25 -04:00
bvandeusenandClaude Opus 4.8	485387ff0b	refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189) ... Heads + CCIP are the tag source and head auto-apply is the earned propagation. The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply path firing in parallel with heads on every accept — exactly the un-earned spray the v2 pivot replaced. Retire both. Behavior change: accepting a suggestion now applies the tag to THAT image only (source='ml_accepted', a head-training positive) — it no longer allowlists + fans the tag across the library via Camie. Propagation is heads' earned auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was un-earned and bypassed the precision gate.) - tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction writes); backfill enqueues it for images with no embedding. - Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat + every enqueue caller (accept/alias/merge/per-image), api/allowlist.py + blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067), AllowlistTable.vue + allowlist store, the accept coverage-projection payload. - AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store the rail still needs); accept returns nothing, API returns {accepted, tag_id}. - tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on ML-applied image_tag sources (incl. head_auto) instead of the allowlist. - UI: MLBackfillCard relabelled to embedding-only; accept toast simplified; MaintenancePanel drops the allowlist tile. Left for a follow-up hygiene pass (now-inert, harmless): the dead settings columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*, video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders trim, and the Camie model download in download_models.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:04:31 -04:00
bvandeusenandClaude Opus 4.8	af7f0078bc	fix(importer): guard _stamp_member_archive against a sidecar-less archive ... A filesystem-imported archive with no adjacent sidecar has no Post, so _post_for_sidecar returns None — and the milestone-#87 stamp call dereferenced post.id. _stamp_member_archive already no-ops on a None post_id (no post → no provenance to stamp); pass None instead of crashing. Caught by the existing test_reimport_archive_is_idempotent (no-sidecar zip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-21 22:28:21 -04:00
bvandeusenandClaude Opus 4.8	5269cd0709	feat(provenance): capture which archive an extracted image came from (#87) ... Images pulled out of a .zip/.rar previously kept no record of WHICH archive they came from — the member->archive link was computed during extraction and discarded, leaving only image->post. So the provenance modal could only scope attachments to the whole post, showing every archive a 'High Resolution Files' bundle carried instead of the one a given file lives in. - ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id (SET NULL), migration 0055. - importer: _import_archive stamps from_attachment_id on every member's provenance row for the post (new + superseded + deduped members), resolving the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only and idempotent, so it doesn't touch the dedup/supersede branches and the backfill is safe to re-run. Nested members link to the outer stored archive. - provenance_service.for_image: when the originating post's provenance row records from_attachment_id, return ONLY that archive; else fall back to the primary-post scoping from 068def2. - ProvenancePanel: heading pluralizes ('Attachment' for a single file). - Backfill: re-running reextract_archive_attachments (ArchiveReextractCard) routes through _import_archive and stamps existing rows — no new code. Tests: capture stamps on fresh import, nested-archive attribution, per-post archive on dedup; for_image filters to the containing archive; reextract backfill stamps the link. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-21 22:22:03 -04:00
bvandeusenandClaude Opus 4.8	f154603811	feat(import): Tier-1 video near-dup by duration+aspect (#871) ... Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:17:36 -04:00
bvandeusenandClaude Opus 4.8	b48ba60830	fix(import): resolve artist from path for enrich-on-duplicate (#718) ... The dedup branches of _import_media linked the existing image to the new post via _apply_sidecar(artist=None), relying on the SIDECAR to carry the artist. But an archive member's artist comes from its path, and under post-first the per-media sidecar is minimal (no artist) — so a re-packed / cross-posted archive image deduped and was left UNLINKED from the new post, i.e. the post showed "no images". Resolve the path-anchored artist (derive_top_level_artist) up-front in _import_media and pass it to both enrich-on-duplicate branches (sha256 + phash larger_exists) and the new-record path. Drop the now-dead _attach_artist helper (its logic is inlined at the single new-record call site). Surfaced by the new test_archive_all_deduped_is_benign_not_flagged (was asserting 2==4: the second post got no provenance links). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 21:25:55 -04:00
bvandeusenandClaude Opus 4.8	8dee2f9628	feat(import): recurse nested archives + precise "no images" reason (#718) ... Root cause (operator-confirmed via event metadata + lsar): a "High Resolution files" pack often wraps a per-chapter .rar/.zip INSIDE one outer archive (incase). _import_archive only extracted one level — a nested-archive member failed is_supported and was skipped, so the real pages were silently dropped and the post showed "archive but no images". The disk scan found this pattern recurring across the attachment store. - Recurse into nested archives via _collect_archive_members: a member that is itself an archive is bomb-probed and extracted too, depth-capped at _ARCHIVE_MAX_DEPTH=3. Nested members attribute to the OUTER archive's sidecar so they link to the right Post. Each level is wrapped so one bad nested archive can't abort the import. The shared path means external (mega/gdrive) archives recurse too. - Replace the catch-all "held no supported members" string with a per-outcome tally (media/deduped/unsupported/failed/nested/nested_rejected). The all-deduped case is now recognised as BENIGN — images already in the library, re-linked to this post via enrich-on-duplicate — and returns attached WITHOUT error, so it no longer false-flags in event metadata.unextracted_archives. Genuine failures carry the precise breakdown. Tests: nested zip-in-cbz imports both inner images + links them to the outer post; all-deduped archive returns attached with error=None and links images to both posts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 21:18:02 -04:00
bvandeusenandClaude Opus 4.8	dcbc3ae335	refactor(ingest): post-first — post-record is the sole body writer on the native path (#856) ... Milestone #67 step 2. On the native core ingester the Post becomes the single authoritative record for body/links/metadata, captured once per post by the post-record; the per-media import only links image provenance + localization. Before: every per-media sidecar carried the full post body, so a post with N images wrote the body N+1 times (post-record + N media) — redundant on disk and a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only body source). - patreon_downloader: the per-media sidecar is now minimal — {category, id, source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body resolution + detail-fetch (the post-record, written first in the walk, already did it). Body no longer duplicated next to each image. - importer: new per-instance `post_first` flag (Importer is per-task). When set, `_apply_sidecar` still writes source_filehash + provenance + primary_post_id but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata, so applying a body-less sidecar would clobber raw_metadata + re-sync links off empty data). Default False keeps gallery-dl writing post fields. - download_service: `_phase3_persist` sets importer.post_first = uses_native_ingester(platform) — the future-proof seam, so a platform migrating onto the native core flips to post-first automatically (step 3). Media imports before post-records but both unify on external_post_id, so the post ends with its body either way. Tests: per-media sidecar is minimal + never hits the detail fetcher; attach post_first=True links provenance/localization but writes no post body/title; post_first=False (gallery-dl) still applies them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 10:10:21 -04:00
bvandeusenandClaude Opus 4.8	0d51b93aa7	refactor(importer): single _apply_post_fields predicate for both ingest paths (#842/#753) ... The per-media path (_apply_sidecar) and the post-record path (upsert_post_record) each carried a VERBATIM copy of the post-field write (url/title/date/description/ attachment_count/raw_metadata + external-link sync). Two copies of one concept = the divergence risk #753 targets. Consolidate into one _apply_post_fields(post, sd) helper both call — a single predicate for how a post body/links get stored, so the two sources can't drift. Behavior identical (fill-with-non-empty); both paths already covered by existing importer tests. Groundwork for the planned post-first ingest model (single authoritative post record; media attaches to it) as more platforms move onto the native ingester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 23:39:30 -04:00
bvandeusenandClaude Opus 4.8	65ec29ba9b	feat(ingest): Recapture mode — re-grab post bodies/links + localize on-disk inline images (#830) ... A plain backfill gates post-body capture on the seen-ledger, so a post whose media is already on disk AND whose post key is already seen never gets its body recaptured (operator-flagged: Industrial Lust description missing). Recovery recaptures unconditionally but re-downloads the whole source. New 'recapture' walk mode (4th beside tick/backfill/recovery): bypasses the post-record gate so EVERY post's body + external links are re-captured (detail-fetching empty bodies) WITHOUT re-downloading on-disk media; and surfaces already-present media via a separate non-deleting relink channel so the importer backfills ImageRecord.source_filehash for inline-image localization. - ingest_core: recapture mode + recapture_records gate bypass + relink collect - patreon_downloader: recapture surfaces seen-on-disk as skipped_disk(path), never refetches seen-missing media, still downloads genuinely-new - importer.relink_source_filehash: NULL-only sha256 backfill, never unlinks - download_service: mode derivation + phase-3 relink loop + lifecycle clear - source_service/api: start_recapture + backfill_recapture field + action - frontend: Recapture kebab action + 'Recapturing' badge across SourceActions/ Row/Card/SubscriptionsTab + sources store - tests across ingester/downloader/importer/source_service/api/download_service Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 20:58:40 -04:00
bvandeusenandClaude Opus 4.8	96c29c370b	feat(ingest): localize inline post-body images to local copies (Phase 2) ... Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 16:39:58 -04:00
bvandeusen	896e4f248c	style(importer): fix ruff import ordering for link_extract	2026-06-14 13:19:51 -04:00
bvandeusenandClaude Opus 4.8	d96918d777	feat(posts): extract + record external file-host links (Phase 3) ... Capture off-platform links (mega/gdrive/mediafire/dropbox/pixeldrain) embedded in post bodies so they're never silently dropped, and surface them in the post view. The download worker (Phase 4) walks these rows. - link_extract.py: pure extractor — <a href> + bare URLs, unwraps Patreon redirect shims, PRESERVES the full url incl. #fragment (mega's key), dedups. Reusable by every platform (runs off Post.description). - external_link model + migration 0049: post_id/artist_id/host/url/label/status /attempts/last_error/attachment_id/timing; CHECK whitelists (full enum incl. worker statuses up front) + (post_id,url) unique. - importer._sync_external_links: insert-missing on both import paths (_apply_sidecar + upsert_post_record) so a re-import never resets a link's status; runs for all platforms. - post_feed_service.get_post: returns external_links (detail-only). - PostCard: renders the links (host chip + label + status) once expanded. - tests: extractor (5 hosts, fragment, shim unwrap, dedup), importer (record + no-dup on reimport), serializer. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:15:36 -04:00
bvandeusenandClaude Opus 4.8	796e92540a	feat(patreon): capture media-less/text-only posts (post-only records) ... Today the ingest core does `if not media: continue`, so a post with no downloadable media (a pure-text post — which often holds the ONLY copy of an external mega/gdrive/pixeldrain link) never upserts a Post. Now the native ingester emits a post-only sidecar (`_post.json`) for every media-less post, gated through the seen-ledger via a synthetic `post:<id>` key so the body is detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on external_post_id so it UPDATES the same Post a media import would create — never doubles, never clobbers a populated body with an empty one. - gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path unaffected — all constructions are keyword). - ingest_core.py: media-less branch (optional client/downloader seams via getattr; stub clients in tests skip it as before). - patreon_client.py: post_record_key(post). patreon_downloader.py: write_post_record + _write_sidecar_data refactor (shared serializer). - importer.py: upsert_post_record. download_service.py: phase-3 import loop. - tests: client/downloader/ingester (gate + recovery)/importer (no-double). Slice 0b of milestone #64. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 12:44:46 -04:00
bvandeusenandClaude Opus 4.8	3610ba495f	feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3) ... Read cutover verified in prod (suggestions + allowlist read image_prediction; backfill complete at 908k rows / 51k images). Removes the old JSON column and everything that fed it: - ImageRecord.tagger_predictions column removed; migration 0046 DROPs it. tagger_model_version kept as the "tagged / current?" signal the backfill sweep reads (needs-tagging check switched to tagger_model_version IS NULL). - tag_and_embed no longer dual-writes the JSON — image_prediction is the only write path. - importer re-import reset drops the JSON line (image_prediction rows are already deleted on re-import). - Retired the one-time #768 backfill task + the #764 prune task, their admin endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard). - Tests seed/assert via image_prediction; stale column refs removed. Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run `VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's the source of truth now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 18:52:33 -04:00
bvandeusenandClaude Opus 4.8	22cdf0f334	feat(ml): read suggestions + allowlist from image_prediction (#768 step 2) ... Switch every prediction READER off the JSON column onto the normalized image_prediction table. Parity by construction: each reader loads the same {raw_name: {category, confidence}} dict it consumed before (via small _load_predictions helpers), so all downstream threshold/alias/merge/consensus logic is byte-identical — only the data source changed. - suggestions.SuggestionService.for_image (and for_selection via it) - ml.apply_allowlist_tags (iterates images that have prediction rows) - importer re-import reset deletes the image's prediction rows The tagger_predictions JSON column is still dual-written (step 1) so it stays valid during transition; the backfill task's NULL check still works. Removing the JSON write + DROP column + retiring the #764 prune is the cleanup follow-up (needs a quiesced-worker window for the DROP lock). Tests: shared tests/_prediction_helpers.seed_predictions seeds the table; read-path tests (suggestions, bulk consensus, allowlist apply, API) seed there instead of ImageRecord.tagger_predictions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 16:03:58 -04:00
bvandeusenandClaude Opus 4.8	a8f624a0f1	fix(posts): link duplicate items to every post + prune bare shells ... The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-08 19:28:33 -04:00
bvandeusenandClaude Opus 4.8	19eb4e9388	feat(import): surface WHY an archive was captured without extracting images ... The recurring "post shows a zip but no images" report had no diagnostic: when _import_archive captured an archive as a bare PostAttachment — because the bomb-guard probe rejected it, or extraction yielded zero members (corrupt / unsupported / missing extractor backend), or it held only non-media files — it returned status="attached" silently. Now those paths set ImportResult.error with the specific reason and log a warning, and download_service records each as {file, reason} under the event's metadata.unextracted_archives (None when every archive extracted cleanly). So the next run names exactly which archives failed and why, instead of leaving the operator to guess. No behaviour change to the happy path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 19:01:22 -04:00
bvandeusenandClaude Opus 4.8	b211900390	refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707) ... Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 11:11:25 -04:00
bvandeusenandClaude Opus 4.8	e05e0b9f37	perf(gallery): materialize indexed effective_date sort key ... The gallery cursored on COALESCE(post.post_date, image_record.created_at) across the Post outer join — an expression spanning two tables that no index can serve, so every /scroll sorted a large slice of the library (and the old frontend fired ten serially). Materialize it: - image_record.effective_date column + ix_image_record_effective_date (effective_date DESC, id DESC); alembic 0035 backfills COALESCE(primary post's post_date, created_at) for existing rows. - gallery_service._effective_date_col() now returns the column, so scroll / timeline / jump / neighbors all order off the index instead of re-deriving the COALESCE. _neighbors reads record.effective_date directly (drops an extra Post lookup). - importer._apply_sidecar maintains it: when a primary post with a date is linked, effective_date = post.post_date; plain inserts keep the created_at-equivalent server default. Tests: sidecar import asserts effective_date == post.post_date; gallery ordering/timeline/jump test seeds set effective_date alongside created_at. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 22:58:46 -04:00
bvandeusen	8d75ade1d5	fix(audit-g5b): race-poisoning in three find-or-create sites ... Audit 2026-06-02 flagged three SELECT-then-INSERT sites that lost the race under concurrent writers / recovery-sweep replays, poisoning the outer transaction with an unrecoverable IntegrityError and crashing the calling task. Same shape as the banked 2026-05-26 ImageProvenance incident (see reference_scalar_one_or_none_duplicates memory). Mirrors the importer._get_or_create pattern in all three: savepoint via begin_nested + IntegrityError rollback to that savepoint + re- SELECT to grab the row the other worker committed first. - importer._capture_attachment: PostAttachment.sha256 UNIQUE. attachments.store is sha-addressed so both workers race to write the same on-disk path (shutil.copy2 + rename is idempotent), so no extra cleanup needed. - TagService.find_or_create: partial uniqueness index on (name, kind, COALESCE(fandom_id, -1)). The previous docstring claimed "INSERT ... ON CONFLICT" but the implementation was SELECT-then-INSERT with no recovery. - ArtistService.find_or_create: already had IntegrityError handling but did session.rollback() (unwinds the WHOLE transaction); now uses begin_nested + sp.rollback() so the surrounding request's progress isn't lost.	2026-06-02 17:02:12 -04:00
bvandeusen	4bff1d8558	fix(audit-g4): status-enum miss batch ... Five extension-miss findings from the 2026-06-02 audit, where a status value was added on one side but a downstream consumer didn't pick it up. - download_service._phase3_persist: explicit branches for ImportResult.status in ('failed','refreshed'). For 'failed' (archive probe crash from _import_archive), unlink the source file so the filesystem scanner doesn't re-import and re-crash on the same archive forever. 'refreshed' is currently unreachable from the download path (no deep=True) but matches the importer's documented contract; treat as 'attached'. - gallery-dl backfill auto-complete now gates on dl_result.success + no error_type, not just return_code==0 + files_downloaded==0. VALIDATION_FAILED exits the subprocess with returncode=0 and files_downloaded=0 when every file was quarantined, matching the prior predicate exactly and zeroing the operator's armed backfill budget on the FIRST quarantine run instead of decrementing. - attach_in_place archive dispatch now threads artist + source_row through _import_archive (and _import_media for archive members) and _supersede. The path-walk fallback (_resolve_artist) is still used by filesystem-import; the download path now binds ImageProvenance to the explicit subscription Source instead of rediscovering by (artist_id, platform). - Three FE handlers now recognize status:'deferred' from /api/sources/<id>/check: SubscriptionsTab.onCheck (was toasting "event #undefined"), SubscriptionsTab.checkAll (was counting deferred as queued), DownloadEventRow.onRetry (was saying "re-queued" when nothing was). Pattern matches DownloadsTab.onRetryAll which already had it. - celery_signals._queue_for now maps backup/admin/library_audit prefixes to 'maintenance' (matching task_routes). TaskRun.queue was returning 'default' for those rows, so per-queue dashboard filters and per-queue threshold overrides (added in G3) silently missed them.	2026-06-02 16:04:59 -04:00
bvandeusen	91be9df671	fix(download): dispatch archive/non-media in attach_in_place; reshuffle showcase on mount ... attach_in_place mirrored only the media flow, so gallery-dl-downloaded zips/PDFs/audio bounced back as `skipped+invalid_image`, which download_service counted as an ingest error and flipped runs to status="error" despite N successful image attaches. Lustria patreon event #38998 (21 images + 1 OST zip) went red for exactly this reason. Now attach_in_place dispatches the same way as import_one: archives → _import_archive (extracts media members, captures archive as PostAttachment), non-media → _capture_attachment. Download_service accepts the new `attached` result and treats non-duplicate skips as soft skips, not ingest errors. Also: ShowcaseView always loadInitial() on mount, not just when the store is empty — Pinia persists across navigations and operator wants a fresh shuffle every time the showcase loads.	2026-06-02 08:16:13 -04:00
bvandeusen	2f66de2928	feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics ... Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.	2026-06-01 14:17:52 -04:00
bvandeusen	6fc8ae3106	fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup ... Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing "phantom" subscriptions like `sidecar:patreon:dpmaker`: 1. `SourceService.list()` returned every Source, no filter on URL. alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into one canonical row per (artist, platform); when no real campaign URL was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>` enabled=false as a disabled anchor. The UI then listed those anchors as if they were polls — disabled, but visible. Fix: `list()` excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True` opts back in for admin tooling. 2. `importer._source_for_sidecar` picked the lowest-id Source for (artist, platform). When alembic 0022 had rewritten a per-post row into a synthetic anchor (lower id) AND the operator later added the real subscription (higher id), every gallery-dl download silently attached its Post to the SYNTHETIC instead of the real Source. Fix: prefer a non-`sidecar:%` URL when one exists; fall back to the synthetic; only create a new synthetic when nothing exists for (artist, platform). alembic 0028 is the data half: for every (artist, platform) with both a synthetic AND a real Source, pre-merge Post+ImageProvenance collisions on the canonical, bulk-repoint Posts/ImageProvenance/ DownloadEvent.source_id onto the real Source, and delete the synthetic. Lone synthetics (no real twin) are left intact — they anchor real imported content the operator may still want; the list-filter hides them so they no longer surface as phantoms.	2026-05-31 23:08:38 -04:00
bvandeusenandClaude Opus 4.7	32bdde049f	refactor(dry-B3): extract _get_or_create race-safe find-or-create in Importer ... _find_or_create_source, _source_for_sidecar, and _find_or_create_post each repeated the SELECT → savepoint-INSERT → on-IntegrityError rollback+re-SELECT pattern. Extracted _get_or_create(stmt, factory): the statement is reused for the scalar_one_or_none lookup and the scalar_one post-conflict re-fetch, so all three are reproduced exactly. Centralizing the race-safe pattern in one place also reduces the risk of the copies drifting (the bug class banked 2026-05-26). Left _upsert_artist (no savepoint by design) and the ImageProvenance void ensure-exists block (no return / no re-select) alone — they don't fit. The rest of the ingest pipeline was already DRY: sidecar parsing lives in utils/sidecar.py, per-platform quirks in the platforms package, and _safe_ext/_categorize_error/_build_config are each single-instance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 15:02:55 -04:00
bvandeusen	e3cdd0f92b	feat(import-resilience L3): subprocess-isolated probes for video + archive ... Layer 3 — prevent the hard worker crash rather than just recovering from it. The realistic process-crash vectors (operator's observed slow/heavy tasks) are video decode and archive extraction; images decode in-process and Pillow raises-and-skips cleanly, and a subprocess per image would wreck deep-scan throughput, so images are intentionally not probed. New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so the spawned child stays light): - probe_video(path): validates the container + first video stream via ffprobe (a separate binary — a decoder crash kills only ffprobe, not the worker). Returns width/height, which the importer didn't capture for videos before. crashed=True only on ffprobe timeout. - probe_archive(path): an uncompressed-size bomb guard (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a spawned child process. A decompression-bomb OOM or native-lib segfault on a malformed archive shows up as a non-zero child exit code → crashed=True, never a dead worker. ProbeResult.crashed distinguishes a HARD failure (subprocess killed / timed out — the poison-pill signature → caller returns terminal 'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap, integrity mismatch → caller's choice of skipped/attached). Wired: - importer._import_media video branch: probe_video before the pipeline; crash → failed, clean reject → invalid_image skip, ok → capture dims. - importer._import_archive: probe_archive before extract_archive; crash → failed, clean reject → still preserve the archive as a PostAttachment (matches extract_archive's fail-soft contract). - ml.tag_and_embed video branch: probe_video before sampling 10 frames, so a corrupt video is rejected (status='bad_video') instead of crashing the ml-worker on frame decode. Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct _inspect_archive size+integrity, in-process _archive_probe_target bomb guard (monkeypatch can't reach a spawned child, so the target is called directly), and a non-video → ok=False that's robust to ffprobe presence in CI.	2026-05-28 00:01:32 -04:00
bvandeusenandClaude Opus 4.7	0bc5767a2b	fix(importer): Source = one per (artist, platform), not one per post — filesystem importer's sidecar paths now reuse the artist's existing subscription Source (or create one synthetic anchor with enabled=False) instead of fabricating a new Source per post URL. Alembic 0022 consolidates existing per-post Sources to canonical (prefers campaign URL; falls back to sidecar:<platform>:<slug>) and re-parents Posts + ImageProvenance, merging Post collisions. ... Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 14:19:29 -04:00
bvandeusenandClaude Opus 4.7	397021dcbd	fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped. ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 13:47:55 -04:00
bvandeusen	4da8d1d774	fix(importer): race-safe savepoint-based find-or-create for Source + Post (uq_source_artist_platform_url UniqueViolation operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 07:51:12 -04:00
bvandeusen	c0c9e56fb9	fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 00:45:08 -04:00
bvandeusen	52d7905c43	perf(importer): cache phash candidates on Importer to fix archive-import soft-timeout (was O(M×N) per-member SELECTs) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 22:25:32 -04:00
bvandeusen	e6ededbe8e	feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 22:16:26 -04:00
bvandeusen	36cc0622cb	fix(importer): sanitize PostAttachment.ext to skip mangled gallery-dl URL-encoded basenames (varchar(32) overrun) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 20:33:44 -04:00
bvandeusenandClaude Opus 4.7	a0470b5f60	feat(importer): _supersede() now applies the new (larger) file's sidecar — operator wanted to scan GS download dir to supersede smaller IR-migrated images AND wire up gallery-dl Post metadata, but supersede was file-only and silently dropped the sidecar. ... _apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance and sets primary_post_id NULL-only, so any IR-migration provenance on the existing row survives untouched and the new GS sidecar adds a second ImageProvenance pointing at the freshly-created Post. Wrapped in try/except so a malformed sidecar can't unwind the file-swap commit — the file replacement is the critical operation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 13:51:51 -04:00
bvandeusenandClaude Opus 4.7	68cffce322	fix(importer): catch PIL OSError during transparency + phash blocks, skip as invalid_image instead of letting Celery autoretry loop forever ... Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only validates header structure but doesn't catch truncated/broken pixel data. The error surfaces later in _transparency_pct (via getchannel 'A' -> load) or compute_phash (load) — both blow up with OSError 'broken data stream when reading image file'. Celery's autoretry_for then bounces the same file forever instead of marking it skipped. Wrap both PIL.load-triggering call sites with try/except OSError -> ImportResult(status=skipped, skip_reason=invalid_image). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 02:34:36 -04:00
bvandeusenandClaude Opus 4.7	e5da27f755	fix(fc3c): origin='downloaded' (existing enum value), not invented 'imported_download' ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 22:01:26 -04:00
bvandeusenandClaude Opus 4.7	a67755c929	feat(fc3c): Importer.attach_in_place + _supersede(new_path=) + _copy_to_library helper ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 20:38:47 -04:00