Hygiene follow-up to the Camie retirement (#1189) — these were left inert to
bound that change; nothing reads them now. Migration 0068 drops:
- ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_
character/general (already dead pre-retirement — scoring uses per-head
thresholds), video_min_tag_frames (only the deleted video-prediction
aggregator used it).
- image_record: tagger_model_version (no writer), centroid_scores (dead JSON
cache, no reader).
Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant
+ video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding
card (interval + max frames only); importer no longer resets the dropped cols;
download_models drops the Camie fetch; stale CASCADE comments in cleanup_service
no longer name the removed tables. Tests updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.
Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.
- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.
Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).
Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render a post body faithfully by serving our stored copies of inline
images instead of hotlinking the public CDN. The join key is the CDN
filehash (32-hex MD5) shared between a body <img src> and the media URL
we downloaded (the same identity extract_media dedups by):
- utils.paths.filehash_from_url — one source of truth for the extractor;
patreon_client._filehash now delegates so capture- and render-time
hashing cannot drift.
- ImageRecord gains source_url (provenance) + source_filehash (indexed
match key); migration 0051.
- the per-media sidecar carries the file's source_url; the importer
persists it (NULL-only) on the ImageRecord via _apply_sidecar.
- post_feed_service.get_post remaps body <img src> -> /images/<path> for
every inline image whose filehash maps to a stored image of THIS
artist; unmatched / pre-Phase-2 images keep hotlinking.
Pre-existing on-disk images have no filehash yet, so they fall back to
hotlinking until re-downloaded; localization is forward-looking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gallery cursored on COALESCE(post.post_date, image_record.created_at)
across the Post outer join — an expression spanning two tables that no
index can serve, so every /scroll sorted a large slice of the library
(and the old frontend fired ten serially). Materialize it:
- image_record.effective_date column + ix_image_record_effective_date
(effective_date DESC, id DESC); alembic 0035 backfills
COALESCE(primary post's post_date, created_at) for existing rows.
- gallery_service._effective_date_col() now returns the column, so scroll
/ timeline / jump / neighbors all order off the index instead of
re-deriving the COALESCE. _neighbors reads record.effective_date
directly (drops an extra Post lookup).
- importer._apply_sidecar maintains it: when a primary post with a date is
linked, effective_date = post.post_date; plain inserts keep the
created_at-equivalent server default.
Tests: sidecar import asserts effective_date == post.post_date; gallery
ordering/timeline/jump test seeds set effective_date alongside created_at.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating),
Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a
kind-aware uniqueness index so the same name can exist across kinds and the same
character name can exist in different fandoms.
Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a
single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs.
Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e
populates this via the integrity verifier.
Drops the unused tag.namespace column from FC-1 — superseded by kind.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the data model from spec §3 in one go so FC-2/FC-3 don't need
schema-adding migrations of their own. Artist is the unified entity for
both gallery 'artist:' tags and GallerySubscriber Subscriptions
(is_subscription flag). ImageProvenance is many-to-one, enabling the
enrich-on-duplicate rule for downloaded content that pHash-matches an
existing record.
The SigLIP embedding column uses pgvector(1152) for SigLIP-so400m;
swapping models in FC-2 will require a column-width migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen