bvandeusen
ed2b1adc2e
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 4s
CI / extension-version (push) Successful in 4s
Build images / build-agent (push) Successful in 11s
Build images / build-ml (push) Successful in 47s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 37s
CI / integration (push) Successful in 3m53s
baseline.yml only ever compared migrations against migrations. The question #3275 exists because nobody had ever asked the other one: does a database built from the MODELS match the one the chain produces? `mode: models` answers it. It applies the candidate autogenerated from the models instead of this tree's revisions, and diffs that against the chain. A clean run means --autogenerate is trustworthy again, which it demonstrably has not been: against the pre-reconciliation models it would have proposed dropping eleven indexes and two uniqueness guarantees. The two extensions are created by hand in that mode. They are database objects rather than table metadata, so no model can carry them — their absence is outside what this comparison asks about, and silently tolerating it is correct rather than a filter that hides a defect. Also declares the HNSW index on the ImageRecord model. SQLAlchemy can express an hnsw access method with an operator class (postgresql_using + postgresql_ops), so there was never a reason for it to live only in 0036. That removes the last item from the list of things a generated baseline cannot reproduce, leaving only the two extensions.
136 lines
6.3 KiB
Python
136 lines
6.3 KiB
Python
"""ImageRecord — the gallery's primary entity, ported from ImageRepo.
|
||
|
||
ML fields and thumbnails are declared now (in FC-1) so FC-2 can populate them
|
||
without a schema migration. The SigLIP embedding column uses pgvector's Vector
|
||
type — pgvector extension is enabled in the initial migration.
|
||
"""
|
||
|
||
from datetime import datetime
|
||
|
||
from pgvector.sqlalchemy import Vector
|
||
from sqlalchemy import (
|
||
BigInteger,
|
||
DateTime,
|
||
Enum,
|
||
Float,
|
||
ForeignKey,
|
||
Index,
|
||
Integer,
|
||
String,
|
||
Text,
|
||
func,
|
||
text,
|
||
)
|
||
from sqlalchemy.orm import Mapped, mapped_column
|
||
|
||
from .base import Base
|
||
|
||
ORIGIN_CHOICES = ("downloaded", "imported_filesystem", "uploaded")
|
||
|
||
|
||
class ImageRecord(Base):
|
||
__tablename__ = "image_record"
|
||
|
||
|
||
__table_args__ = (
|
||
# alembic 0036, and the last thing in this schema that lived only in a
|
||
# migration. SQLAlchemy CAN express an hnsw index with an operator
|
||
# class, so there is no reason for it to be invisible to the models —
|
||
# and its absence was the quietest failure of the lot: everything
|
||
# works, similarity search just silently stops using an index.
|
||
Index(
|
||
"ix_image_record_siglip_hnsw",
|
||
"siglip_embedding",
|
||
postgresql_using="hnsw",
|
||
postgresql_ops={"siglip_embedding": "vector_cosine_ops"},
|
||
),
|
||
# alembic 0035/0071: the date-ordered browse indexes (#3275).
|
||
Index("ix_image_record_effective_date", text("effective_date DESC"), text("id DESC")),
|
||
Index("ix_image_record_earliest_post_date", text("earliest_post_date DESC"), text("id DESC")),
|
||
)
|
||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||
|
||
# On-disk identity
|
||
path: Mapped[str] = mapped_column(Text, nullable=False, unique=True)
|
||
sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True, index=True)
|
||
phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
|
||
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||
mime: Mapped[str] = mapped_column(String(64), nullable=False)
|
||
width: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||
height: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||
# Video container duration (seconds); NULL for images. The Tier-1 video
|
||
# near-dup key (#871): two videos of the same artist with matching duration
|
||
# (+ aspect) are the same content across re-encodes — dedup like image pHash.
|
||
duration_seconds: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||
|
||
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
|
||
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
|
||
integrity_status: Mapped[str] = mapped_column(
|
||
String(24), nullable=False, default="unknown", index=True,
|
||
server_default="unknown",
|
||
)
|
||
|
||
# Thumbnail (populated by FC-2)
|
||
thumbnail_path: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||
|
||
# Source provenance for downloaded media (#830 Phase 2). `source_url` is the
|
||
# CDN/origin URL the file was fetched from (debugging + future re-fetch).
|
||
# `source_filehash` is the URL's 32-hex CDN identity segment
|
||
# (utils.paths.filehash_from_url) — the JOIN KEY that maps a post body's
|
||
# inline `<img src=CDN>` back to this local copy so the rendered body serves
|
||
# our stored image instead of hotlinking the public source. Indexed for the
|
||
# render-time lookup. NULL for filesystem-imported / pre-Phase-2 rows.
|
||
source_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||
source_filehash: Mapped[str | None] = mapped_column(
|
||
String(32), nullable=True, index=True
|
||
)
|
||
|
||
# Origin / provenance pointers
|
||
origin: Mapped[str] = mapped_column(Enum(*ORIGIN_CHOICES, name="origin_enum"), nullable=False)
|
||
primary_post_id: Mapped[int | None] = mapped_column(
|
||
ForeignKey("post.id", ondelete="SET NULL"), nullable=True, index=True
|
||
)
|
||
# FC-2d-vii-c: canonical per-image artist (the single source of truth
|
||
# for attribution; provenance posts remain lineage detail).
|
||
artist_id: Mapped[int | None] = mapped_column(
|
||
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
|
||
)
|
||
|
||
# ML fields (populated by the ml-worker / GPU agent). 1152 = SigLIP-so400m
|
||
# embedding dim; siglip_model_version stamps which model produced it (so an
|
||
# operator model swap, #1190, can re-embed the stale rows). A different-dim
|
||
# model would need a column-width migration.
|
||
siglip_embedding: Mapped[list[float] | None] = mapped_column(Vector(1152), nullable=True)
|
||
siglip_model_version: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||
|
||
created_at: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
# Denormalized gallery sort key = COALESCE(primary post's post_date,
|
||
# created_at) (alembic 0035). The gallery used to compute this as a
|
||
# COALESCE across the Post outer join on every /scroll, which can't use
|
||
# an index and re-sorted a large slice of the library per page (×10 with
|
||
# the old serial batching). Materializing it lets the cursor scroll read
|
||
# ix_image_record_effective_date directly. Maintained by the importer
|
||
# (services/importer.py _apply_sidecar) when a primary post with a date
|
||
# is linked; plain inserts keep the created_at-equivalent server default.
|
||
effective_date: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
# Denormalized ORIGINAL-publish sort key (alembic 0071) = MIN(post_date)
|
||
# across ALL of the image's provenance posts, else created_at. effective_date
|
||
# above keys off the PRIMARY post (often the repost/download the file came
|
||
# from); this keys off the earliest publish across EVERY post the image
|
||
# appears in, so the gallery can sort by when content was first posted rather
|
||
# than when it was downloaded (operator-flagged 2026-07-01). Maintained by
|
||
# services/importer.py, recomputed whenever a dated post is linked.
|
||
earliest_post_date: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
updated_at: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True),
|
||
nullable=False,
|
||
server_default=func.now(),
|
||
onupdate=func.now(),
|
||
)
|