bvandeusen
5e1996e77f
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s
Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.
Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:
* 92 columns gained server_default. The models carried Python-side
`default=` only, so the ORM filled the value and the column had no
database default. Anything inserting outside the ORM behaved
differently from production.
* Eleven indexes that existed only in migrations are now declared:
the three backup_run reporting indexes, the two date-ordered
image_record browse indexes, import_task and presentation_review,
and the three task_run history indexes. All use text() for their DESC
ordering and postgresql_where for the partial one.
* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
because neither is expressible as a UniqueConstraint:
uq_tag_name_kind_fandom — an EXPRESSION index over
(name, kind, COALESCE(fandom_id, 0))
uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
IS NULL
post.py already had a comment describing the second one. The comment
was right; nothing declared it.
* The two external_link enum CHECKs (host, status) — rule 36 territory,
and absent from the model entirely.
* Two indexes were named explicitly. A bare index=True generated
ix_tag_alias_canonical_tag_id where the database has
ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
index that was already there under another name. Same for
tag_suggestion_rejection.
Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.
Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.
The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
125 lines
5.7 KiB
Python
125 lines
5.7 KiB
Python
"""ImageRecord — the gallery's primary entity, ported from ImageRepo.
|
||
|
||
ML fields and thumbnails are declared now (in FC-1) so FC-2 can populate them
|
||
without a schema migration. The SigLIP embedding column uses pgvector's Vector
|
||
type — pgvector extension is enabled in the initial migration.
|
||
"""
|
||
|
||
from datetime import datetime
|
||
|
||
from pgvector.sqlalchemy import Vector
|
||
from sqlalchemy import (
|
||
BigInteger,
|
||
DateTime,
|
||
Enum,
|
||
Float,
|
||
ForeignKey,
|
||
Index,
|
||
Integer,
|
||
String,
|
||
Text,
|
||
func,
|
||
text,
|
||
)
|
||
from sqlalchemy.orm import Mapped, mapped_column
|
||
|
||
from .base import Base
|
||
|
||
ORIGIN_CHOICES = ("downloaded", "imported_filesystem", "uploaded")
|
||
|
||
|
||
class ImageRecord(Base):
|
||
__tablename__ = "image_record"
|
||
|
||
|
||
__table_args__ = (
|
||
# alembic 0035/0071: the date-ordered browse indexes (#3275).
|
||
Index("ix_image_record_effective_date", text("effective_date DESC"), text("id DESC")),
|
||
Index("ix_image_record_earliest_post_date", text("earliest_post_date DESC"), text("id DESC")),
|
||
)
|
||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||
|
||
# On-disk identity
|
||
path: Mapped[str] = mapped_column(Text, nullable=False, unique=True)
|
||
sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True, index=True)
|
||
phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
|
||
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
|
||
mime: Mapped[str] = mapped_column(String(64), nullable=False)
|
||
width: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||
height: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||
# Video container duration (seconds); NULL for images. The Tier-1 video
|
||
# near-dup key (#871): two videos of the same artist with matching duration
|
||
# (+ aspect) are the same content across re-encodes — dedup like image pHash.
|
||
duration_seconds: Mapped[float | None] = mapped_column(Float, nullable=True)
|
||
|
||
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
|
||
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
|
||
integrity_status: Mapped[str] = mapped_column(
|
||
String(24), nullable=False, default="unknown", index=True,
|
||
server_default="unknown",
|
||
)
|
||
|
||
# Thumbnail (populated by FC-2)
|
||
thumbnail_path: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||
|
||
# Source provenance for downloaded media (#830 Phase 2). `source_url` is the
|
||
# CDN/origin URL the file was fetched from (debugging + future re-fetch).
|
||
# `source_filehash` is the URL's 32-hex CDN identity segment
|
||
# (utils.paths.filehash_from_url) — the JOIN KEY that maps a post body's
|
||
# inline `<img src=CDN>` back to this local copy so the rendered body serves
|
||
# our stored image instead of hotlinking the public source. Indexed for the
|
||
# render-time lookup. NULL for filesystem-imported / pre-Phase-2 rows.
|
||
source_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||
source_filehash: Mapped[str | None] = mapped_column(
|
||
String(32), nullable=True, index=True
|
||
)
|
||
|
||
# Origin / provenance pointers
|
||
origin: Mapped[str] = mapped_column(Enum(*ORIGIN_CHOICES, name="origin_enum"), nullable=False)
|
||
primary_post_id: Mapped[int | None] = mapped_column(
|
||
ForeignKey("post.id", ondelete="SET NULL"), nullable=True, index=True
|
||
)
|
||
# FC-2d-vii-c: canonical per-image artist (the single source of truth
|
||
# for attribution; provenance posts remain lineage detail).
|
||
artist_id: Mapped[int | None] = mapped_column(
|
||
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
|
||
)
|
||
|
||
# ML fields (populated by the ml-worker / GPU agent). 1152 = SigLIP-so400m
|
||
# embedding dim; siglip_model_version stamps which model produced it (so an
|
||
# operator model swap, #1190, can re-embed the stale rows). A different-dim
|
||
# model would need a column-width migration.
|
||
siglip_embedding: Mapped[list[float] | None] = mapped_column(Vector(1152), nullable=True)
|
||
siglip_model_version: Mapped[str | None] = mapped_column(String(128), nullable=True)
|
||
|
||
created_at: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
# Denormalized gallery sort key = COALESCE(primary post's post_date,
|
||
# created_at) (alembic 0035). The gallery used to compute this as a
|
||
# COALESCE across the Post outer join on every /scroll, which can't use
|
||
# an index and re-sorted a large slice of the library per page (×10 with
|
||
# the old serial batching). Materializing it lets the cursor scroll read
|
||
# ix_image_record_effective_date directly. Maintained by the importer
|
||
# (services/importer.py _apply_sidecar) when a primary post with a date
|
||
# is linked; plain inserts keep the created_at-equivalent server default.
|
||
effective_date: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
# Denormalized ORIGINAL-publish sort key (alembic 0071) = MIN(post_date)
|
||
# across ALL of the image's provenance posts, else created_at. effective_date
|
||
# above keys off the PRIMARY post (often the repost/download the file came
|
||
# from); this keys off the earliest publish across EVERY post the image
|
||
# appears in, so the gallery can sort by when content was first posted rather
|
||
# than when it was downloaded (operator-flagged 2026-07-01). Maintained by
|
||
# services/importer.py, recomputed whenever a dated post is linked.
|
||
earliest_post_date: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||
)
|
||
updated_at: Mapped[datetime] = mapped_column(
|
||
DateTime(timezone=True),
|
||
nullable=False,
|
||
server_default=func.now(),
|
||
onupdate=func.now(),
|
||
)
|