db: reconcile the models with the deployed schema (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s

Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.

Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:

* 92 columns gained server_default. The models carried Python-side
  `default=` only, so the ORM filled the value and the column had no
  database default. Anything inserting outside the ORM behaved
  differently from production.

* Eleven indexes that existed only in migrations are now declared:
  the three backup_run reporting indexes, the two date-ordered
  image_record browse indexes, import_task and presentation_review,
  and the three task_run history indexes. All use text() for their DESC
  ordering and postgresql_where for the partial one.

* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
  because neither is expressible as a UniqueConstraint:
    uq_tag_name_kind_fandom  — an EXPRESSION index over
                               (name, kind, COALESCE(fandom_id, 0))
    uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
                               IS NULL
  post.py already had a comment describing the second one. The comment
  was right; nothing declared it.

* The two external_link enum CHECKs (host, status) — rule 36 territory,
  and absent from the model entirely.

* Two indexes were named explicitly. A bare index=True generated
  ix_tag_alias_canonical_tag_id where the database has
  ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
  index that was already there under another name. Same for
  tag_suggestion_rejection.

Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.

Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.

The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
This commit is contained in:
2026-08-30 14:42:30 -04:00
parent 98b56330d0
commit 5e1996e77f
26 changed files with 259 additions and 90 deletions
@@ -0,0 +1,48 @@
"""Reconcile the database with what the models have always claimed (#3275).
Milestone 328 discovered ~130 places where the ORM models and the deployed
schema disagreed. Almost all of them were the MODEL being wrong — missing
`server_default`s, indexes and CHECK constraints that only ever existed in a
migration — and those are fixed in the model files with no DDL at all, because
the database already had them.
This migration carries the remainder: the one case where the MODEL was right
and the database was missing something.
`tag.fandom_id` is declared `index=True` on the model, but no migration ever
created that index. Every autogenerate run since would have proposed adding
it; nobody ran one, so the model and the database simply drifted apart and
stayed that way.
Deliberately NOT in this migration: making `image_record.sha256` unique. The
model says `unique=True` and `0001` created a plain, non-unique index, so
duplicates are possible today and the ORM believes they are not. Adding the
constraint is a real change that FAILS if duplicates already exist, and if
they do exist the right response is a dedupe decision rather than a constraint
— so it needs an answer about live data before it is written, not after.
Tracked in #3275.
Revision ID: 0088
Revises: 0087
Create Date: 2026-08-30
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0088"
down_revision: Union[str, None] = "0087"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# IF NOT EXISTS because the index is what the model already asks for: any
# database built from metadata rather than from this chain will have it,
# and this migration must be a no-op there rather than an error.
op.execute("CREATE INDEX IF NOT EXISTS ix_tag_fandom_id ON tag (fandom_id)")
def downgrade() -> None:
op.execute("DROP INDEX IF EXISTS ix_tag_fandom_id")
+2 -2
View File
@@ -27,10 +27,10 @@ class Artist(Base):
notes: Mapped[str | None] = mapped_column(Text, nullable=True) notes: Mapped[str | None] = mapped_column(Text, nullable=True)
# True once a Source is attached; flips false if all sources removed. # True once a Source is attached; flips false if all sources removed.
is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Per-artist scheduling overrides; null means "use global default". # Per-artist scheduling overrides; null means "use global default".
auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
check_interval_seconds: Mapped[int | None] = mapped_column(Integer, nullable=True) check_interval_seconds: Mapped[int | None] = mapped_column(Integer, nullable=True)
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
+9 -1
View File
@@ -20,7 +20,7 @@ feedback_check_existing_enums):
from datetime import datetime from datetime import datetime
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text from sqlalchemy import BigInteger, DateTime, ForeignKey, Index, Integer, JSON, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -29,10 +29,18 @@ from .base import Base
class BackupRun(Base): class BackupRun(Base):
__tablename__ = "backup_run" __tablename__ = "backup_run"
__table_args__ = (
# alembic 0017: reporting indexes, never declared on the model (#3275).
Index("ix_backup_run_kind_started", "kind", text("started_at DESC")),
Index("ix_backup_run_status_finished", "status", text("finished_at DESC")),
Index("ix_backup_run_tag_partial", "tag", postgresql_where=text("tag IS NOT NULL")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True) kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True, String(16), nullable=False, default="pending", index=True,
server_default="pending",
) )
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True) tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False) triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
+2 -2
View File
@@ -25,8 +25,8 @@ class DownloadEvent(Base):
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default="0")
files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
metadata_: Mapped[dict] = mapped_column( metadata_: Mapped[dict] = mapped_column(
"metadata", JSONB, nullable=False, default=dict, "metadata", JSONB, nullable=False, default=dict,
+11
View File
@@ -16,6 +16,7 @@ doesn't delete the link record).
from datetime import datetime from datetime import datetime
from sqlalchemy import ( from sqlalchemy import (
CheckConstraint,
DateTime, DateTime,
Float, Float,
ForeignKey, ForeignKey,
@@ -38,6 +39,16 @@ STATUSES = ("pending", "downloading", "downloaded", "failed", "skipped", "dead")
class ExternalLink(Base): class ExternalLink(Base):
__tablename__ = "external_link" __tablename__ = "external_link"
__table_args__ = ( __table_args__ = (
# alembic 0028 enum CHECKs. Rule 36 territory: a new host or status value
# needs its constraint swapped in the same migration (#3275).
CheckConstraint(
"host IN ('mega', 'gdrive', 'mediafire', 'dropbox', 'pixeldrain')",
name="ck_external_link_host",
),
CheckConstraint(
"status IN ('pending', 'downloading', 'downloaded', 'failed', 'skipped', 'dead')",
name="ck_external_link_status",
),
# One row per (post, url). The full url (incl. #fragment) is the identity # One row per (post, url). The full url (incl. #fragment) is the identity
# — the same file linked twice in a post collapses to one row. # — the same file linked twice in a post collapses to one row.
Index("uq_external_link_post_url", "post_id", "url", unique=True), Index("uq_external_link_post_url", "post_id", "url", unique=True),
+3 -2
View File
@@ -50,7 +50,8 @@ class GpuJob(Base):
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'. # What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
task: Mapped[str] = mapped_column(String(32), nullable=False) task: Mapped[str] = mapped_column(String(32), nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True String(16), nullable=False, default="pending", index=True,
server_default="pending",
) )
# pending | leased | done | error # pending | leased | done | error
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True) lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
@@ -60,7 +61,7 @@ class GpuJob(Base):
lease_expires_at: Mapped[datetime | None] = mapped_column( lease_expires_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True DateTime(timezone=True), nullable=True
) )
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Triage verdict for an ERRORED job (#125): NULL = not yet probed; # Triage verdict for an ERRORED job (#125): NULL = not yet probed;
# 'defect' = the integrity probe says the FILE itself is bad (surfaced for # 'defect' = the integrity probe says the FILE itself is bad (surfaced for
+3 -2
View File
@@ -24,10 +24,11 @@ class HeadAutoApplyRun(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
# dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing # dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing
# (preview/apply parity, rule 93). # (preview/apply parity, rule 93).
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | error # running | ready | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
+2 -2
View File
@@ -24,9 +24,9 @@ class HeadMetric(Base):
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
) )
# An auto-applied (source='head_auto') tag the operator later REMOVED. # An auto-applied (source='head_auto') tag the operator later REMOVED.
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# A tag with a head that the operator added by HAND (the head missed it). # A tag with a head that the operator added by HAND (the head missed it).
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
+3 -3
View File
@@ -28,9 +28,9 @@ class HeadMetricsSnapshot(Base):
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
) )
# Current count of source='head_auto' applications still standing. # Current count of source='head_auto' applications still standing.
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# The head's measured quality at snapshot time (null if no head exists). # The head's measured quality at snapshot time (null if no head exists).
ap: Mapped[float | None] = mapped_column(Float, nullable=True) ap: Mapped[float | None] = mapped_column(Float, nullable=True)
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True) precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
+2 -1
View File
@@ -24,7 +24,8 @@ class HeadTrainingRun(Base):
# Training parameters: {min_positives, neg_ratio, precision_target, ...}. # Training parameters: {min_positives, neg_ratio, precision_target, ...}.
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | error # running | ready | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
+10 -1
View File
@@ -14,10 +14,12 @@ from sqlalchemy import (
Enum, Enum,
Float, Float,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Text, Text,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -29,6 +31,12 @@ ORIGIN_CHOICES = ("downloaded", "imported_filesystem", "uploaded")
class ImageRecord(Base): class ImageRecord(Base):
__tablename__ = "image_record" __tablename__ = "image_record"
__table_args__ = (
# alembic 0035/0071: the date-ordered browse indexes (#3275).
Index("ix_image_record_effective_date", text("effective_date DESC"), text("id DESC")),
Index("ix_image_record_earliest_post_date", text("earliest_post_date DESC"), text("id DESC")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
# On-disk identity # On-disk identity
@@ -47,7 +55,8 @@ class ImageRecord(Base):
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'. # Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'. # Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
integrity_status: Mapped[str] = mapped_column( integrity_status: Mapped[str] = mapped_column(
String(24), nullable=False, default="unknown", index=True String(24), nullable=False, default="unknown", index=True,
server_default="unknown",
) )
# Thumbnail (populated by FC-2) # Thumbnail (populated by FC-2)
+7 -7
View File
@@ -21,17 +21,17 @@ class ImportBatch(Base):
) )
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0) total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0) imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0) skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0) failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0) attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# Deep-scan only: count of already-imported files whose sidecar metadata # Deep-scan only: count of already-imported files whose sidecar metadata
# got re-applied this run (post/source/provenance upsert). Stays 0 on # got re-applied this run (post/source/provenance upsert). Stays 0 on
# quick-scan batches. See `Importer.import_one(deep_scan=True)`. # quick-scan batches. See `Importer.import_one(deep_scan=True)`.
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0) refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True) status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True, server_default="running")
# running | complete | cancelled # running | complete | cancelled
tasks = relationship("ImportTask", back_populates="batch", cascade="all, delete-orphan") tasks = relationship("ImportTask", back_populates="batch", cascade="all, delete-orphan")
+25 -14
View File
@@ -17,60 +17,71 @@ class ImportSettings(Base):
__table_args__ = (CheckConstraint("id = 1", name="singleton"),) __table_args__ = (CheckConstraint("id = 1", name="singleton"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import") import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import", server_default="/import")
min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0) min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0) min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9) transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9, server_default="0.9")
skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95) single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30) single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10) phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
# FC-3c downloader knobs # FC-3c downloader knobs
download_rate_limit_seconds: Mapped[float] = mapped_column( download_rate_limit_seconds: Mapped[float] = mapped_column(
Float, nullable=False, default=3.0 Float, nullable=False, default=3.0,
server_default="3",
) )
download_validate_files: Mapped[bool] = mapped_column( download_validate_files: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
# FC-3d scheduling knobs # FC-3d scheduling knobs
download_schedule_default_seconds: Mapped[int] = mapped_column( download_schedule_default_seconds: Mapped[int] = mapped_column(
Integer, nullable=False, default=28800 Integer, nullable=False, default=28800,
server_default="28800",
) )
download_event_retention_days: Mapped[int] = mapped_column( download_event_retention_days: Mapped[int] = mapped_column(
Integer, nullable=False, default=90 Integer, nullable=False, default=90,
server_default="90",
) )
download_failure_warning_threshold: Mapped[int] = mapped_column( download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5 Integer, nullable=False, default=5,
server_default="5",
) )
# FC-3h backup knobs. # FC-3h backup knobs.
backup_db_nightly_enabled: Mapped[bool] = mapped_column( backup_db_nightly_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, Boolean, nullable=False, default=False,
server_default="false",
) )
backup_db_nightly_hour_utc: Mapped[int] = mapped_column( backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
Integer, nullable=False, default=3, Integer, nullable=False, default=3,
server_default="3",
) )
backup_db_keep_last_n: Mapped[int] = mapped_column( backup_db_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=14, Integer, nullable=False, default=14,
server_default="14",
) )
backup_images_keep_last_n: Mapped[int] = mapped_column( backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3, Integer, nullable=False, default=3,
server_default="3",
) )
# FC-6.3 series continuation matcher. enabled gates the rescan; threshold is # FC-6.3 series continuation matcher. enabled gates the rescan; threshold is
# the weighted-score cut-off (0..1) above which a pending suggestion is made. # the weighted-score cut-off (0..1) above which a pending suggestion is made.
series_suggest_enabled: Mapped[bool] = mapped_column( series_suggest_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True, Boolean, nullable=False, default=True,
server_default="true",
) )
series_suggest_threshold: Mapped[float] = mapped_column( series_suggest_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.5, Float, nullable=False, default=0.5,
server_default="0.5",
) )
# #830 off-platform file-host downloads — per-host enable lever (default on, # #830 off-platform file-host downloads — per-host enable lever (default on,
+9 -3
View File
@@ -13,10 +13,12 @@ from sqlalchemy import (
Boolean, Boolean,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Text, Text,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -26,6 +28,10 @@ from .base import Base
class ImportTask(Base): class ImportTask(Base):
__tablename__ = "import_task" __tablename__ = "import_task"
__table_args__ = (
Index("ix_import_task_created_at_desc", text("created_at DESC")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
batch_id: Mapped[int] = mapped_column( batch_id: Mapped[int] = mapped_column(
ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True
@@ -33,14 +39,14 @@ class ImportTask(Base):
source_path: Mapped[str] = mapped_column(Text, nullable=False) source_path: Mapped[str] = mapped_column(Text, nullable=False)
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True) status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True, server_default="pending")
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks # Poison-pill circuit breaker (alembic 0026). recovery_count tracks
# how many times the stuck-task sweep has re-queued this row; after # how many times the stuck-task sweep has re-queued this row; after
# the cap it's failed with a diagnostic instead of looping. refetched # the cap it's failed with a diagnostic instead of looping. refetched
# bounds the one-shot re-download remediation to a single attempt. # bounds the one-shot re-download remediation to a single attempt.
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
result_image_id: Mapped[int | None] = mapped_column( result_image_id: Mapped[int | None] = mapped_column(
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
+8 -5
View File
@@ -8,7 +8,7 @@ reads it and routes through cleanup_service.delete_images.
from datetime import datetime from datetime import datetime
from typing import Any from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func from sqlalchemy import DateTime, Integer, String, Text, func, text
from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -23,6 +23,7 @@ class LibraryAuditRun(Base):
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True, String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | applied | cancelled | error # running | ready | applied | cancelled | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
@@ -31,14 +32,16 @@ class LibraryAuditRun(Base):
finished_at: Mapped[datetime | None] = mapped_column( finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, DateTime(timezone=True), nullable=True,
) )
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list) matched_ids: Mapped[list[int]] = mapped_column(
JSONB, nullable=False, default=list, server_default=text("'[]'::jsonb")
)
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes # Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes
# from, and the last time a chunk made progress (so the recovery sweep can # from, and the last time a chunk made progress (so the recovery sweep can
# tell a progressing multi-chunk audit from a stuck one). # tell a progressing multi-chunk audit from a stuck one).
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0) resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
last_progress_at: Mapped[datetime | None] = mapped_column( last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, DateTime(timezone=True), nullable=True,
) )
+64 -31
View File
@@ -31,17 +31,20 @@ class MLSettings(Base):
# queueing embed work nothing will consume (the daily GPU 'embed' backfill # queueing embed work nothing will consume (the daily GPU 'embed' backfill
# covers those images instead). # covers those images instead).
cpu_embed_enabled: Mapped[bool] = mapped_column( cpu_embed_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
# Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not # Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not
# a fixed count) so coverage reflects real screen time regardless of length; # a fixed count) so coverage reflects real screen time regardless of length;
# cap the total so a long video can't explode into hundreds of embeds. The # cap the total so a long video can't explode into hundreds of embeds. The
# per-frame SigLIP embeddings are mean-pooled. Operator-tunable. # per-frame SigLIP embeddings are mean-pooled. Operator-tunable.
video_frame_interval_seconds: Mapped[float] = mapped_column( video_frame_interval_seconds: Mapped[float] = mapped_column(
Float, nullable=False, default=4.0 Float, nullable=False, default=4.0,
server_default="4",
) )
video_max_frames: Mapped[int] = mapped_column( video_max_frames: Mapped[int] = mapped_column(
Integer, nullable=False, default=64 Integer, nullable=False, default=64,
server_default="64",
) )
# Tagging-v2 head training (#114). The head is the suggestion source that # Tagging-v2 head training (#114). The head is the suggestion source that
# LEARNS from the operator's tags (replacing Camie + centroid). A concept # LEARNS from the operator's tags (replacing Camie + centroid). A concept
@@ -49,10 +52,12 @@ class MLSettings(Base):
# head_auto_apply_precision is the precision bar a head must clear (at some # head_auto_apply_precision is the precision bar a head must clear (at some
# operating point) to "graduate" into earned auto-apply. Operator-tunable. # operating point) to "graduate" into earned auto-apply. Operator-tunable.
head_min_positives: Mapped[int] = mapped_column( head_min_positives: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
head_auto_apply_precision: Mapped[float] = mapped_column( head_auto_apply_precision: Mapped[float] = mapped_column(
Float, nullable=False, default=0.97 Float, nullable=False, default=0.97,
server_default="0.97",
) )
# Earned auto-apply (#114). A graduated head fires (tags images without a # Earned auto-apply (#114). A graduated head fires (tags images without a
# human) when this master switch is on AND the head has at least # human) when this master switch is on AND the head has at least
@@ -61,29 +66,34 @@ class MLSettings(Base):
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support + # default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
# measured-precision gates keep it safe, and every auto-tag is reversible. # measured-precision gates keep it safe, and every auto-tag is reversible.
head_auto_apply_enabled: Mapped[bool] = mapped_column( head_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
head_auto_apply_min_positives: Mapped[int] = mapped_column( head_auto_apply_min_positives: Mapped[int] = mapped_column(
# Support floor raised 30→50 (operator-asked 2026-07-06): a head needs # Support floor raised 30→50 (operator-asked 2026-07-06): a head needs
# more human labels before it may fire without a human. # more human labels before it may fire without a human.
Integer, nullable=False, default=50 Integer, nullable=False, default=50,
server_default="30",
) )
# CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75 # CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
# over-fired (high-reference characters matched a scatter of images); 0.85 # over-fired (high-reference characters matched a scatter of images); 0.85
# keeps the confident single-character matches. Tunable from the agent card. # keeps the confident single-character matches. Tunable from the agent card.
ccip_match_threshold: Mapped[float] = mapped_column( ccip_match_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.85 Float, nullable=False, default=0.85,
server_default="0.85",
) )
# CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold, # CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
# above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out); # above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
# single-character references + the high bar keep it safe, every tag reversible. # single-character references + the high bar keep it safe, every tag reversible.
ccip_auto_apply_enabled: Mapped[bool] = mapped_column( ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
ccip_auto_apply_threshold: Mapped[float] = mapped_column( ccip_auto_apply_threshold: Mapped[float] = mapped_column(
# Raised 0.92→0.95 (operator-asked 2026-07-06) so only very confident # Raised 0.92→0.95 (operator-asked 2026-07-06) so only very confident
# character matches auto-tag. # character matches auto-tag.
Float, nullable=False, default=0.95 Float, nullable=False, default=0.95,
server_default="0.92",
) )
# -- Presentation chrome auto-hide (#141) ------------------------------- # -- Presentation chrome auto-hide (#141) -------------------------------
# `banner` (chrome — clusters on UI, not content) auto-applies on the sweep # `banner` (chrome — clusters on UI, not content) auto-applies on the sweep
@@ -95,13 +105,16 @@ class MLSettings(Base):
# (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor # (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor
# screenshot` are no longer chrome — they went to the PROCESS path below. # screenshot` are no longer chrome — they went to the PROCESS path below.
presentation_auto_apply_enabled: Mapped[bool] = mapped_column( presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
presentation_auto_apply_threshold: Mapped[float] = mapped_column( presentation_auto_apply_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.90 Float, nullable=False, default=0.90,
server_default="0.90",
) )
presentation_conflict_threshold: Mapped[float] = mapped_column( presentation_conflict_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50 Float, nullable=False, default=0.50,
server_default="0.50",
) )
# -- Process auto-apply (#1464) ---------------------------------------- # -- Process auto-apply (#1464) ----------------------------------------
# `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program # `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program
@@ -115,24 +128,29 @@ class MLSettings(Base):
# (PresentationReview, mode='process') rather than silently marked. OFF by # (PresentationReview, mode='process') rather than silently marked. OFF by
# default — a new whole-library auto-tagger is opt-in; every auto-tag reversible. # default — a new whole-library auto-tagger is opt-in; every auto-tag reversible.
process_auto_apply_enabled: Mapped[bool] = mapped_column( process_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False Boolean, nullable=False, default=False,
server_default="false",
) )
process_auto_apply_threshold: Mapped[float] = mapped_column( process_auto_apply_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.90 Float, nullable=False, default=0.90,
server_default="0.9",
) )
process_conflict_threshold: Mapped[float] = mapped_column( process_conflict_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50 Float, nullable=False, default=0.50,
server_default="0.5",
) )
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069); # Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
# existing libraries keep their stored value until the operator re-embeds. # existing libraries keep their stored value until the operator re-embeds.
embedder_model_version: Mapped[str] = mapped_column( embedder_model_version: Mapped[str] = mapped_column(
String(128), nullable=False, default="siglip2-so400m-patch16-512" String(128), nullable=False, default="siglip2-so400m-patch16-512",
server_default="siglip2-so400m-patch16-512",
) )
# The HF model NAME the embedder loads (server CPU embed + announced to the # The HF model NAME the embedder loads (server CPU embed + announced to the
# GPU agent in the lease). Operator-settable so the embedder is a choice, not # GPU agent in the lease). Operator-settable so the embedder is a choice, not
# a hardcode (#1190): set name + version together, then re-embed + retrain. # a hardcode (#1190): set name + version together, then re-embed + retrain.
embedder_model_name: Mapped[str] = mapped_column( embedder_model_name: Mapped[str] = mapped_column(
String(128), nullable=False, default="google/siglip2-so400m-patch16-512" String(128), nullable=False, default="google/siglip2-so400m-patch16-512",
server_default="google/siglip2-so400m-patch16-512",
) )
# -- Crop proposers / detectors (#1202, #134) -------------------------- # -- Crop proposers / detectors (#1202, #134) --------------------------
# WHERE-to-crop YOLO detectors feeding the crop→SigLIP bag + CCIP. Config # WHERE-to-crop YOLO detectors feeding the crop→SigLIP bag + CCIP. Config
@@ -145,20 +163,24 @@ class MLSettings(Base):
# person: general COCO figure detector for Western/realistic art the anime # person: general COCO figure detector for Western/realistic art the anime
# person-detector misses → NMS-merged with imgutils → CCIP + concept. # person-detector misses → NMS-merged with imgutils → CCIP + concept.
detector_person_enabled: Mapped[bool] = mapped_column( detector_person_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_person_weights: Mapped[str] = mapped_column( detector_person_weights: Mapped[str] = mapped_column(
String(512), nullable=False, default="yolo11n.pt" String(512), nullable=False, default="yolo11n.pt",
server_default="yolo11n.pt",
) )
detector_person_conf: Mapped[float] = mapped_column( detector_person_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.35 Float, nullable=False, default=0.35,
server_default="0.35",
) )
# anatomy: booru_yolo anime/furry/NSFW torso components → concept crops. # anatomy: booru_yolo anime/furry/NSFW torso components → concept crops.
# Default = yolov11m_aa22 (26 classes, best mAP50-95 0.96), committed in the # Default = yolov11m_aa22 (26 classes, best mAP50-95 0.96), committed in the
# upstream repo so the URL resolves. License UNSTATED — fine for a private # upstream repo so the URL resolves. License UNSTATED — fine for a private
# homelab (operator accepted #1202). # homelab (operator accepted #1202).
detector_anatomy_enabled: Mapped[bool] = mapped_column( detector_anatomy_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_anatomy_weights: Mapped[str] = mapped_column( detector_anatomy_weights: Mapped[str] = mapped_column(
String(512), nullable=False, String(512), nullable=False,
@@ -166,37 +188,47 @@ class MLSettings(Base):
"https://github.com/aperveyev/booru_yolo/raw/main/models/" "https://github.com/aperveyev/booru_yolo/raw/main/models/"
"yolov11m_aa22.pt" "yolov11m_aa22.pt"
), ),
server_default="https://github.com/aperveyev/booru_yolo/raw/main/models/yolov11m_aa22.pt",
) )
detector_anatomy_conf: Mapped[float] = mapped_column( detector_anatomy_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30 Float, nullable=False, default=0.30,
server_default="0.30",
) )
# panel: comic page → panel regions → concept crops (Apache-2.0, YOLOv12x). # panel: comic page → panel regions → concept crops (Apache-2.0, YOLOv12x).
detector_panel_enabled: Mapped[bool] = mapped_column( detector_panel_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_panel_weights: Mapped[str] = mapped_column( detector_panel_weights: Mapped[str] = mapped_column(
String(512), nullable=False, String(512), nullable=False,
default="mosesb/best-comic-panel-detection::best.pt", default="mosesb/best-comic-panel-detection::best.pt",
server_default="mosesb/best-comic-panel-detection::best.pt",
) )
detector_panel_conf: Mapped[float] = mapped_column( detector_panel_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30 Float, nullable=False, default=0.30,
server_default="0.30",
) )
# Per-frame caps bound the crop→embed explosion; max_regions is the hard # Per-frame caps bound the crop→embed explosion; max_regions is the hard
# per-job backstop; dedupe_iou drops near-duplicate crops before the embed. # per-job backstop; dedupe_iou drops near-duplicate crops before the embed.
detector_max_figures: Mapped[int] = mapped_column( detector_max_figures: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_components: Mapped[int] = mapped_column( detector_max_components: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_panels: Mapped[int] = mapped_column( detector_max_panels: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_regions: Mapped[int] = mapped_column( detector_max_regions: Mapped[int] = mapped_column(
Integer, nullable=False, default=128 Integer, nullable=False, default=128,
server_default="128",
) )
detector_dedupe_iou: Mapped[float] = mapped_column( detector_dedupe_iou: Mapped[float] = mapped_column(
Float, nullable=False, default=0.85 Float, nullable=False, default=0.85,
server_default="0.85",
) )
# -- CCIP character prototypes (#1317) --------------------------------- # -- CCIP character prototypes (#1317) ---------------------------------
# The per-character reference set is precomputed + refreshed INCREMENTALLY # The per-character reference set is precomputed + refreshed INCREMENTALLY
@@ -208,7 +240,8 @@ class MLSettings(Base):
String(128), nullable=True String(128), nullable=True
) )
ccip_prototype_cap: Mapped[int] = mapped_column( ccip_prototype_cap: Mapped[int] = mapped_column(
Integer, nullable=False, default=64 Integer, nullable=False, default=64,
server_default="64",
) )
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+1 -1
View File
@@ -35,7 +35,7 @@ class PatreonFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+1 -1
View File
@@ -35,7 +35,7 @@ class PixivFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+7 -1
View File
@@ -9,15 +9,17 @@ artist-filter queries don't depend on the Source detour).
from datetime import datetime from datetime import datetime
from sqlalchemy import ( from sqlalchemy import (
JSON,
CheckConstraint, CheckConstraint,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
JSON,
String, String,
Text, Text,
UniqueConstraint, UniqueConstraint,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -27,6 +29,10 @@ from .base import Base
class Post(Base): class Post(Base):
__tablename__ = "post" __tablename__ = "post"
__table_args__ = ( __table_args__ = (
# alembic 0030. The comment above described this index; nothing declared
# it, so autogenerate proposed dropping it (#3275).
Index("uq_post_artist_external_id_null_source", "artist_id", "external_post_id",
unique=True, postgresql_where=text("source_id IS NULL")),
# Source-bound dedup. Postgres treats NULL != NULL so rows # Source-bound dedup. Postgres treats NULL != NULL so rows
# with source_id IS NULL aren't deduped by this constraint; # with source_id IS NULL aren't deduped by this constraint;
# the partial unique index `uq_post_artist_external_id_null_source` # the partial unique index `uq_post_artist_external_id_null_source`
+5 -1
View File
@@ -11,7 +11,7 @@ are pruned by retention.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, Float, ForeignKey, String, func from sqlalchemy import DateTime, Float, ForeignKey, Index, String, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -20,6 +20,10 @@ from .base import Base
class PresentationReview(Base): class PresentationReview(Base):
__tablename__ = "presentation_review" __tablename__ = "presentation_review"
__table_args__ = (
Index("ix_presentation_review_resolved_at", "resolved_at"),
)
image_record_id: Mapped[int] = mapped_column( image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
) )
+3 -3
View File
@@ -5,7 +5,7 @@ Multiple sources per artist support creators with cross-platform presence.
from datetime import datetime from datetime import datetime
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Integer, String, Text from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, JSON, String, Text
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base from .base import Base
@@ -20,7 +20,7 @@ class Source(Base):
) )
platform: Mapped[str] = mapped_column(String(64), nullable=False) platform: Mapped[str] = mapped_column(String(64), nullable=False)
url: Mapped[str] = mapped_column(Text, nullable=False) url: Mapped[str] = mapped_column(Text, nullable=False)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
config_overrides: Mapped[dict | None] = mapped_column(JSON, nullable=True) config_overrides: Mapped[dict | None] = mapped_column(JSON, nullable=True)
@@ -32,7 +32,7 @@ class Source(Base):
# by _update_source_health alongside last_error; cleared on 'ok'. # by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True) error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True) check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0) consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# alembic 0031: sticky deep-scan budget. When > 0, the next N download # alembic 0031: sticky deep-scan budget. When > 0, the next N download
# runs use gallery-dl's full-walk config (skip: True + 1800s timeout); # runs use gallery-dl's full-walk config (skip: True + 1800s timeout);
@@ -34,7 +34,7 @@ class SubscribeStarFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+8 -1
View File
@@ -15,11 +15,13 @@ from sqlalchemy import (
Column, Column,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Table, Table,
false, false,
func, func,
text,
) )
from sqlalchemy import ( from sqlalchemy import (
Enum as SQLEnum, Enum as SQLEnum,
@@ -67,7 +69,7 @@ image_tag = Table(
primary_key=True, primary_key=True,
), ),
Column("tag_id", ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True), Column("tag_id", ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True),
Column("source", String(32), nullable=False, default="manual"), Column("source", String(32), nullable=False, default="manual", server_default="manual"),
Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()), Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()),
) )
@@ -75,6 +77,10 @@ image_tag = Table(
class Tag(Base): class Tag(Base):
__tablename__ = "tag" __tablename__ = "tag"
__table_args__ = ( __table_args__ = (
# alembic 0002. An EXPRESSION index — COALESCE cannot be expressed as a
# UniqueConstraint, which is why it only ever existed in a migration (#3275).
Index("uq_tag_name_kind_fandom", "name", "kind", text("COALESCE(fandom_id, 0)"),
unique=True),
CheckConstraint( CheckConstraint(
"(fandom_id IS NULL) OR (kind = 'character')", "(fandom_id IS NULL) OR (kind = 'character')",
name="ck_tag_fandom_requires_character", name="ck_tag_fandom_requires_character",
@@ -87,6 +93,7 @@ class Tag(Base):
SQLEnum(TagKind, name="tag_kind", values_callable=lambda e: [m.value for m in e]), SQLEnum(TagKind, name="tag_kind", values_callable=lambda e: [m.value for m in e]),
nullable=False, nullable=False,
default=TagKind.general, default=TagKind.general,
server_default="general",
) )
fandom_id: Mapped[int | None] = mapped_column( fandom_id: Mapped[int | None] = mapped_column(
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True, index=True ForeignKey("tag.id", ondelete="SET NULL"), nullable=True, index=True
+9 -2
View File
@@ -5,7 +5,7 @@ in image_prediction stay unmolested.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, String, func from sqlalchemy import DateTime, ForeignKey, Index, String, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -14,10 +14,17 @@ from .base import Base
class TagAlias(Base): class TagAlias(Base):
__tablename__ = "tag_alias" __tablename__ = "tag_alias"
__table_args__ = (
# Named explicitly: the database calls this ix_tag_alias_canonical, while
# a bare index=True on the column would generate ix_tag_alias_canonical_tag_id
# and silently propose a drop+create on the next autogenerate (#3275).
Index("ix_tag_alias_canonical", "canonical_tag_id"),
)
alias_string: Mapped[str] = mapped_column(String(255), primary_key=True) alias_string: Mapped[str] = mapped_column(String(255), primary_key=True)
alias_category: Mapped[str] = mapped_column(String(32), primary_key=True) alias_category: Mapped[str] = mapped_column(String(32), primary_key=True)
canonical_tag_id: Mapped[int] = mapped_column( canonical_tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("tag.id", ondelete="CASCADE"), nullable=False
) )
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
@@ -5,7 +5,7 @@ Prevents re-suggestion AND prevents allowlist auto-apply on that image.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, func from sqlalchemy import DateTime, ForeignKey, Index, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -14,11 +14,16 @@ from .base import Base
class TagSuggestionRejection(Base): class TagSuggestionRejection(Base):
__tablename__ = "tag_suggestion_rejection" __tablename__ = "tag_suggestion_rejection"
__table_args__ = (
# Named explicitly; see tag_alias for why (#3275).
Index("ix_tag_suggestion_rejection_tag", "tag_id"),
)
image_record_id: Mapped[int] = mapped_column( image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
) )
tag_id: Mapped[int] = mapped_column( tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
) )
rejected_at: Mapped[datetime] = mapped_column( rejected_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+9 -1
View File
@@ -15,7 +15,7 @@ backend.app.tasks.maintenance.recover_stalled_task_runs (Beat 5 min).
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, Integer, String, Text from sqlalchemy import DateTime, Index, Integer, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -24,6 +24,13 @@ from .base import Base
class TaskRun(Base): class TaskRun(Base):
__tablename__ = "task_run" __tablename__ = "task_run"
__table_args__ = (
# alembic 0016: the three task-history indexes (#3275).
Index("ix_task_run_name_started", "task_name", text("started_at DESC")),
Index("ix_task_run_queue_started", "queue", text("started_at DESC")),
Index("ix_task_run_status_started", "status", text("started_at DESC")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
celery_task_id: Mapped[str] = mapped_column( celery_task_id: Mapped[str] = mapped_column(
String(64), nullable=False, index=True, String(64), nullable=False, index=True,
@@ -40,6 +47,7 @@ class TaskRun(Base):
duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True) duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True, String(16), nullable=False, default="running", index=True,
server_default="running",
) )
error_type: Mapped[str | None] = mapped_column(String(128), nullable=True) error_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
error_message: Mapped[str | None] = mapped_column(Text, nullable=True) error_message: Mapped[str | None] = mapped_column(Text, nullable=True)