Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s
Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.
Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:
* 92 columns gained server_default. The models carried Python-side
`default=` only, so the ORM filled the value and the column had no
database default. Anything inserting outside the ORM behaved
differently from production.
* Eleven indexes that existed only in migrations are now declared:
the three backup_run reporting indexes, the two date-ordered
image_record browse indexes, import_task and presentation_review,
and the three task_run history indexes. All use text() for their DESC
ordering and postgresql_where for the partial one.
* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
because neither is expressible as a UniqueConstraint:
uq_tag_name_kind_fandom — an EXPRESSION index over
(name, kind, COALESCE(fandom_id, 0))
uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
IS NULL
post.py already had a comment describing the second one. The comment
was right; nothing declared it.
* The two external_link enum CHECKs (host, status) — rule 36 territory,
and absent from the model entirely.
* Two indexes were named explicitly. A bare index=True generated
ix_tag_alias_canonical_tag_id where the database has
ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
index that was already there under another name. Same for
tag_suggestion_rejection.
Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.
Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.
The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
157 lines
7.3 KiB
Python
157 lines
7.3 KiB
Python
"""ImportSettings — single-row table holding the importer's tunable knobs.
|
|
|
|
Enforced as a single row via a CHECK (id = 1) constraint. The application
|
|
always SELECTs id=1 and never inserts/deletes after the initial migration.
|
|
"""
|
|
|
|
from sqlalchemy import Boolean, CheckConstraint, Float, Integer, Text, select
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class ImportSettings(Base):
|
|
__tablename__ = "import_settings"
|
|
# Bare constraint name — Base.metadata's naming convention applies the
|
|
# ck_<table>_<name> prefix, producing the final ck_import_settings_singleton.
|
|
__table_args__ = (CheckConstraint("id = 1", name="singleton"),)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import", server_default="/import")
|
|
|
|
min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
|
min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
|
|
|
skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
|
transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9, server_default="0.9")
|
|
|
|
skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
|
|
single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
|
|
single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
|
|
|
|
phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
|
|
|
|
# FC-3c downloader knobs
|
|
download_rate_limit_seconds: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=3.0,
|
|
server_default="3",
|
|
)
|
|
download_validate_files: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True,
|
|
server_default="true",
|
|
)
|
|
|
|
# FC-3d scheduling knobs
|
|
download_schedule_default_seconds: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=28800,
|
|
server_default="28800",
|
|
)
|
|
download_event_retention_days: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=90,
|
|
server_default="90",
|
|
)
|
|
download_failure_warning_threshold: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=5,
|
|
server_default="5",
|
|
)
|
|
|
|
# FC-3h backup knobs.
|
|
backup_db_nightly_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=False,
|
|
server_default="false",
|
|
)
|
|
backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=3,
|
|
server_default="3",
|
|
)
|
|
backup_db_keep_last_n: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=14,
|
|
server_default="14",
|
|
)
|
|
backup_images_keep_last_n: Mapped[int] = mapped_column(
|
|
Integer, nullable=False, default=3,
|
|
server_default="3",
|
|
)
|
|
|
|
# FC-6.3 series continuation matcher. enabled gates the rescan; threshold is
|
|
# the weighted-score cut-off (0..1) above which a pending suggestion is made.
|
|
series_suggest_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True,
|
|
server_default="true",
|
|
)
|
|
series_suggest_threshold: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=0.5,
|
|
server_default="0.5",
|
|
)
|
|
|
|
# #830 off-platform file-host downloads — per-host enable lever (default on,
|
|
# rule #26). Column names are extdl_<host>_enabled so the worker reads them
|
|
# via getattr(settings, f"extdl_{host}_enabled", True).
|
|
extdl_mega_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
extdl_gdrive_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
extdl_mediafire_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
extdl_dropbox_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
extdl_pixeldrain_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
|
|
# -- Post-text translation via the Interpreter LAN service (milestone 143).
|
|
# Off by default with NO default host — it needs a reachable Interpreter
|
|
# service (the operator's, behind a reverse proxy), which not every install
|
|
# has; the operator sets the URL and flips it on. Empty base_url OR disabled
|
|
# → the translate sweep no-ops.
|
|
translation_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=False, server_default="false",
|
|
)
|
|
interpreter_base_url: Mapped[str] = mapped_column(
|
|
Text, nullable=False, default="", server_default="",
|
|
)
|
|
translation_target_lang: Mapped[str] = mapped_column(
|
|
Text, nullable=False, default="en", server_default="en",
|
|
)
|
|
# The latin-script acceptance floor for the translation gate: a translation
|
|
# whose Interpreter-reported confidence is below this is kept as the original
|
|
# (operator-tunable, milestone 155). Default 0.9 — stricter than the old
|
|
# hardcoded 0.8, because Interpreter confidently mis-detects short ASCII
|
|
# English (e.g. "… WIP Part 1") as a European language at ~0.86. CJK stays
|
|
# trusted regardless (script-detected). Per-post overrides handle the misses.
|
|
translation_min_confidence: Mapped[float] = mapped_column(
|
|
Float, nullable=False, default=0.9, server_default="0.9",
|
|
)
|
|
|
|
# Title-based WIP auto-tagging (task #1458). When a freshly-imported post's
|
|
# TITLE explicitly declares work-in-progress ("WIP" / "work in progress"),
|
|
# the importer applies the `wip` system tag to its images — the artist's own
|
|
# label, used to keep unfinished pieces out of the Explore/gallery browse. ON
|
|
# by default (rule 26 — the feature works out of the box). Gates only the
|
|
# LIVE import hook; the existing catalogue is caught by the operator-triggered
|
|
# "Scan existing posts" backfill (which runs regardless of this flag).
|
|
wip_title_tagging_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=True, server_default="true",
|
|
)
|
|
# Soft WIP title tier (#1474): also tag sketch/doodle/scribble titles, but with
|
|
# a PROVISIONAL source (`wip_title_soft`) that never trains the head, since these
|
|
# are lower-precision (a finished "sketch" isn't WIP). OFF by default — a lower-
|
|
# precision tier is opt-in (the ring-loud audit surfaces false positives).
|
|
wip_soft_title_tagging_enabled: Mapped[bool] = mapped_column(
|
|
Boolean, nullable=False, default=False, server_default="false",
|
|
)
|
|
|
|
@classmethod
|
|
async def load(cls, session) -> ImportSettings:
|
|
"""The singleton settings row (id=1), via an async session."""
|
|
return (await session.execute(select(cls).where(cls.id == 1))).scalar_one()
|
|
|
|
@classmethod
|
|
def load_sync(cls, session) -> ImportSettings:
|
|
"""The singleton settings row (id=1), via a sync session."""
|
|
return session.execute(select(cls).where(cls.id == 1)).scalar_one()
|