bvandeusen
5e1996e77f
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s
Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.
Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:
* 92 columns gained server_default. The models carried Python-side
`default=` only, so the ORM filled the value and the column had no
database default. Anything inserting outside the ORM behaved
differently from production.
* Eleven indexes that existed only in migrations are now declared:
the three backup_run reporting indexes, the two date-ordered
image_record browse indexes, import_task and presentation_review,
and the three task_run history indexes. All use text() for their DESC
ordering and postgresql_where for the partial one.
* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
because neither is expressible as a UniqueConstraint:
uq_tag_name_kind_fandom — an EXPRESSION index over
(name, kind, COALESCE(fandom_id, 0))
uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
IS NULL
post.py already had a comment describing the second one. The comment
was right; nothing declared it.
* The two external_link enum CHECKs (host, status) — rule 36 territory,
and absent from the model entirely.
* Two indexes were named explicitly. A bare index=True generated
ix_tag_alias_canonical_tag_id where the database has
ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
index that was already there under another name. Same for
tag_suggestion_rejection.
Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.
Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.
The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
77 lines
3.1 KiB
Python
77 lines
3.1 KiB
Python
"""GpuJob — a unit of GPU work the desktop agent pulls over HTTP (#114).
|
|
|
|
The durable work list that lets the agent stay HTTP-only: the server enqueues a
|
|
job per (image, task) — e.g. detect figures + CCIP-embed — and the agent LEASES a
|
|
batch, computes on its GPU, then SUBMITS results, all over the already-exposed web
|
|
API. Redis/Postgres stay private. A lease has an expiry; the lease query itself
|
|
re-claims expired leases (agent died / stopped mid-batch), so the queue is
|
|
self-healing without a separate sweep. One job is per ITEM; the agent fans a
|
|
VIDEO out into per-frame instances internally (see image_region.frame_time).
|
|
|
|
State: pending → leased → done | error (a failure under the attempt cap returns to
|
|
pending for another agent).
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
DateTime,
|
|
ForeignKey,
|
|
Index,
|
|
Integer,
|
|
String,
|
|
Text,
|
|
func,
|
|
text,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class GpuJob(Base):
|
|
__tablename__ = "gpu_job"
|
|
|
|
# Partial indexes over just the live slice (see migration 0070): the lease
|
|
# reads the lowest-id pending jobs on the hot path, and reclaims expired
|
|
# leases as a backstop — both stay O(batch) as done/error history grows.
|
|
__table_args__ = (
|
|
Index("ix_gpu_job_pending", "id", postgresql_where=text("status = 'pending'")),
|
|
Index(
|
|
"ix_gpu_job_leased_expires", "lease_expires_at",
|
|
postgresql_where=text("status = 'leased'"),
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
image_record_id: Mapped[int] = mapped_column(
|
|
ForeignKey("image_record.id", ondelete="CASCADE"), index=True
|
|
)
|
|
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
|
|
task: Mapped[str] = mapped_column(String(32), nullable=False)
|
|
status: Mapped[str] = mapped_column(
|
|
String(16), nullable=False, default="pending", index=True,
|
|
server_default="pending",
|
|
)
|
|
# pending | leased | done | error
|
|
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
|
leased_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True
|
|
)
|
|
lease_expires_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True
|
|
)
|
|
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
|
|
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
# Triage verdict for an ERRORED job (#125): NULL = not yet probed;
|
|
# 'defect' = the integrity probe says the FILE itself is bad (surfaced for
|
|
# recovery, excluded from /retry_errors); 'file_ok' = the file passes —
|
|
# the failure was operational (timeout/transient), safe to retry.
|
|
triage_status: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|