bvandeusenandClaude Opus 5
73eeb7a377
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m5s
Discord is a delivery channel, not a publisher. One message is not one post,
and today every message lands as its own `post` row, so chat lines compete
with authored work for the same surface. Rather than demote them into a
second-class feed, FC now writes the post itself: one row per DROP, its
images the drop's images, its body the messages' text in arrival order.
Synthesising a `Post` (rather than inventing a parallel entity) is the whole
point — the result is post-shaped by construction, so feed, provenance,
translation, attachments and series keep working on it unchanged.
The predicate is three axes ANDed, and the time one does the real work:
same source AND cosine distance <= threshold AND no gap > window
Similarity alone over-groups, and that is the failure that would make this
useless: any two pieces of the same character by the same artist sit close in
SigLIP space, so a cosine-only rule collapses a month of one character into a
single "post". Two details inside the predicate are load-bearing —
* distance is measured to the group's SEED, never to the previous member,
because chaining lets a group DRIFT: twenty small steps walk from one piece
to a completely different one, each hop individually within threshold;
* the window is measured between CONSECUTIVE messages, not from the first, so
an artist trickling variants out over an evening stays one drop.
Why a post-import sweep and not part of ingest. The obvious alternative was to
migrate Discord to the native post-first ingester (#1266) and group at capture
time. That cannot work: the grouping signal is `siglip_embedding`, which is
produced asynchronously AFTER import (tasks/ml.py, the GPU backfill), so at
capture time there is nothing to group on. Grouping is necessarily something
that happens once the vectors catch up — hence a re-runnable sweep that skips
what it cannot yet place, and an hourly (not daily) cadence.
The honesty rule, enforced in the schema. `post.synthesized_by` names the
grouper; `synthesis_details` records the members, the count, and the
thresholds AS THEY WERE (they are operator-tunable, so without that "why did
it group these" is unanswerable a month later). Member posts are absorbed, not
destroyed — they remain the images' true origin and the audit trail — and
`absorbed_by_post_id` is ON DELETE SET NULL, so deleting a synthetic post
releases its members back into the feed in one DELETE with no repair step.
`post_title` stays NULL deliberately: a synthesised title is the one place
this could put words in a creator's mouth.
Two guards the first draft would have failed:
* the per-run cap took the lowest post IDs, not the oldest posts — DISTINCT ON
forces its own ORDER BY, so the sort now happens outside the subquery;
* a cap landing mid-drop would have published a truncated group claiming to be
a whole drop, so the last group is left for the next run.
And one vacuous test caught before it shipped: the support vector perturbed a
single component of an all-ones vector, moving it ~1e-6, so every distance
assertion passed regardless of what the predicate did. `_vec` now builds a
unit vector at a stated angle, where distance is exactly 1 - cos(delta) —
rule 167, a guard has to be able to fail.
UI (rule 27) follows in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
139 lines
6.7 KiB
Python
139 lines
6.7 KiB
Python
"""Post — provenance anchor for one creator post (may contain many images).
|
|
|
|
`source_id` is nullable since alembic 0030 — filesystem-imported posts
|
|
with no live subscription have NULL source_id. `artist_id` is the
|
|
denormalized always-present link to the creator (added in 0030 so
|
|
artist-filter queries don't depend on the Source detour).
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
JSON,
|
|
CheckConstraint,
|
|
DateTime,
|
|
ForeignKey,
|
|
Index,
|
|
Integer,
|
|
String,
|
|
Text,
|
|
UniqueConstraint,
|
|
func,
|
|
text,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class Post(Base):
|
|
__tablename__ = "post"
|
|
__table_args__ = (
|
|
# alembic 0030. The comment above described this index; nothing declared
|
|
# it, so autogenerate proposed dropping it (#3275).
|
|
Index("uq_post_artist_external_id_null_source", "artist_id", "external_post_id",
|
|
unique=True, postgresql_where=text("source_id IS NULL")),
|
|
# Source-bound dedup. Postgres treats NULL != NULL so rows
|
|
# with source_id IS NULL aren't deduped by this constraint;
|
|
# the partial unique index `uq_post_artist_external_id_null_source`
|
|
# (created in alembic 0030) covers that case via
|
|
# (artist_id, external_post_id).
|
|
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
|
CheckConstraint(
|
|
"translation_override IN ('auto', 'force', 'original')",
|
|
# Bare name: Base.metadata's naming convention prepends
|
|
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
|
|
# alembic 0088, which renames the four constraints that shipped
|
|
# that way (#3275).
|
|
name="translation_override",
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
source_id: Mapped[int | None] = mapped_column(
|
|
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
|
|
)
|
|
# Denormalized; always equals source.artist_id when source_id is set
|
|
# (the importer is responsible for keeping them consistent on insert).
|
|
# Filter queries (artist detail, artist-scoped posts feed) use this
|
|
# directly instead of joining through Source.
|
|
artist_id: Mapped[int] = mapped_column(
|
|
ForeignKey("artist.id", ondelete="CASCADE"),
|
|
nullable=False, index=True,
|
|
)
|
|
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
post_title: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
post_date: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
|
|
raw_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
|
|
|
description: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
attachment_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
|
|
# -- Post-text translation (milestone 143). Filled by the translate_posts
|
|
# sweep via the Interpreter LAN service so viewing is instant.
|
|
# translated_source_lang is the DETECTED original language; "en" (or a
|
|
# passthrough) means nothing to translate and the *_translated columns stay
|
|
# NULL. engine_version keys the Interpreter cache — re-runs are ~1ms and a
|
|
# model upgrade re-translates instead of serving stale.
|
|
post_title_translated: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
description_translated: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
translated_source_lang: Mapped[str | None] = mapped_column(
|
|
String(8), nullable=True
|
|
)
|
|
translation_engine_version: Mapped[str | None] = mapped_column(
|
|
String(128), nullable=True
|
|
)
|
|
translated_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True
|
|
)
|
|
# Sticky per-post override of the translation decision (milestone 155):
|
|
# 'auto' = the acceptance gate decides; 'force' = always store Interpreter's
|
|
# translation even below the confidence floor (rescue a skipped legit-foreign
|
|
# title); 'original' = never translate, keep the original (kill a confidently
|
|
# mis-flagged one the floor can't catch). The sweep reads this on every run,
|
|
# and re-translate leaves 'original' posts alone, so the choice survives a
|
|
# Re-translate-all.
|
|
translation_override: Mapped[str] = mapped_column(
|
|
String(16), nullable=False, default="auto", server_default="auto",
|
|
)
|
|
|
|
downloaded_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
|
|
# -- Synthetic posts (milestone 388). ----------------------------------
|
|
# Discord is a delivery CHANNEL, not a publisher: one message is not one
|
|
# post. So FC authors the post itself, grouping a creator's variant drop
|
|
# into a single row (services/discord_grouping.py).
|
|
#
|
|
# NULL for every post a creator actually wrote — which is all of them until
|
|
# a grouper runs. Non-NULL names the grouper that authored this row, and is
|
|
# the ONE flag the UI keys off to say so. The honesty rule is the whole
|
|
# point: a synthetic post must never present itself as authored, and a
|
|
# column that is absent-or-a-name makes "was this us?" answerable from the
|
|
# row rather than inferred from its shape.
|
|
#
|
|
# Plain String, no CHECK (rule 36 considered and declined) — same reasoning
|
|
# as source.error_type and service_seen.kind. There is exactly one grouper
|
|
# today; a second would be a value, not an invariant.
|
|
synthesized_by: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
|
# What it was built from, so the operator can audit a grouping FC invented:
|
|
# member post ids, message count, and the thresholds in force when the
|
|
# decision was made. That last part matters — the thresholds are operator-
|
|
# tunable, so "why did it group these" is unanswerable a month later
|
|
# without recording the values that produced it.
|
|
synthesis_details: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
|
# Set on a MEMBER post, pointing at the synthetic post that absorbed it.
|
|
# The feed hides absorbed posts (they are the chat lines the synthetic post
|
|
# replaced); every other surface still reaches them by id, because they
|
|
# remain the image's true origin and the grouping has to be inspectable.
|
|
#
|
|
# Self-FK, ON DELETE SET NULL: deleting a synthetic post un-absorbs its
|
|
# members and they return to the feed on their own. That is the reversal
|
|
# path, and it is one DELETE — nothing to undo by hand.
|
|
absorbed_by_post_id: Mapped[int | None] = mapped_column(
|
|
ForeignKey("post.id", ondelete="SET NULL"), nullable=True, index=True
|
|
)
|