feat: FC authors the post that Discord never wrote (milestone 388 step E2)
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m5s
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 1m5s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m5s
Discord is a delivery channel, not a publisher. One message is not one post,
and today every message lands as its own `post` row, so chat lines compete
with authored work for the same surface. Rather than demote them into a
second-class feed, FC now writes the post itself: one row per DROP, its
images the drop's images, its body the messages' text in arrival order.
Synthesising a `Post` (rather than inventing a parallel entity) is the whole
point — the result is post-shaped by construction, so feed, provenance,
translation, attachments and series keep working on it unchanged.
The predicate is three axes ANDed, and the time one does the real work:
same source AND cosine distance <= threshold AND no gap > window
Similarity alone over-groups, and that is the failure that would make this
useless: any two pieces of the same character by the same artist sit close in
SigLIP space, so a cosine-only rule collapses a month of one character into a
single "post". Two details inside the predicate are load-bearing —
* distance is measured to the group's SEED, never to the previous member,
because chaining lets a group DRIFT: twenty small steps walk from one piece
to a completely different one, each hop individually within threshold;
* the window is measured between CONSECUTIVE messages, not from the first, so
an artist trickling variants out over an evening stays one drop.
Why a post-import sweep and not part of ingest. The obvious alternative was to
migrate Discord to the native post-first ingester (#1266) and group at capture
time. That cannot work: the grouping signal is `siglip_embedding`, which is
produced asynchronously AFTER import (tasks/ml.py, the GPU backfill), so at
capture time there is nothing to group on. Grouping is necessarily something
that happens once the vectors catch up — hence a re-runnable sweep that skips
what it cannot yet place, and an hourly (not daily) cadence.
The honesty rule, enforced in the schema. `post.synthesized_by` names the
grouper; `synthesis_details` records the members, the count, and the
thresholds AS THEY WERE (they are operator-tunable, so without that "why did
it group these" is unanswerable a month later). Member posts are absorbed, not
destroyed — they remain the images' true origin and the audit trail — and
`absorbed_by_post_id` is ON DELETE SET NULL, so deleting a synthetic post
releases its members back into the feed in one DELETE with no repair step.
`post_title` stays NULL deliberately: a synthesised title is the one place
this could put words in a creator's mouth.
Two guards the first draft would have failed:
* the per-run cap took the lowest post IDs, not the oldest posts — DISTINCT ON
forces its own ORDER BY, so the sort now happens outside the subquery;
* a cap landing mid-drop would have published a truncated group claiming to be
a whole drop, so the last group is left for the next run.
And one vacuous test caught before it shipped: the support vector perturbed a
single component of an all-ones vector, moving it ~1e-6, so every distance
assertion passed regardless of what the predicate did. `_vec` now builds a
unit vector at a stated angle, where distance is exactly 1 - cos(delta) —
rule 167, a guard has to be able to fail.
UI (rule 27) follows in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
This commit is contained in:
@@ -1131,3 +1131,40 @@ def vacuum_analyze() -> dict:
|
||||
done.append(table)
|
||||
log.info("vacuum_analyze complete: %s", done)
|
||||
return {"vacuumed": done}
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.maintenance.group_discord_drops",
|
||||
soft_time_limit=1800, time_limit=2100,
|
||||
)
|
||||
def group_discord_drops() -> str:
|
||||
"""Milestone 388 E2: group Discord message-posts into the drops FC authors.
|
||||
|
||||
Lives on the MAINTENANCE lane, not the ml lane, even though it reads SigLIP
|
||||
vectors — it does no inference and imports no ML library, and the ml-worker
|
||||
is an OPTIONAL container (B3). Routing it to 'ml' would silently disable
|
||||
grouping on every stack that runs a GPU agent and drops that container,
|
||||
which is the same trap gpu_queue.py was moved here to avoid.
|
||||
|
||||
Async body under its own loop, per the _async_session contract: the sweep
|
||||
needs pgvector column reads and the shared services are async.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from ..services.discord_grouping import sweep
|
||||
from ._async_session import async_session_factory
|
||||
|
||||
async def _run() -> dict:
|
||||
async_factory, engine = async_session_factory()
|
||||
try:
|
||||
async with async_factory() as session:
|
||||
result = await sweep(session)
|
||||
await session.commit()
|
||||
return result
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
res = asyncio.run(_run())
|
||||
if not res["enabled"]:
|
||||
return "disabled"
|
||||
return f"sources={res['sources']} created={res['posts_created']}"
|
||||
|
||||
Reference in New Issue
Block a user