feat: an open grouping — a later drop joins its post (milestone 388 step E3)

A synthetic post is no longer sealed at creation. A creator who adds two more
variants the next day extends the existing post, its body grows with the new
messages, and no rival post appears. That is what makes chat capture read as
content trickling in rather than as a stream of separate arrivals.

The sweep now runs two passes per source and the ORDER is load-bearing: offer
new messages to still-open groups BEFORE founding new ones, because whichever
runs first claims a message.

E3's three named problems, each answered rather than discovered later:

**Bridging.** A candidate near two groups joins NEITHER. Nearest-wins would
silently make an arbitrary choice between two posts the operator may already
have seen; merging them is worse still, because a merge rewrites history and
anything pointing at the absorbed post dangles. Leaving it to found its own
group is the recoverable failure. AMBIGUITY_MARGIN is a module constant and
deliberately not a setting — it is not a quality dial anyone would tune toward
a better feed, and exposing it would invite turning it to zero, which is
exactly the silent arbitrary choice it prevents.

**Re-surfacing without thrashing.** A grouping has two dates, and which one
orders the feed is a real decision, so the feed orders by neither directly.
Ordering by when the drop STARTED buries a group that grows a week later under
a week of other posts — defeating the point of keeping it open. Ordering by
every growth lets a group gaining one image a day live permanently at the top,
so chat out-competes authored posts for the front page — the opposite of "post
pacing stays front and centre". Instead `resurfaced_at` moves only when growth
clears BOTH a minimum-images bar and a cooldown, so a drip-feed updates in
place and a genuine second wave resurfaces exactly once. It is NULL on every
ordinary post, so the sort key COALESCEs through it without moving anything
that is not a grouping.

**Reopening forever.** Groups close after a quiet period — artists reuse
characters for years, and a group left open indefinitely will eventually
absorb something it shouldn't. Openness is DERIVED, not stored: a group is
open if it grew (or started) within the window. Lowering the setting closes
old groups and raising it reopens them, with nothing to repair either way; a
stored closed_at would have needed a sweep to set it and a repair path to ever
change the policy.

Rule 89 is satisfied structurally rather than by a parallel mechanism:
celery_signals writes a TaskRun for every task, which already supplies
duration, the 5-minute stalled-run recovery, and retention pruning. What this
step owed on top of that was a wall-clock limit (present) and idempotence —
re-running the joiner adds nothing, asserted directly rather than left to the
unique (image, post) constraint to catch.

Two bugs fixed in the writing, one of which my own test would have hit:

* `assign_to_group` sorted bare (distance, Post) tuples, which falls through
  to comparing Posts when two distances tie — and a perfectly symmetric
  bridge, the exact case the function exists for, would have raised TypeError
  instead of declining to choose. Now keyed on the distance alone.
* The cursor was still built from `post_date or downloaded_at` while the
  ORDER BY had gained `resurfaced_at`. Two expressions that disagree at a page
  boundary don't error, they silently skip or repeat rows; both sites now go
  through one `_post_sort_value`, and a test pages through one row at a time
  to prove the walk matches the whole list.

Image linking is now one shared helper rather than written twice, because
creation and joining would otherwise be free to drift on exactly the detail
(which post owns the image) that makes a grouping reversible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9

tests/test_discord_grouping.py

@@ -16,9 +16,12 @@ from sqlalchemy import select
 from backend.app.models import Artist, ImageProvenance, ImageRecord, MLSettings, Post, Source
 from backend.app.services.discord_grouping import (
     DROP_GROUPER,
+    assign_to_group,
     build_groups,
     cosine_distance,
     group_source,
+    join_open_groups,
+    should_resurface,
     sweep,
 )
 from backend.app.services.post_feed_service import PostFeedService
@@ -387,3 +390,341 @@ async def test_the_sweep_only_touches_discord_sources(db):
     db.expunge_all()
     assert (await db.get(Post, a.id)).absorbed_by_post_id is None
     assert (await db.get(Post, b.id)).absorbed_by_post_id is None
+
+
+# --- E3: an open grouping -------------------------------------------------
+#
+# Three hard problems, one test class each. The anti-thrash rule is ASSERTED
+# rather than assumed, because "a group that grows daily monopolises the feed"
+# is the kind of defect nobody notices until they have lived with it.
+
+
+def test_growth_below_the_minimum_never_moves_the_post():
+    """One image a day must not chain bumps together."""
+    now = datetime.now(UTC)
+    assert not should_resurface(
+        images_since_surface=1, last_surface_at=now - timedelta(days=30),
+        now=now, min_images=2, cooldown=timedelta(hours=24),
+    )
+
+
+def test_growth_inside_the_cooldown_never_moves_the_post():
+    """Even a big second wave waits — otherwise a burst bumps once per sweep."""
+    now = datetime.now(UTC)
+    assert not should_resurface(
+        images_since_surface=10, last_surface_at=now - timedelta(hours=1),
+        now=now, min_images=2, cooldown=timedelta(hours=24),
+    )
+
+
+def test_a_real_second_wave_moves_the_post_once():
+    now = datetime.now(UTC)
+    assert should_resurface(
+        images_since_surface=3, last_surface_at=now - timedelta(days=2),
+        now=now, min_images=2, cooldown=timedelta(hours=24),
+    )
+
+
+def _group(post_id: int, angle: float):
+    return (Post(id=post_id), _vec(angle))
+
+
+def test_a_candidate_near_two_groups_joins_neither():
+    """Bridging. Merging would rewrite history — two posts the operator may
+    already have seen become one — so the recoverable answer is to leave it and
+    let it found its own group."""
+    left, right = _group(1, -0.30), _group(2, 0.30)
+    assert assign_to_group(_vec(0.0), [left, right], max_distance=0.5) is None
+
+
+def test_a_candidate_clearly_nearer_one_group_joins_it():
+    near, far = _group(1, 0.02), _group(2, 1.2)
+    chosen = assign_to_group(_vec(0.0), [near, far], max_distance=0.5)
+    assert chosen is not None and chosen.id == 1
+
+
+def test_a_candidate_near_nothing_joins_nothing():
+    assert assign_to_group(
+        _vec(0.0), [_group(1, math.pi / 2)], max_distance=0.10,
+    ) is None
+
+
+@pytest.mark.asyncio
+async def test_a_later_variant_joins_the_existing_post_instead_of_founding_one(db):
+    artist, source = await _seed(db, name="rejoin-artist")
+    start = datetime.now(UTC) - timedelta(days=3)
+    await _message(db, source, artist, ext="r1", at=start, vec=_vec(0.0), text="first")
+    await _message(
+        db, source, artist, ext="r2", at=start + timedelta(minutes=2),
+        vec=_vec(0.05), text="second",
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    group = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalar_one()
+    group_id = group.id
+
+    # A day later — far outside the drop window, well inside the open period.
+    await _message(
+        db, source, artist, ext="r3", at=start + timedelta(days=1),
+        vec=_vec(0.06), text="next day",
+    )
+    await db.commit()
+
+    joined = await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2,
+        resurface_cooldown_hours=24,
+    )
+    await db.commit()
+    assert joined == 1
+
+    # No second post — that is the point of the step.
+    posts = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalars().all()
+    assert len(posts) == 1
+
+    db.expunge_all()
+    grown = await db.get(Post, group_id)
+    assert grown.last_grew_at is not None
+    assert grown.synthesis_details["message_count"] == 3
+    # The new message's text joined the body, in arrival order.
+    assert grown.description == "first\n\nsecond\n\nnext day"
+
+
+@pytest.mark.asyncio
+async def test_a_variant_after_the_group_closed_starts_a_new_post(db):
+    """Groups must not stay open forever — artists reuse characters for years."""
+    artist, source = await _seed(db, name="closed-artist")
+    start = datetime.now(UTC) - timedelta(days=60)
+    await _message(db, source, artist, ext="c1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="c2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    await _message(
+        db, source, artist, ext="c3", at=datetime.now(UTC) - timedelta(days=1),
+        vec=_vec(0.06),
+    )
+    await db.commit()
+
+    joined = await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2,
+        resurface_cooldown_hours=24,
+    )
+    await db.commit()
+    assert joined == 0, "a closed group must not absorb a two-month-later drop"
+
+    created = await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+    assert created == 1
+
+
+@pytest.mark.asyncio
+async def test_joining_does_not_duplicate_images(db):
+    """The unique (image, post) constraint is the backstop, not the argument —
+    a second join pass must add nothing at all."""
+    artist, source = await _seed(db, name="nodupe-artist")
+    start = datetime.now(UTC) - timedelta(days=3)
+    await _message(db, source, artist, ext="d1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="d2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    await _message(
+        db, source, artist, ext="d3", at=start + timedelta(days=1), vec=_vec(0.06),
+    )
+    await db.commit()
+
+    kwargs = {
+        "max_distance": 0.10, "window_minutes": 60, "close_after_hours": 168,
+        "resurface_min_images": 2, "resurface_cooldown_hours": 24,
+    }
+    assert await join_open_groups(db, source, **kwargs) == 1
+    await db.commit()
+    assert await join_open_groups(db, source, **kwargs) == 0
+    await db.commit()
+
+    group = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalar_one()
+    links = (await db.execute(
+        select(ImageProvenance.image_record_id).where(ImageProvenance.post_id == group.id)
+    )).scalars().all()
+    assert len(links) == len(set(links)) == 3
+
+
+@pytest.mark.asyncio
+async def test_a_group_that_grows_by_one_does_not_move_in_the_feed(db):
+    """The anti-thrash rule, end to end. A drip-feed updates IN PLACE."""
+    artist, source = await _seed(db, name="dripfeed-artist")
+    start = datetime.now(UTC) - timedelta(days=3)
+    await _message(db, source, artist, ext="t1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="t2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    await _message(
+        db, source, artist, ext="t3", at=start + timedelta(days=1), vec=_vec(0.06),
+    )
+    await db.commit()
+    await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2, resurface_cooldown_hours=24,
+    )
+    await db.commit()
+
+    group = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalar_one()
+    assert group.last_grew_at is not None, "it grew"
+    assert group.resurfaced_at is None, "but one image must not move it"
+    assert group.synthesis_details["images_since_surface"] == 1
+
+
+@pytest.mark.asyncio
+async def test_a_second_wave_resurfaces_the_post(db):
+    artist, source = await _seed(db, name="secondwave-artist")
+    start = datetime.now(UTC) - timedelta(days=3)
+    await _message(db, source, artist, ext="w1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="w2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    for i, ext in enumerate(("w3", "w4", "w5")):
+        await _message(
+            db, source, artist, ext=ext,
+            at=start + timedelta(days=1, minutes=i), vec=_vec(0.06),
+        )
+    await db.commit()
+    await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2, resurface_cooldown_hours=24,
+    )
+    await db.commit()
+
+    group = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalar_one()
+    assert group.resurfaced_at is not None
+    # The counter resets, so the NEXT trickle starts from zero rather than
+    # riding the same three images into a second bump.
+    assert group.synthesis_details["images_since_surface"] == 0
+
+
+@pytest.mark.asyncio
+async def test_a_resurfaced_group_sorts_by_when_it_moved_not_when_it_started(db):
+    """The feed ordering decision, pinned. The group's drop began three days
+    ago — older than the decoy post — but it resurfaced just now, so it leads."""
+    artist, source = await _seed(db, name="feedorder-artist")
+    start = datetime.now(UTC) - timedelta(days=3)
+    await _message(db, source, artist, ext="f1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="f2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    # An ordinary post from yesterday: newer than the drop's start, older than
+    # the moment the group resurfaces.
+    decoy = Post(
+        source_id=source.id, artist_id=artist.id, external_post_id="decoy",
+        post_date=datetime.now(UTC) - timedelta(days=1), post_title="Decoy",
+    )
+    db.add(decoy)
+    await db.flush()
+    decoy_id = decoy.id
+
+    for i, ext in enumerate(("f3", "f4", "f5")):
+        await _message(
+            db, source, artist, ext=ext,
+            at=start + timedelta(days=1, minutes=i), vec=_vec(0.06),
+        )
+    await db.commit()
+    await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2, resurface_cooldown_hours=24,
+    )
+    await db.commit()
+
+    group = (await db.execute(
+        select(Post).where(Post.synthesized_by == DROP_GROUPER)
+    )).scalar_one()
+    page = await PostFeedService(db).scroll(
+        cursor=None, artist_id=artist.id, limit=50,
+    )
+    ids = [i["id"] for i in page["items"]]
+    assert ids.index(group.id) < ids.index(decoy_id), (
+        "a resurfaced group must lead a post published after the drop STARTED"
+    )
+    # And it is still reported as a grouping that grew.
+    item = next(i for i in page["items"] if i["id"] == group.id)
+    assert item["last_grew_at"] is not None
+
+
+@pytest.mark.asyncio
+async def test_pagination_stays_stable_when_a_group_has_resurfaced(db):
+    """The cursor is built in Python and the ORDER BY in SQL. If they disagree
+    on resurfaced_at's precedence, rows are silently skipped or repeated at
+    every page boundary — so page through one at a time and check."""
+    artist, source = await _seed(db, name="cursor-artist")
+    start = datetime.now(UTC) - timedelta(days=5)
+    await _message(db, source, artist, ext="x1", at=start, vec=_vec(0.0))
+    await _message(
+        db, source, artist, ext="x2", at=start + timedelta(minutes=2), vec=_vec(0.05),
+    )
+    await db.commit()
+    await group_source(db, source, max_distance=0.10, window_minutes=60)
+    await db.commit()
+
+    for i in range(4):
+        db.add(Post(
+            source_id=source.id, artist_id=artist.id,
+            external_post_id=f"plain{i}",
+            post_date=datetime.now(UTC) - timedelta(days=i + 1),
+        ))
+    for i, ext in enumerate(("x3", "x4", "x5")):
+        await _message(
+            db, source, artist, ext=ext,
+            at=start + timedelta(days=1, minutes=i), vec=_vec(0.06),
+        )
+    await db.commit()
+    await join_open_groups(
+        db, source, max_distance=0.10, window_minutes=60,
+        close_after_hours=168, resurface_min_images=2, resurface_cooldown_hours=24,
+    )
+    await db.commit()
+
+    svc = PostFeedService(db)
+    everything = [i["id"] for i in (
+        await svc.scroll(cursor=None, artist_id=artist.id, limit=100)
+    )["items"]]
+
+    walked, cursor = [], None
+    while True:
+        page = await svc.scroll(
+            cursor=cursor, artist_id=artist.id, limit=1,
+        )
+        walked.extend(i["id"] for i in page["items"])
+        cursor = page["next_cursor"]
+        if cursor is None:
+            break
+    assert walked == everything