feat: link the Patreon teaser to the Discord drop it announced (388 step E5)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-web (push) Successful in 1m3s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m53s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m12s
The point of the milestone rather than its tail. Two of the operator's artists post a deliberately cropped fragment on Patreon to signal that the real thing has landed in their Discord; this proposes those pairs. Confirm-only, following the FC-6.3 series matcher. A wrongly-asserted association tells the operator two different pieces are one, which is strictly worse than no link: no link leaves them where they already were, a wrong one actively misinforms and then propagates into whatever reads it. So the matcher's job is a SHORT list worth reading, not a long list worth trusting. **The threshold sits above every single signal weight, and that is the design.** Proximity is 0.55, declaration 0.45, the cut 0.60 — so neither signal can carry a pair alone. That makes "time proximity alone is never sufficient" an arithmetic property rather than an aspiration: on a busy day an artist posts several times, and a matcher that could pair on proximity alone would turn every one of those days into false pairs until the review queue got abandoned. A guard test asserts the relationship against WEIGHTS directly, so it survives any refactor of the scorer, and says in its own failure message not to fix it by lowering the assertion. **Crop-to-source matching is HELD, on the plan's instruction** — real work with real false-positive risk, worth building only once signals 1 and 2 are shown insufficient against the operator's actual artists. Worth stating: a naive whole-image SigLIP similarity is NOT that signal. A cropped teaser and its full version are precisely the pair a whole-image comparison handles worst, so adding one as a "bonus" would mostly add noise while looking like progress. Two premises in the plan corrected in the building: * **E4 is not actually a prerequisite.** A Patreon Source and a Discord Source the operator has added under one Artist already share `Post.artist_id`, and the synthetic grouping inherits it. E4 EXTENDS this to creators FC has to learn the association for; it is not needed to represent one FC was told. Same-artist is then a hard filter, not a scored signal — two different creators posting minutes apart is a coincidence, not evidence. * **`link_extract` cannot supply the declaration signal.** It exists, but `SUPPORTED_HOSTS` is file hosts only and `host_for()` returns None for a Discord URL, so no ExternalLink row is ever written for one. The signal reads the post body directly instead. And a bug my own test would have caught: `declared_signal` stripped the HTML before looking for an invite, but `html_to_plain` discards attributes and these creators put the invite in an anchor's `href` — so the strongest form of the signal was being thrown away, leaving only whatever the link text said. The invite now matches the raw body; the bare mention still matches stripped text, so `\bdiscord\b` is tested against prose rather than against markup. Dismissed rows are kept, not deleted: the row is what remembers the rejection, and re-proposing a rejected pair on every scan is the one behaviour that makes a review queue get ignored. Both FKs CASCADE, so E3's one-DELETE reversal cannot leave a proposal pointing at a post that no longer exists. Only ACCEPTED links reach the post payload. A pending proposal is a question for the review queue, not a claim to render beside the artwork. UI (rule 27) follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
This commit is contained in:
@@ -5,6 +5,8 @@ from quart import Blueprint, jsonify, request
|
||||
from ..extensions import get_session
|
||||
from ..models import ImportSettings, Post
|
||||
from ..services import interpreter_client as ic
|
||||
from ..services.post_association_service import PostAssociationService
|
||||
from ..services.post_association_service import rescan as association_rescan
|
||||
from ..services.post_feed_service import PostFeedService
|
||||
from ..services.source_service import KNOWN_PLATFORMS
|
||||
from ..utils.text import html_to_plain
|
||||
@@ -165,3 +167,46 @@ async def set_translation_override(post_id: int):
|
||||
"translated_source_lang": post.translated_source_lang,
|
||||
"applied": applied,
|
||||
})
|
||||
|
||||
|
||||
# --- #388 E5: the announcement review queue -------------------------------
|
||||
#
|
||||
# Confirm-only, following the series-suggestion routes (api/tags.py). Nothing
|
||||
# here links anything on its own: the matcher proposes, the operator decides.
|
||||
|
||||
|
||||
@posts_bp.route("/associations", methods=["GET"])
|
||||
async def list_associations():
|
||||
async with get_session() as session:
|
||||
return jsonify({"items": await PostAssociationService(session).list_pending()})
|
||||
|
||||
|
||||
@posts_bp.route("/associations/<int:association_id>/accept", methods=["POST"])
|
||||
async def accept_association(association_id: int):
|
||||
async with get_session() as session:
|
||||
result = await PostAssociationService(session).accept(association_id)
|
||||
if result is None:
|
||||
return _bad("association not found", 404)
|
||||
await session.commit()
|
||||
return jsonify(result)
|
||||
|
||||
|
||||
@posts_bp.route("/associations/<int:association_id>/dismiss", methods=["POST"])
|
||||
async def dismiss_association(association_id: int):
|
||||
async with get_session() as session:
|
||||
result = await PostAssociationService(session).dismiss(association_id)
|
||||
if result is None:
|
||||
return _bad("association not found", 404)
|
||||
await session.commit()
|
||||
return jsonify(result)
|
||||
|
||||
|
||||
@posts_bp.route("/associations/rescan", methods=["POST"])
|
||||
async def rescan_associations():
|
||||
"""Manual re-scan. The beat sweep only looks at recent posts (a pair has to
|
||||
be within the window to exist at all); this is the button for a first run
|
||||
over a library that predates the feature."""
|
||||
async with get_session() as session:
|
||||
result = await association_rescan(session)
|
||||
await session.commit()
|
||||
return jsonify(result)
|
||||
|
||||
@@ -39,6 +39,10 @@ _EDITABLE_FIELDS = (
|
||||
"download_failure_warning_threshold",
|
||||
"series_suggest_enabled",
|
||||
"series_suggest_threshold",
|
||||
# #388 E5 — the announcement matcher (Patreon teaser ↔ Discord drop).
|
||||
"discord_link_enabled",
|
||||
"discord_link_threshold",
|
||||
"discord_link_window_hours",
|
||||
"extdl_mega_enabled",
|
||||
"extdl_gdrive_enabled",
|
||||
"extdl_mediafire_enabled",
|
||||
@@ -150,6 +154,22 @@ async def update_import_settings():
|
||||
return jsonify(
|
||||
{"error": "series_suggest_threshold must be a number in [0, 1]"}
|
||||
), 400
|
||||
if "discord_link_enabled" in body and not isinstance(
|
||||
body["discord_link_enabled"], bool
|
||||
):
|
||||
return jsonify({"error": "discord_link_enabled must be a boolean"}), 400
|
||||
if "discord_link_threshold" in body:
|
||||
v = body["discord_link_threshold"]
|
||||
if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
|
||||
return jsonify(
|
||||
{"error": "discord_link_threshold must be a number in [0, 1]"}
|
||||
), 400
|
||||
if "discord_link_window_hours" in body:
|
||||
v = body["discord_link_window_hours"]
|
||||
if not isinstance(v, (int, float)) or isinstance(v, bool) or v <= 0:
|
||||
return jsonify(
|
||||
{"error": "discord_link_window_hours must be a positive number"}
|
||||
), 400
|
||||
if "wip_title_tagging_enabled" in body and not isinstance(
|
||||
body["wip_title_tagging_enabled"], bool
|
||||
):
|
||||
|
||||
@@ -207,6 +207,13 @@ def make_celery() -> Celery:
|
||||
# (#388 E2), so this sweep is what picks up a drop once its
|
||||
# vectors have caught up. No-op unless discord_grouping_enabled.
|
||||
},
|
||||
"match-post-associations-hourly": {
|
||||
"task": "backend.app.tasks.maintenance.match_post_associations",
|
||||
"schedule": 3600.0, # hourly, and AFTER the grouper's own cadence
|
||||
# by construction: a pair cannot be proposed until the drop it
|
||||
# points at exists as a grouping (#388 E5). No-op unless
|
||||
# discord_link_enabled.
|
||||
},
|
||||
"integrity-verify-weekly": {
|
||||
"task": "backend.app.tasks.maintenance.verify_integrity",
|
||||
"schedule": 604800.0, # weekly
|
||||
|
||||
@@ -28,6 +28,7 @@ from .pixiv_failed_media import PixivFailedMedia
|
||||
from .pixiv_seen_media import PixivSeenMedia
|
||||
from .platform_membership import PlatformMembership
|
||||
from .post import Post
|
||||
from .post_association import PostAssociation
|
||||
from .post_attachment import PostAttachment, attachment_download_url
|
||||
from .presentation_review import PresentationReview
|
||||
from .series_chapter import SeriesChapter
|
||||
@@ -59,6 +60,7 @@ __all__ = [
|
||||
"SubscribeStarFailedMedia",
|
||||
"SubscribeStarSeenMedia",
|
||||
"Post",
|
||||
"PostAssociation",
|
||||
"PostAttachment",
|
||||
"attachment_download_url",
|
||||
"PresentationReview",
|
||||
|
||||
@@ -97,6 +97,33 @@ class ImportSettings(Base):
|
||||
server_default="0.5",
|
||||
)
|
||||
|
||||
# Milestone 388 E5 — the announcement matcher: "this Patreon post announced
|
||||
# that Discord drop". Lives here rather than in MLSettings, with the series
|
||||
# matcher it is modelled on, because it runs no inference: the signals are
|
||||
# time proximity and whether the post says so.
|
||||
discord_link_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, default=True,
|
||||
server_default="true",
|
||||
)
|
||||
# The weighted-score cut-off. 0.60 is not arbitrary: it is deliberately set
|
||||
# ABOVE the largest single signal weight, which is what makes "time
|
||||
# proximity alone must never be sufficient" an ARITHMETIC property rather
|
||||
# than a hope. On a busy day an artist posts several times; if proximity
|
||||
# could carry a pair by itself, every one of those days would produce false
|
||||
# pairs and the review queue would be abandoned. See
|
||||
# post_association_service.WEIGHTS — a guard test pins the relationship.
|
||||
discord_link_threshold: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=0.60,
|
||||
server_default="0.60",
|
||||
)
|
||||
# How far apart the announcement and the drop may be. The Patreon post
|
||||
# exists IN ORDER TO announce the drop, so they are minutes-to-hours apart;
|
||||
# a day is generous and still excludes "same week".
|
||||
discord_link_window_hours: Mapped[float] = mapped_column(
|
||||
Float, nullable=False, default=24.0,
|
||||
server_default="24",
|
||||
)
|
||||
|
||||
# #830 off-platform file-host downloads — per-host enable lever (default on,
|
||||
# rule #26). Column names are extdl_<host>_enabled so the worker reads them
|
||||
# via getattr(settings, f"extdl_{host}_enabled", True).
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
"""PostAssociation — "this Patreon post announced that Discord drop".
|
||||
|
||||
Milestone 388, step E5, and the point of the milestone rather than its tail.
|
||||
|
||||
Two of the operator's artists post a deliberately CROPPED fragment on Patreon
|
||||
to signal that the real thing has landed in their Discord. The Patreon post is
|
||||
the announcement; the Discord grouping (milestone 388 E2) is the payload. This
|
||||
row is the link between them.
|
||||
|
||||
## Directional, and NOT a merge
|
||||
|
||||
`announcement` → `payload` is asymmetric on purpose. The teaser announces the
|
||||
drop; the drop does not announce the teaser, and a symmetric "related posts"
|
||||
edge would lose the only thing that makes the pair interesting.
|
||||
|
||||
Nor are the two collapsed into one post. The creator published twice,
|
||||
deliberately, on two platforms with different audiences — flattening that
|
||||
hides the very behaviour being modelled, and would destroy the operator's
|
||||
ability to see that the Patreon post is a teaser at all.
|
||||
|
||||
## Confirm-only, following FC-6.3 (task 737)
|
||||
|
||||
`status` starts at `pending` and nothing is linked until the operator accepts.
|
||||
A wrongly-asserted association tells them two different pieces are one, which
|
||||
is worse than no link: no link leaves them where they already are, a wrong one
|
||||
actively misinforms. Same reason the series matcher writes to a review queue
|
||||
instead of filing posts on its own.
|
||||
|
||||
`status` is a plain String, no CHECK — matching `series_suggestion.status`,
|
||||
which records the same check-existing-enums lesson.
|
||||
|
||||
## Why pHash could not do this, and the correction matters
|
||||
|
||||
The original plan claimed this link was already sitting in `image_provenance`
|
||||
via pHash dedup. It is not. `compute_phash` is `imagehash.phash` at
|
||||
`hash_size=8` — a DCT hash over the WHOLE image, robust to rescaling and
|
||||
recompression but NOT to cropping, because a crop changes the global
|
||||
signature. Cross-platform provenance still links straight re-posts; it does
|
||||
nothing for a cropped teaser and its full version, which is precisely the pair
|
||||
the operator described. Hence a scored proposal rather than a lookup.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
DateTime,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Integer,
|
||||
String,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class PostAssociation(Base):
|
||||
__tablename__ = "post_association"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"announcement_post_id", "payload_post_id",
|
||||
name="uq_post_association_pair",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
# The teaser — a real post the creator wrote (Patreon, today).
|
||||
announcement_post_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
# What it announced — a synthetic Discord grouping, today. CASCADE on both
|
||||
# sides: an association to a post that no longer exists is not a fact worth
|
||||
# keeping, and E3's reversal path (delete the grouping) should not leave a
|
||||
# dangling proposal behind.
|
||||
payload_post_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
|
||||
score: Mapped[float] = mapped_column(Float, nullable=False)
|
||||
# Per-signal strengths as scored, so a proposal stays explicable after the
|
||||
# weights or the threshold are tuned. Without it, "why was this suggested"
|
||||
# is unanswerable the moment anything moves.
|
||||
signals: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||
# pending | linked | dismissed. A DISMISSED row is kept, not deleted — it
|
||||
# is what stops the matcher proposing the same rejected pair on every
|
||||
# subsequent scan, which is the behaviour that makes a review queue
|
||||
# unusable.
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16), nullable=False, server_default="pending", index=True
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False,
|
||||
server_default=func.now(), onupdate=func.now(),
|
||||
)
|
||||
@@ -0,0 +1,308 @@
|
||||
"""The announcement matcher: which Patreon post announced which Discord drop.
|
||||
|
||||
Milestone 388, step E5.
|
||||
|
||||
Two of the operator's artists post a deliberately CROPPED fragment on Patreon
|
||||
to signal that the real thing has landed in their Discord. This service
|
||||
proposes those pairs, and proposes only — the operator accepts or dismisses,
|
||||
following the FC-6.3 series matcher (task 737) rather than linking on its own.
|
||||
|
||||
## Why confirm-only is not caution for its own sake
|
||||
|
||||
A wrongly-asserted association tells the operator that two different pieces are
|
||||
one. That is strictly worse than no link at all: no link leaves them exactly
|
||||
where they already were, a wrong one actively misinforms and then propagates
|
||||
into whatever reads the association. So the matcher's job is to make a SHORT
|
||||
list worth reading, not a long list worth trusting.
|
||||
|
||||
## Signals, and the one deliberately NOT built
|
||||
|
||||
1. **Time proximity.** The Patreon post exists in order to announce the drop,
|
||||
so the two are minutes-to-hours apart. Nearly free, and strong.
|
||||
2. **The post says so.** These announcements routinely name Discord or carry
|
||||
an invite link, which is close to a declaration.
|
||||
|
||||
3. **Crop-to-source matching is HELD, on the plan's own instruction** — it is
|
||||
real work with real false-positive risk, and it is only worth building once
|
||||
1 and 2 are shown to be insufficient against the operator's actual artists.
|
||||
Nothing here should be read as evidence it is unnecessary; it is deferred,
|
||||
and the thing that would justify it is an empty review queue on a pair the
|
||||
operator can see with their own eyes.
|
||||
|
||||
Note also that a naive whole-image SigLIP similarity is NOT that signal. A
|
||||
cropped teaser and its full version are exactly the pair a whole-image
|
||||
comparison handles worst, so adding one as a "bonus" would mostly add noise
|
||||
while looking like progress.
|
||||
|
||||
## Creator identity comes free, so E4 is not actually a prerequisite
|
||||
|
||||
The plan listed E4 (creator identity across the two channels) as a dependency.
|
||||
It is not one for the pairs that matter today: a Patreon `Source` and a Discord
|
||||
`Source` the operator has added under the same `Artist` already share
|
||||
`Post.artist_id`, and the synthetic grouping inherits it (E2). E4 EXTENDS this
|
||||
to creators whose association FC has to learn rather than being told; it is not
|
||||
needed to represent an association FC already knows.
|
||||
|
||||
## A correction the plan carried, worth restating
|
||||
|
||||
`link_extract.py` does exist and does capture off-platform links — but only
|
||||
file hosts (`SUPPORTED_HOSTS` is mega/gdrive/mediafire/dropbox/pixeldrain).
|
||||
`host_for()` returns None for a Discord URL, so no `ExternalLink` row is ever
|
||||
written for one. The declaration signal therefore reads the post body itself
|
||||
rather than the extracted-links table the plan assumed it could use.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from ..models import ImportSettings, Post, PostAssociation
|
||||
from ..utils.text import html_to_plain
|
||||
from .discord_grouping import DROP_GROUPER
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Additive weights, summing to 1.0. Kept as constants rather than settings —
|
||||
# the sensitivity knob that matters is the threshold, and per-signal weights
|
||||
# are an over-tune (same call as series_match_service.WEIGHTS).
|
||||
#
|
||||
# THE RELATIONSHIP TO THE THRESHOLD IS THE DESIGN. No single weight may reach
|
||||
# the default threshold, which is what makes "time proximity alone is never
|
||||
# enough" arithmetic rather than aspirational: on a busy day an artist posts
|
||||
# several times, and a matcher that could pair on proximity alone would turn
|
||||
# every busy day into false pairs. A guard test pins this.
|
||||
WEIGHTS = {"proximity": 0.55, "declared": 0.45}
|
||||
|
||||
# A Discord INVITE in the body is close to a declaration; the bare word is
|
||||
# weaker but still meaningful, because these posts are short and on-topic.
|
||||
_INVITE = re.compile(r"discord\.(?:gg|com/invite)/", re.I)
|
||||
_MENTION = re.compile(r"\bdiscord\b", re.I)
|
||||
|
||||
DECLARED_INVITE = 1.0
|
||||
DECLARED_MENTION = 0.6
|
||||
|
||||
MAX_CANDIDATES = 25
|
||||
|
||||
|
||||
def proximity_signal(gap: timedelta, window: timedelta) -> float:
|
||||
"""1.0 when the two posts are simultaneous, decaying linearly to 0 at the
|
||||
window's edge. Linear rather than a step, so a pair an hour outside a
|
||||
hand-tuned window degrades instead of vanishing."""
|
||||
if window <= timedelta(0):
|
||||
return 0.0
|
||||
seconds = abs(gap.total_seconds())
|
||||
if seconds >= window.total_seconds():
|
||||
return 0.0
|
||||
return round(1.0 - (seconds / window.total_seconds()), 4)
|
||||
|
||||
|
||||
def declared_signal(description: str | None) -> float:
|
||||
"""Does the announcement say, in its own body, that this is about Discord?
|
||||
|
||||
The invite is matched against the RAW body and the bare mention against the
|
||||
stripped text, which is not fussiness — post bodies are HTML, and these
|
||||
creators put the invite in an anchor's `href`. `html_to_plain` discards
|
||||
attributes, so stripping first would have thrown away the strongest form of
|
||||
the signal and left only whatever the link text happened to say.
|
||||
|
||||
The mention still reads stripped text, so `\bdiscord\b` is matched against
|
||||
prose rather than against markup and URLs, where it would fire on any link
|
||||
that merely passes through a discord domain.
|
||||
"""
|
||||
if not description:
|
||||
return 0.0
|
||||
if _INVITE.search(description):
|
||||
return DECLARED_INVITE
|
||||
if _MENTION.search(html_to_plain(description) or ""):
|
||||
return DECLARED_MENTION
|
||||
return 0.0
|
||||
|
||||
|
||||
def weighted_score(signals: dict) -> float:
|
||||
return round(sum(WEIGHTS[k] * signals.get(k, 0.0) for k in WEIGHTS), 4)
|
||||
|
||||
|
||||
def _post_time(post: Post) -> datetime:
|
||||
return post.post_date or post.downloaded_at
|
||||
|
||||
|
||||
class PostAssociationService:
|
||||
def __init__(self, session: AsyncSession):
|
||||
self.session = session
|
||||
|
||||
async def _decided(self, announcement_id: int) -> set[int]:
|
||||
"""Payload posts already proposed for this announcement, in ANY status.
|
||||
|
||||
Dismissed pairs are included deliberately: re-proposing a pair the
|
||||
operator has already rejected on every subsequent scan is the single
|
||||
behaviour that makes a review queue get ignored.
|
||||
"""
|
||||
rows = (await self.session.execute(
|
||||
select(PostAssociation.payload_post_id)
|
||||
.where(PostAssociation.announcement_post_id == announcement_id)
|
||||
)).scalars().all()
|
||||
return set(rows)
|
||||
|
||||
async def _candidate_groups(
|
||||
self, announcement: Post, *, window: timedelta,
|
||||
) -> list[Post]:
|
||||
"""Synthetic Discord groupings by the SAME artist, inside the window.
|
||||
|
||||
Same-artist is the identity signal and it is free (see the module
|
||||
docstring on E4). It is also a hard filter rather than a scored one:
|
||||
two different creators posting minutes apart is a coincidence, not
|
||||
evidence, and letting it score at all would mean a busy hour across the
|
||||
library could out-vote everything else.
|
||||
"""
|
||||
at = _post_time(announcement)
|
||||
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
|
||||
return (await self.session.execute(
|
||||
select(Post)
|
||||
.where(
|
||||
Post.artist_id == announcement.artist_id,
|
||||
Post.synthesized_by == DROP_GROUPER,
|
||||
Post.id != announcement.id,
|
||||
sort_key >= at - window,
|
||||
sort_key <= at + window,
|
||||
)
|
||||
.order_by(sort_key)
|
||||
.limit(MAX_CANDIDATES)
|
||||
)).scalars().all()
|
||||
|
||||
async def match_post(
|
||||
self, announcement_id: int, *, threshold: float, window_hours: float,
|
||||
) -> int:
|
||||
"""Score one announcement against nearby groupings. Returns proposals made."""
|
||||
announcement = await self.session.get(Post, announcement_id)
|
||||
if announcement is None or announcement.synthesized_by is not None:
|
||||
# A synthetic post cannot announce anything — FC wrote it.
|
||||
return 0
|
||||
|
||||
window = timedelta(hours=window_hours)
|
||||
declared = declared_signal(announcement.description)
|
||||
already = await self._decided(announcement_id)
|
||||
|
||||
made = 0
|
||||
for group in await self._candidate_groups(announcement, window=window):
|
||||
if group.id in already:
|
||||
continue
|
||||
signals = {
|
||||
"proximity": proximity_signal(
|
||||
_post_time(group) - _post_time(announcement), window,
|
||||
),
|
||||
"declared": declared,
|
||||
}
|
||||
score = weighted_score(signals)
|
||||
if score < threshold:
|
||||
continue
|
||||
self.session.add(PostAssociation(
|
||||
announcement_post_id=announcement.id,
|
||||
payload_post_id=group.id,
|
||||
score=score,
|
||||
signals=signals,
|
||||
status="pending",
|
||||
))
|
||||
made += 1
|
||||
return made
|
||||
|
||||
async def list_pending(self) -> list[dict]:
|
||||
rows = (await self.session.execute(
|
||||
select(PostAssociation)
|
||||
.where(PostAssociation.status == "pending")
|
||||
.order_by(PostAssociation.score.desc(), PostAssociation.id.desc())
|
||||
)).scalars().all()
|
||||
return [
|
||||
{
|
||||
"id": a.id,
|
||||
"announcement_post_id": a.announcement_post_id,
|
||||
"payload_post_id": a.payload_post_id,
|
||||
"score": a.score,
|
||||
"signals": a.signals,
|
||||
}
|
||||
for a in rows
|
||||
]
|
||||
|
||||
async def accept(self, association_id: int) -> dict | None:
|
||||
a = await self.session.get(PostAssociation, association_id)
|
||||
if a is None:
|
||||
return None
|
||||
a.status = "linked"
|
||||
return {"id": a.id, "status": a.status}
|
||||
|
||||
async def dismiss(self, association_id: int) -> dict | None:
|
||||
a = await self.session.get(PostAssociation, association_id)
|
||||
if a is None:
|
||||
return None
|
||||
# Kept, not deleted — the row is what remembers the rejection.
|
||||
a.status = "dismissed"
|
||||
return {"id": a.id, "status": a.status}
|
||||
|
||||
async def linked_for(self, post_ids: list[int]) -> dict[int, list[dict]]:
|
||||
"""Accepted links touching these posts, keyed by post id, BOTH ways.
|
||||
|
||||
A post is either end of the relationship, and each end wants the other
|
||||
one: the teaser wants "the full set is over here", the grouping wants
|
||||
"this is what announced me". One query, both directions.
|
||||
"""
|
||||
if not post_ids:
|
||||
return {}
|
||||
rows = (await self.session.execute(
|
||||
select(PostAssociation).where(
|
||||
PostAssociation.status == "linked",
|
||||
or_(
|
||||
PostAssociation.announcement_post_id.in_(post_ids),
|
||||
PostAssociation.payload_post_id.in_(post_ids),
|
||||
),
|
||||
)
|
||||
)).scalars().all()
|
||||
out: dict[int, list[dict]] = {}
|
||||
for a in rows:
|
||||
if a.announcement_post_id in post_ids:
|
||||
out.setdefault(a.announcement_post_id, []).append(
|
||||
{"role": "announces", "post_id": a.payload_post_id, "id": a.id}
|
||||
)
|
||||
if a.payload_post_id in post_ids:
|
||||
out.setdefault(a.payload_post_id, []).append(
|
||||
{"role": "announced_by", "post_id": a.announcement_post_id, "id": a.id}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict:
|
||||
"""Score every recent non-synthetic post against nearby groupings."""
|
||||
settings = await ImportSettings.load(session)
|
||||
if not settings.discord_link_enabled:
|
||||
return {"enabled": False, "scanned": 0, "proposed": 0}
|
||||
|
||||
now = now or datetime.now(UTC)
|
||||
window_hours = float(settings.discord_link_window_hours)
|
||||
# Only look at announcements that could still have a partner in range —
|
||||
# a full-library rescan is the manual button's job, not the sweep's.
|
||||
horizon = now - timedelta(hours=window_hours * 2)
|
||||
sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
|
||||
ids = (await session.execute(
|
||||
select(Post.id).where(
|
||||
Post.synthesized_by.is_(None),
|
||||
Post.absorbed_by_post_id.is_(None),
|
||||
sort_key >= horizon,
|
||||
)
|
||||
)).scalars().all()
|
||||
|
||||
svc = PostAssociationService(session)
|
||||
proposed = 0
|
||||
for pid in ids:
|
||||
proposed += await svc.match_post(
|
||||
pid,
|
||||
threshold=float(settings.discord_link_threshold),
|
||||
window_hours=window_hours,
|
||||
)
|
||||
log.info(
|
||||
"discord announcement matcher: scanned %d post(s), proposed %d pair(s)",
|
||||
len(ids), proposed,
|
||||
)
|
||||
return {"enabled": True, "scanned": len(ids), "proposed": proposed}
|
||||
@@ -175,9 +175,10 @@ class PostFeedService:
|
||||
post_ids = [p.id for p, _, _ in rows]
|
||||
thumbs_map = await self._thumbnails_for(post_ids)
|
||||
atts_map = await self._attachments_for(post_ids)
|
||||
links_map = await self._links_for(post_ids)
|
||||
|
||||
items = [
|
||||
self._to_dict(post, artist, source, thumbs_map, atts_map)
|
||||
self._to_dict(post, artist, source, thumbs_map, atts_map, links_map)
|
||||
for post, artist, source in rows
|
||||
]
|
||||
return {"items": items, "next_cursor": next_cursor}
|
||||
@@ -218,6 +219,7 @@ class PostFeedService:
|
||||
atts_map = await self._attachments_for([anchor_post.id])
|
||||
anchor_item = self._to_dict(
|
||||
anchor_post, anchor_artist, anchor_source, thumbs_map, atts_map,
|
||||
await self._links_for([anchor_post.id]),
|
||||
)
|
||||
return {
|
||||
"items": newer["items"] + [anchor_item] + older["items"],
|
||||
@@ -241,7 +243,10 @@ class PostFeedService:
|
||||
# the default arg.
|
||||
thumbs_map = await self._thumbnails_for([post.id], limit=None)
|
||||
atts_map = await self._attachments_for([post.id])
|
||||
item = self._to_dict(post, artist, source, thumbs_map, atts_map)
|
||||
item = self._to_dict(
|
||||
post, artist, source, thumbs_map, atts_map,
|
||||
await self._links_for([post.id]),
|
||||
)
|
||||
item["description_full"] = html_to_plain(post.description)
|
||||
# Full (uncapped) translated description for the detail view (#143).
|
||||
item["description_translated_full"] = post.description_translated
|
||||
@@ -407,9 +412,25 @@ class PostFeedService:
|
||||
})
|
||||
return out
|
||||
|
||||
async def _links_for(self, post_ids: list[int]) -> dict[int, list[dict]]:
|
||||
"""Accepted announcement links touching these posts (#388 E5).
|
||||
|
||||
Only ACCEPTED ones. A pending proposal is a question for the review
|
||||
queue, not a claim to render beside the artwork — showing one here
|
||||
would assert a link the operator has not agreed to, which is the exact
|
||||
failure the confirm-only design exists to prevent.
|
||||
"""
|
||||
# Imported here rather than at module scope: post_association_service
|
||||
# imports discord_grouping, which imports the models, and the feed
|
||||
# service is imported by the API at startup. A local import keeps that
|
||||
# chain out of the import graph for a purely optional read.
|
||||
from .post_association_service import PostAssociationService
|
||||
|
||||
return await PostAssociationService(self.session).linked_for(post_ids)
|
||||
|
||||
def _to_dict(
|
||||
self, post: Post, artist: Artist, source: Source | None,
|
||||
thumbs_map: dict, atts_map: dict,
|
||||
thumbs_map: dict, atts_map: dict, links_map: dict | None = None,
|
||||
) -> dict:
|
||||
plain_full = html_to_plain(post.description) if post.description else None
|
||||
if plain_full is None:
|
||||
@@ -453,6 +474,11 @@ class PostFeedService:
|
||||
# ago" — which is the whole signal that chat content is trickling
|
||||
# in. NULL means it has not grown since it was created.
|
||||
"last_grew_at": post.last_grew_at.isoformat() if post.last_grew_at else None,
|
||||
# Accepted links only (#388 E5): [{role, post_id, id}], where role
|
||||
# is "announces" (this post is the teaser) or "announced_by" (this
|
||||
# post is the drop). Always a list so the UI never branches on
|
||||
# absence.
|
||||
"associations": (links_map or {}).get(post.id, []),
|
||||
# Non-null on a chat message a synthetic post absorbed. The feed
|
||||
# filters these out, but `around`/`get_post` still reach them, and
|
||||
# the UI uses this to explain why a post it linked to is not in the
|
||||
|
||||
@@ -1171,3 +1171,36 @@ def group_discord_drops() -> str:
|
||||
f"sources={res['sources']} created={res['posts_created']} "
|
||||
f"joined={res['images_joined']}"
|
||||
)
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.maintenance.match_post_associations",
|
||||
soft_time_limit=900, time_limit=1200,
|
||||
)
|
||||
def match_post_associations() -> str:
|
||||
"""Milestone 388 E5: propose which Patreon post announced which Discord drop.
|
||||
|
||||
Proposes only — every pair lands in a review queue and nothing is linked
|
||||
until the operator accepts. Maintenance lane for the same reason as the
|
||||
grouper: no inference, no ML library, and it must not depend on the
|
||||
optional ml-worker being present.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from ..services.post_association_service import rescan
|
||||
from ._async_session import async_session_factory
|
||||
|
||||
async def _run() -> dict:
|
||||
async_factory, engine = async_session_factory()
|
||||
try:
|
||||
async with async_factory() as session:
|
||||
result = await rescan(session)
|
||||
await session.commit()
|
||||
return result
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
res = asyncio.run(_run())
|
||||
if not res["enabled"]:
|
||||
return "disabled"
|
||||
return f"scanned={res['scanned']} proposed={res['proposed']}"
|
||||
|
||||
Reference in New Issue
Block a user