feat: the membership sweep, and the state that makes its failures readable (387 C3)

A daily sweep that walks each platform's roster into `platform_membership`.
Daily because memberships change on a BILLING cycle, not a download cadence.

**`membership_sync` is the part that earns its keep.** Without it three very
different situations are one indistinguishable state — the account subscribes
to nothing, the sweep never ran, the sweep failed — and all three leave zero
rows in `platform_membership`. "You are tracking 12 sources you no longer
subscribe to" is correct in the first case and an invitation to cancel things
the operator is actively paying for in the other two. So C4 gates its
CONCLUSIONS on `last_success_at`, not merely its display, and `roster_is_fresh`
is computed server-side so no caller can forget to.

Two timestamps rather than one: `last_attempt_at` moves every run,
`last_success_at` only on a clean walk. The gap between them is the signal —
a sweep hammering a broken credential every day must not look healthy because
it ran recently, and there is a test for exactly that.

Rejected shortcuts, both tempting: `MAX(platform_membership.last_seen_at)`
cannot tell "synced fine, found nothing" from "never synced"; `task_run` is
worse, since its retention prunes ok rows after 24h and a sweep that last
succeeded three days ago would leave no trace at all.

**The fetch completes before anything is written.** That ordering is the safety
property: a walk that dies mid-pagination writes nothing, so a failure can
never leave a roster half this week's and half last week's. `touch_membership`
never deletes, so a failure cannot empty the roster either — but "intact"
should mean intact, not merely non-empty.

Rule 89's four, each where it actually lives: recovery is "run it again"
(upsert, no deletes); retention is C1's age-out-never-delete, because
disappearing IS the signal; the wall-clock deadline is per-platform and
distinct from the per-REQUEST timeout the client already has (rule 156 — a
paginated roster answering every page slowly-but-within-timeout would never
trip that one and would sit on a worker indefinitely); duration comes from the
existing TaskRun signal plumbing.

**A bug caught in review, not production:** the broad `except Exception` would
have swallowed Celery's SoftTimeLimitExceeded — which is an ORDINARY Exception
subclass, not a BaseException — letting the sweep run past the soft limit into
the hard one, where it is SIGKILLed mid-transaction. A sweep that cannot be
stopped is worse than one that fails. Now re-raised explicitly, with a test
that also asserts SoftTimeLimitExceeded is still an Exception, so the re-raise
cannot quietly become dead code.

Rule 164 is why this ships with UI rather than backend-only: a roster that
never synced must be VISIBLE as such. The card says "never synced" in words and
states no count at all — rendering it as 0 is the precise conflation the whole
step exists to prevent — while a real zero behind a real sync is reported as
zero, because that one IS an answer. Pinned in both directions.

Three independent gates decide whether a platform is swept — registered here,
client exposes `iter_memberships`, credential exists — each silent, so adding
SubscribeStar (D1) is one line and nothing else. A missing credential is not an
error: recording a failure would light up the UI for a feature never enabled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9

backend/app/api/sources.py

@@ -1,10 +1,11 @@
 """FC-3a: CRUD over Source rows. FC-3c adds POST /<id>/check."""
 
 from quart import Blueprint, jsonify, request
-from sqlalchemy import select
+from sqlalchemy import func, select
 
 from ..extensions import get_session
-from ..models import DownloadEvent, Source
+from ..models import DownloadEvent, MembershipSync, PlatformMembership, Source
+from ..services.membership_roster import roster_is_fresh
 from ..services.scheduler_service import active_platform_cooldowns, scheduler_status
 from ..services.source_service import (
     KNOWN_PLATFORMS,
@@ -288,3 +289,57 @@ async def check_source(source_id: int):
     download_source.delay(source_id)
 
     return jsonify({"download_event_id": event_id, "status": "pending"}), 202
+
+
+# --- #387 C3: the membership roster's sync state --------------------------
+#
+# Rule 164's visibility requirement lives here. A roster that failed to sync,
+# or never has, must be DISTINGUISHABLE from an account that subscribes to
+# nothing — otherwise the reconciliation this unlocks would tell the operator
+# to cancel sources they are actively paying for.
+
+
+@sources_bp.route("/membership-sync", methods=["GET"])
+async def membership_sync_status():
+    async with get_session() as session:
+        rows = (await session.execute(select(MembershipSync))).scalars().all()
+        counts = dict(
+            (await session.execute(
+                select(PlatformMembership.platform, func.count())
+                .group_by(PlatformMembership.platform)
+            )).all()
+        )
+    return jsonify({"platforms": [
+        {
+            "platform": r.platform,
+            "last_attempt_at": r.last_attempt_at.isoformat() if r.last_attempt_at else None,
+            # NULL here means NEVER, and the UI must say so in words. Rendering
+            # it as 0 or as "-" is the exact conflation this endpoint exists to
+            # prevent.
+            "last_success_at": r.last_success_at.isoformat() if r.last_success_at else None,
+            "last_count": r.last_count,
+            "last_error_type": r.last_error_type,
+            "last_error_message": r.last_error_message,
+            # Whether a CONCLUSION may be drawn from this roster — not merely
+            # whether it looks recent. C4 gates on this, and it is computed
+            # server-side so no caller can forget to.
+            "fresh": roster_is_fresh(r),
+            "known_memberships": counts.get(r.platform, 0),
+        }
+        for r in sorted(rows, key=lambda r: r.platform)
+    ]})
+
+
+@sources_bp.route("/membership-sync", methods=["POST"])
+async def trigger_membership_sync():
+    """Run the roster sweep now.
+
+    The beat schedule runs daily, which is right for a billing-cycle fact but
+    far too slow when the operator has just connected a credential and wants to
+    see whether it works. Queued rather than run inline: it crosses the network
+    to an external service and the request path is not where that belongs.
+    """
+    from ..tasks.maintenance import sync_memberships
+
+    sync_memberships.delay()
+    return jsonify({"queued": True})

backend/app/celery_app.py

@@ -214,6 +214,12 @@ def make_celery() -> Celery:
                 # points at exists as a grouping (#388 E5). No-op unless
                 # discord_link_enabled.
             },
+            "sync-memberships-daily": {
+                "task": "backend.app.tasks.maintenance.sync_memberships",
+                "schedule": 86400.0,  # daily — memberships change on a BILLING
+                # cycle, not a download cadence (#387 C3). No-op per platform
+                # when the client lacks the seam or no credential exists.
+            },
             "integrity-verify-weekly": {
                 "task": "backend.app.tasks.maintenance.verify_integrity",
                 "schedule": 604800.0,  # weekly

backend/app/models/__init__.py

@@ -21,6 +21,7 @@ from .import_batch import ImportBatch
 from .import_settings import ImportSettings
 from .import_task import ImportTask
 from .library_audit_run import LibraryAuditRun
+from .membership_sync import MembershipSync
 from .ml_settings import MLSettings
 from .patreon_failed_media import PatreonFailedMedia
 from .patreon_seen_media import PatreonSeenMedia
@@ -82,6 +83,7 @@ __all__ = [
     "ImportTask",
     "ImportSettings",
     "LibraryAuditRun",
+    "MembershipSync",
     "MLSettings",
     "HeadAutoApplyRun",
     "HeadMetric",

backend/app/models/membership_sync.py

@@ -0,0 +1,77 @@
+"""membership_sync — did the roster actually sync, and when.
+
+Milestone 387, step C3.
+
+`platform_membership` records what was SEEN. This records whether looking
+happened at all, and that is a different fact — the one that makes an empty
+roster readable.
+
+## Why this table has to exist
+
+Without it, three very different situations are one indistinguishable state:
+
+* the account genuinely subscribes to nothing,
+* the sweep has never run,
+* the sweep ran and failed.
+
+All three produce zero rows in `platform_membership`. Telling the operator
+"you are tracking 12 sources you do not subscribe to" is correct in the first
+case and catastrophic in the other two — it is an invitation to cancel things
+they are actively paying for. C4 must therefore gate its CONCLUSIONS on
+`last_success_at`, not merely display it.
+
+`MAX(platform_membership.last_seen_at)` was the tempting shortcut and does not
+work: it cannot distinguish "synced fine, found nothing" from "never synced".
+`task_run` was the other candidate and is worse — its retention prunes ok rows
+after 24h, so a sweep that last succeeded three days ago would leave no trace
+at all.
+
+## Separate attempt and success timestamps, deliberately
+
+`last_attempt_at` moves every run; `last_success_at` moves only on a clean
+walk. The GAP between them is the staleness signal, and keeping them apart is
+what lets the UI say "last synced 3 days ago, last tried 20 minutes ago,
+failing" — which is a different message from either half alone.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import DateTime, Integer, String, Text, UniqueConstraint, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class MembershipSync(Base):
+    __tablename__ = "membership_sync"
+    __table_args__ = (
+        UniqueConstraint("platform", name="uq_membership_sync_platform"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    platform: Mapped[str] = mapped_column(String(64), nullable=False)
+
+    # Moves on EVERY run, success or not — so "we are trying" is visible even
+    # while "we are succeeding" is not.
+    last_attempt_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # Moves only on a COMPLETE walk. This is the freshness signal C4 gates its
+    # conclusions on; NULL means never — which must never be rendered as zero.
+    last_success_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # How many memberships the last SUCCESSFUL walk saw. Paired with
+    # last_success_at so "0" is only ever readable as a real zero.
+    last_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
+
+    # Cleared on success. Plain String, no CHECK — this carries an exception
+    # class name (PatreonAuthError, PatreonDriftError, ...) and the vocabulary
+    # is whatever the client raises, exactly as source.error_type works.
+    last_error_type: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    last_error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
+
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False,
+        server_default=func.now(), onupdate=func.now(),
+    )

backend/app/services/membership_roster.py

@@ -28,11 +28,14 @@ from __future__ import annotations
 
 import logging
 
-from sqlalchemy import func
+from collections.abc import Awaitable, Callable
+from datetime import UTC, datetime, timedelta
+
+from sqlalchemy import func, select
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from ..models import PlatformMembership
+from ..models import MembershipSync, PlatformMembership
 
 log = logging.getLogger(__name__)
 
@@ -146,3 +149,130 @@ async def touch_membership(
         },
     )
     await session.execute(stmt)
+
+
+# ---------------------------------------------------------------------------
+# The sweep, and the state that makes its failures readable (#387 C3)
+# ---------------------------------------------------------------------------
+#
+# How long a successful sync stays trustworthy. Beyond this the roster is
+# STALE, and C4 must refuse to draw conclusions from it — "you are tracking 12
+# sources you no longer subscribe to", computed from a roster that stopped
+# syncing a week ago, is an invitation to cancel things the operator is still
+# paying for.
+#
+# Generous relative to the daily cadence: a few missed runs are a blip, not a
+# reason to stop trusting a roster that changes on a billing cycle.
+ROSTER_STALE_AFTER = timedelta(days=3)
+
+
+async def get_sync_state(session: AsyncSession, platform: str) -> MembershipSync | None:
+    return (await session.execute(
+        select(MembershipSync).where(MembershipSync.platform == platform)
+    )).scalar_one_or_none()
+
+
+def roster_is_fresh(state: MembershipSync | None, *, now: datetime | None = None) -> bool:
+    """May a caller draw CONCLUSIONS from this roster?
+
+    False for never-synced and for stale, and those are deliberately the same
+    answer here even though the UI must tell them apart: both mean the roster
+    is not evidence. The asymmetry that matters is that `False` never means
+    "you subscribe to nothing" — it means "we do not know", and a caller that
+    cannot represent "we do not know" must not be asking this question.
+    """
+    if state is None or state.last_success_at is None:
+        return False
+    now = now or datetime.now(UTC)
+    return (now - state.last_success_at) <= ROSTER_STALE_AFTER
+
+
+async def _record_sync(session: AsyncSession, platform: str, **values) -> None:
+    stmt = pg_insert(MembershipSync).values(platform=platform, **values)
+    await session.execute(stmt.on_conflict_do_update(
+        constraint="uq_membership_sync_platform",
+        set_={**values, "updated_at": func.now()},
+    ))
+
+
+async def sync_platform(
+    session: AsyncSession,
+    *,
+    platform: str,
+    fetch: Callable[[], Awaitable[list]],
+    now: datetime | None = None,
+) -> dict:
+    """Walk one platform's roster and record what happened.
+
+    `fetch` is injected rather than built here so the error-to-state mapping —
+    the part with the consequences — is testable without a credential, and so
+    this service needs to know nothing about how any particular client is
+    constructed.
+
+    THE FETCH COMPLETES BEFORE ANYTHING IS WRITTEN. That ordering is the whole
+    safety property: a walk that dies half way through pagination writes
+    nothing, so a failure can never leave a roster that is partly this week's
+    and partly last week's. (`touch_membership` never deletes, so a failure
+    cannot empty the roster either — but "intact" should mean intact, not
+    merely non-empty.)
+
+    Returns a summary dict; never raises for a platform failure, because one
+    platform failing must not abort the others.
+    """
+    now = now or datetime.now(UTC)
+    await _record_sync(session, platform, last_attempt_at=now)
+    await session.commit()
+
+    try:
+        memberships = await fetch()
+    except Exception as exc:  # noqa: BLE001 - deliberately broad, see below
+        # Broad on purpose: a sweep is a background job, and ANY escape here
+        # kills the run for every other platform too. The exception's class
+        # name is recorded so the distinction the client drew (auth vs drift
+        # vs transport) survives into the UI, which is where it is actionable.
+        #
+        # EXCEPT the worker asking us to stop. Celery raises its soft time
+        # limit as an ordinary Exception subclass, so a broad catch swallows
+        # the shutdown request and lets the sweep run on into the HARD limit,
+        # where it is SIGKILLed mid-transaction. A sweep that cannot be stopped
+        # is worse than one that fails. (KeyboardInterrupt and SystemExit are
+        # BaseException and pass through this clause already.)
+        from celery.exceptions import SoftTimeLimitExceeded
+
+        if isinstance(exc, SoftTimeLimitExceeded):
+            raise
+        await session.rollback()
+        await _record_sync(
+            session, platform,
+            last_error_type=type(exc).__name__,
+            last_error_message=str(exc)[:2000],
+        )
+        await session.commit()
+        log.warning("membership sync failed for %s: %s", platform, exc)
+        return {"platform": platform, "ok": False, "error": type(exc).__name__}
+
+    for m in memberships:
+        await touch_membership(
+            session,
+            platform=platform,
+            external_campaign_id=m.campaign_id,
+            display_name=m.display_name,
+            url=m.url,
+            status=m.status,
+            tier_names=m.tier_names or None,
+            amount_cents=m.amount_cents,
+            currency=m.currency,
+            details={**(m.details or {}), "is_free_member": m.is_free_member},
+        )
+    await _record_sync(
+        session, platform,
+        last_success_at=now,
+        last_count=len(memberships),
+        # Cleared on success — a stale error beside a fresh success would read
+        # as "still broken" forever.
+        last_error_type=None,
+        last_error_message=None,
+    )
+    await session.commit()
+    log.info("membership sync ok for %s: %d membership(s)", platform, len(memberships))
+    return {"platform": platform, "ok": True, "count": len(memberships)}

backend/app/tasks/maintenance.py

@@ -1204,3 +1204,111 @@ def match_post_associations() -> str:
     if not res["enabled"]:
         return "disabled"
     return f"scanned={res['scanned']} proposed={res['proposed']}"
+
+
+# The wall-clock budget for ONE platform's roster walk. Rule 156 distinguishes
+# this from the per-REQUEST timeout the client already has: a paginated roster
+# behind an endpoint that answers every page slowly-but-within-timeout would
+# never trip that one, and would sit on a worker indefinitely. This is the wait
+# that bounds the whole walk.
+MEMBERSHIP_SYNC_BUDGET_SECONDS = 240.0
+
+
+@celery.task(
+    name="backend.app.tasks.maintenance.sync_memberships",
+    soft_time_limit=900, time_limit=1200,
+)
+def sync_memberships() -> str:
+    """Milestone 387 C3: walk each platform's membership roster into the DB.
+
+    Daily, because memberships change on a BILLING cycle rather than a download
+    cadence — polling an account-scoped endpoint more often would be both
+    pointless and less polite than the browser.
+
+    Rule 89's four, and where each actually lives:
+      * recovery — `touch_membership` is an upsert and never deletes, so
+        recovery is simply the next run; a dead run leaves the previous roster
+        intact rather than a half-written one (`sync_platform` completes the
+        fetch before writing anything).
+      * retention — per C1, rows age out and are never deleted on
+        disappearance, because disappearing IS the signal C4 reads.
+      * wall-clock timeout — MEMBERSHIP_SYNC_BUDGET_SECONDS per platform,
+        plus the task's own soft/hard limits.
+      * duration tracking — the TaskRun celery-signal plumbing, same as every
+        other sweep here.
+
+    Rule 164: this is the rule's own named exception — a genuinely external
+    feature that may call out. It never gates startup, and a failure leaves a
+    VISIBLE stale state (membership_sync) rather than an empty roster that
+    reads as "you subscribe to nothing".
+    """
+    import asyncio
+
+    from ..services.credential_crypto import CredentialCrypto
+    from ..services.credential_service import CredentialService
+    from ..services.membership_roster import sync_platform
+    from ..services.patreon_client import PatreonClient
+    from ._async_session import async_session_factory
+
+    key_path = IMAGES_ROOT / "secrets" / "credential_key.b64"
+
+    # platform -> how to build a client from a cookies path. A platform is in
+    # the sweep only if it is here AND its client exposes `iter_memberships`
+    # AND a credential exists — three independent gates, each silent, so
+    # adding SubscribeStar (D1) is one line here and nothing else.
+    builders = {"patreon": PatreonClient}
+
+    async def _run() -> dict:
+        async_factory, engine = async_session_factory()
+        results = []
+        try:
+            for platform, build in builders.items():
+                async with async_factory() as session:
+                    cred = CredentialService(session, CredentialCrypto(key_path))
+                    cookies = await cred.get_cookies_path(platform)
+                if cookies is None:
+                    # No credential is not an error — the operator simply has
+                    # not connected this platform. Recording a failure here
+                    # would light up the UI for a feature they never enabled.
+                    results.append({"platform": platform, "skipped": "no credential"})
+                    continue
+
+                client = build(str(cookies))
+                if getattr(client, "iter_memberships", None) is None:
+                    # The seam, probed not required (#387 C2). A client without
+                    # it makes the feature invisible for that platform — no
+                    # flag, no config row, no "unsupported" branch.
+                    results.append({"platform": platform, "skipped": "no seam"})
+                    continue
+
+                async def fetch(_client=client):
+                    # The client is sync (`requests`); run it off the loop so a
+                    # slow roster does not block the event loop, and bound the
+                    # whole walk rather than only its individual requests.
+                    def _walk():
+                        user_id = _client.current_user_id()
+                        return list(_client.iter_memberships(user_id))
+
+                    return await asyncio.wait_for(
+                        asyncio.to_thread(_walk),
+                        timeout=MEMBERSHIP_SYNC_BUDGET_SECONDS,
+                    )
+
+                async with async_factory() as session:
+                    results.append(
+                        await sync_platform(session, platform=platform, fetch=fetch)
+                    )
+            return {"results": results}
+        finally:
+            await engine.dispose()
+
+    res = asyncio.run(_run())
+    parts = []
+    for r in res["results"]:
+        if "skipped" in r:
+            parts.append(f"{r['platform']}=skipped({r['skipped']})")
+        elif r.get("ok"):
+            parts.append(f"{r['platform']}={r['count']}")
+        else:
+            parts.append(f"{r['platform']}=FAILED({r['error']})")
+    return " ".join(parts) or "no platforms"