feat: the membership sweep, and the state that makes its failures readable (387 C3)
CI / lint (push) Failing after 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m35s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m55s
CI / lint (push) Failing after 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m35s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m55s
A daily sweep that walks each platform's roster into `platform_membership`. Daily because memberships change on a BILLING cycle, not a download cadence. **`membership_sync` is the part that earns its keep.** Without it three very different situations are one indistinguishable state — the account subscribes to nothing, the sweep never ran, the sweep failed — and all three leave zero rows in `platform_membership`. "You are tracking 12 sources you no longer subscribe to" is correct in the first case and an invitation to cancel things the operator is actively paying for in the other two. So C4 gates its CONCLUSIONS on `last_success_at`, not merely its display, and `roster_is_fresh` is computed server-side so no caller can forget to. Two timestamps rather than one: `last_attempt_at` moves every run, `last_success_at` only on a clean walk. The gap between them is the signal — a sweep hammering a broken credential every day must not look healthy because it ran recently, and there is a test for exactly that. Rejected shortcuts, both tempting: `MAX(platform_membership.last_seen_at)` cannot tell "synced fine, found nothing" from "never synced"; `task_run` is worse, since its retention prunes ok rows after 24h and a sweep that last succeeded three days ago would leave no trace at all. **The fetch completes before anything is written.** That ordering is the safety property: a walk that dies mid-pagination writes nothing, so a failure can never leave a roster half this week's and half last week's. `touch_membership` never deletes, so a failure cannot empty the roster either — but "intact" should mean intact, not merely non-empty. Rule 89's four, each where it actually lives: recovery is "run it again" (upsert, no deletes); retention is C1's age-out-never-delete, because disappearing IS the signal; the wall-clock deadline is per-platform and distinct from the per-REQUEST timeout the client already has (rule 156 — a paginated roster answering every page slowly-but-within-timeout would never trip that one and would sit on a worker indefinitely); duration comes from the existing TaskRun signal plumbing. **A bug caught in review, not production:** the broad `except Exception` would have swallowed Celery's SoftTimeLimitExceeded — which is an ORDINARY Exception subclass, not a BaseException — letting the sweep run past the soft limit into the hard one, where it is SIGKILLed mid-transaction. A sweep that cannot be stopped is worse than one that fails. Now re-raised explicitly, with a test that also asserts SoftTimeLimitExceeded is still an Exception, so the re-raise cannot quietly become dead code. Rule 164 is why this ships with UI rather than backend-only: a roster that never synced must be VISIBLE as such. The card says "never synced" in words and states no count at all — rendering it as 0 is the precise conflation the whole step exists to prevent — while a real zero behind a real sync is reported as zero, because that one IS an answer. Pinned in both directions. Three independent gates decide whether a platform is swept — registered here, client exposes `iter_memberships`, credential exists — each silent, so adding SubscribeStar (D1) is one line and nothing else. A missing credential is not an error: recording a failure would light up the UI for a feature never enabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9
This commit is contained in:
@@ -1204,3 +1204,111 @@ def match_post_associations() -> str:
|
||||
if not res["enabled"]:
|
||||
return "disabled"
|
||||
return f"scanned={res['scanned']} proposed={res['proposed']}"
|
||||
|
||||
|
||||
# The wall-clock budget for ONE platform's roster walk. Rule 156 distinguishes
|
||||
# this from the per-REQUEST timeout the client already has: a paginated roster
|
||||
# behind an endpoint that answers every page slowly-but-within-timeout would
|
||||
# never trip that one, and would sit on a worker indefinitely. This is the wait
|
||||
# that bounds the whole walk.
|
||||
MEMBERSHIP_SYNC_BUDGET_SECONDS = 240.0
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.maintenance.sync_memberships",
|
||||
soft_time_limit=900, time_limit=1200,
|
||||
)
|
||||
def sync_memberships() -> str:
|
||||
"""Milestone 387 C3: walk each platform's membership roster into the DB.
|
||||
|
||||
Daily, because memberships change on a BILLING cycle rather than a download
|
||||
cadence — polling an account-scoped endpoint more often would be both
|
||||
pointless and less polite than the browser.
|
||||
|
||||
Rule 89's four, and where each actually lives:
|
||||
* recovery — `touch_membership` is an upsert and never deletes, so
|
||||
recovery is simply the next run; a dead run leaves the previous roster
|
||||
intact rather than a half-written one (`sync_platform` completes the
|
||||
fetch before writing anything).
|
||||
* retention — per C1, rows age out and are never deleted on
|
||||
disappearance, because disappearing IS the signal C4 reads.
|
||||
* wall-clock timeout — MEMBERSHIP_SYNC_BUDGET_SECONDS per platform,
|
||||
plus the task's own soft/hard limits.
|
||||
* duration tracking — the TaskRun celery-signal plumbing, same as every
|
||||
other sweep here.
|
||||
|
||||
Rule 164: this is the rule's own named exception — a genuinely external
|
||||
feature that may call out. It never gates startup, and a failure leaves a
|
||||
VISIBLE stale state (membership_sync) rather than an empty roster that
|
||||
reads as "you subscribe to nothing".
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from ..services.credential_crypto import CredentialCrypto
|
||||
from ..services.credential_service import CredentialService
|
||||
from ..services.membership_roster import sync_platform
|
||||
from ..services.patreon_client import PatreonClient
|
||||
from ._async_session import async_session_factory
|
||||
|
||||
key_path = IMAGES_ROOT / "secrets" / "credential_key.b64"
|
||||
|
||||
# platform -> how to build a client from a cookies path. A platform is in
|
||||
# the sweep only if it is here AND its client exposes `iter_memberships`
|
||||
# AND a credential exists — three independent gates, each silent, so
|
||||
# adding SubscribeStar (D1) is one line here and nothing else.
|
||||
builders = {"patreon": PatreonClient}
|
||||
|
||||
async def _run() -> dict:
|
||||
async_factory, engine = async_session_factory()
|
||||
results = []
|
||||
try:
|
||||
for platform, build in builders.items():
|
||||
async with async_factory() as session:
|
||||
cred = CredentialService(session, CredentialCrypto(key_path))
|
||||
cookies = await cred.get_cookies_path(platform)
|
||||
if cookies is None:
|
||||
# No credential is not an error — the operator simply has
|
||||
# not connected this platform. Recording a failure here
|
||||
# would light up the UI for a feature they never enabled.
|
||||
results.append({"platform": platform, "skipped": "no credential"})
|
||||
continue
|
||||
|
||||
client = build(str(cookies))
|
||||
if getattr(client, "iter_memberships", None) is None:
|
||||
# The seam, probed not required (#387 C2). A client without
|
||||
# it makes the feature invisible for that platform — no
|
||||
# flag, no config row, no "unsupported" branch.
|
||||
results.append({"platform": platform, "skipped": "no seam"})
|
||||
continue
|
||||
|
||||
async def fetch(_client=client):
|
||||
# The client is sync (`requests`); run it off the loop so a
|
||||
# slow roster does not block the event loop, and bound the
|
||||
# whole walk rather than only its individual requests.
|
||||
def _walk():
|
||||
user_id = _client.current_user_id()
|
||||
return list(_client.iter_memberships(user_id))
|
||||
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(_walk),
|
||||
timeout=MEMBERSHIP_SYNC_BUDGET_SECONDS,
|
||||
)
|
||||
|
||||
async with async_factory() as session:
|
||||
results.append(
|
||||
await sync_platform(session, platform=platform, fetch=fetch)
|
||||
)
|
||||
return {"results": results}
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
res = asyncio.run(_run())
|
||||
parts = []
|
||||
for r in res["results"]:
|
||||
if "skipped" in r:
|
||||
parts.append(f"{r['platform']}=skipped({r['skipped']})")
|
||||
elif r.get("ok"):
|
||||
parts.append(f"{r['platform']}={r['count']}")
|
||||
else:
|
||||
parts.append(f"{r['platform']}=FAILED({r['error']})")
|
||||
return " ".join(parts) or "no platforms"
|
||||
|
||||
Reference in New Issue
Block a user