fix: a restart no longer strands downloads — the download lane closes its predecessor's runs as interrupted and frees the platform locks at boot (#4433)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 3s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Failing after 31s
CI and images / integration (push) Successful in 2m23s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

A walk that outlives the 90s stop grace is SIGKILLed: its event never
finalizes and its platform lock is held for the 27-min TTL. The 30-min
sweep then errored every stranded event and bumped consecutive_failures,
backing sources off (and blocking backfills) as if the platform failed.

On worker_ready, the process consuming 'download' ends pre-boot
pending/running events as skipped with error_type 'interrupted', leaves
the source untouched so the next tick resumes it, and releases the
serialized platforms' locks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-25 10:29:05 -04:00
co-authored by Claude Opus 5.5
parent 32874ca678
commit 0fad744bfb
5 changed files with 232 additions and 2 deletions
+19
View File
@@ -60,3 +60,22 @@ def platform_lock(platform: str, *, ttl_seconds: int):
except redis.RedisError as exc: # pragma: no cover - broker outage
log.warning("platform_lock unavailable for %s: %s", platform, exc)
return None
def release_all_platform_locks() -> int:
"""Drop every serialized platform's lock. Returns how many were held.
Only for the download lane's boot (#4433). A worker that restarts mid-walk
is SIGKILLed past its stop grace, so its `finally` never releases the lock,
and the TTL keeps every other source on that platform bouncing for up to
27 minutes after the new worker is ready. At boot no walk of ours can be
running, so a held lock names a dead one. Assumes one download consumer —
the only shape FC deploys; a second replica booting would free a live
walk's lock (not corrupt it: the walk runs on, a second walk may overlap it).
"""
try:
client = _redis()
return int(client.delete(*(f"{_LOCK_PREFIX}{p}" for p in SERIALIZED_PLATFORMS)))
except redis.RedisError as exc: # pragma: no cover - broker outage
log.warning("could not release platform locks at boot: %s", exc)
return 0