fix(download): release DB connections across the gallery-dl subprocess
Backfill events were STILL stranding empty after the timeout-ladder fix.
Worker logs showed the salvage path working ("Download timeout for
anduo/patreon after 1170.0s (18 files written)") but then:
Retry in 3s: DBAPIError(ConnectionDoesNotExistError: connection was
closed in the middle of operation)
...succeeded in 0.149s <- in-flight guard no-op
Root cause: DownloadService held the async + sync DB connections checked
out across the entire (≤19.5-min backfill) gallery-dl subprocess. The
server reaps the idle connection, so phase 3's first query hits a dead
socket. That DBAPIError trips download_source's autoretry_for, the retry
re-enters _phase1_setup, sees the event still 'running', returns
in_flight and no-ops — leaving the event to be stranded empty by the
recovery sweep. pool_pre_ping was already on both engines but can't help
a *held* connection (it only validates on pool checkout).
Fix:
- DownloadService.download_source closes the async + sync sessions after
phase 1, before the subprocess, so phase 3 re-acquires a live
connection (matches the class's "Phase 2 — no DB connection" docstring).
- The per-task async engine switches to NullPool so phase 3 always opens
a fresh connection rather than a pooled one the server may have reaped.
Tests: assert connections are released before gdl.download runs and the
event still finalizes; assert the task engine uses NullPool. Also fixes a
stale 1800s->1170s comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -91,10 +91,25 @@ class DownloadService:
|
||||
return setup["event_id"]
|
||||
ctx = setup
|
||||
|
||||
# Release the phase-1 DB connections before the (up to ~19.5-min in
|
||||
# backfill) gallery-dl subprocess. Held checked-out across that idle
|
||||
# window, the asyncpg/psycopg connections get reaped by the server,
|
||||
# and phase 3's first query then hits a dead socket
|
||||
# (asyncpg ConnectionDoesNotExistError) → download_source autoretry →
|
||||
# _phase1_setup's in-flight guard no-ops the retry → the event
|
||||
# strands empty for the recovery sweep (Anduo #40014, 2026-06-04).
|
||||
# pool_pre_ping can't help a *held* connection — it only validates on
|
||||
# pool checkout. Closing returns them to the pool so phase 3 re-
|
||||
# acquires a live one (the async task engine uses NullPool, the sync
|
||||
# engine pre_ping + pool_recycle=300). This is what makes the
|
||||
# "Phase 2 — no DB connection" contract in the class docstring true.
|
||||
await self.async_session.close()
|
||||
self.sync_session.close()
|
||||
|
||||
source_config = SourceConfig.from_dict(ctx["config_overrides"] or {})
|
||||
# alembic 0031 / plan #544: derive skip_value + timeout from the
|
||||
# source's backfill_runs_remaining counter. When > 0, walk the full
|
||||
# post history (skip: True + 1800s); when 0, exit gallery-dl after
|
||||
# post history (skip: True + 1170s); when 0, exit gallery-dl after
|
||||
# 20 contiguous archived items (skip: "exit:20" + the default
|
||||
# 870s). Operator sets backfill via POST /api/sources/{id}/backfill.
|
||||
backfill_remaining = ctx.get("backfill_runs_remaining", 0) or 0
|
||||
|
||||
@@ -10,6 +10,7 @@ disposes it (``await engine.dispose()``) when its loop ends.
|
||||
"""
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
from sqlalchemy.pool import NullPool
|
||||
|
||||
from ..config import get_config
|
||||
|
||||
@@ -17,5 +18,13 @@ from ..config import get_config
|
||||
def async_session_factory():
|
||||
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
|
||||
cfg = get_config()
|
||||
engine = create_async_engine(cfg.database_url, future=True, pool_pre_ping=True)
|
||||
# NullPool: this engine lives for ONE task (created + disposed per
|
||||
# asyncio.run loop), so intra-task connection pooling buys nothing and
|
||||
# actively bit us — download_source releases its phase-1 connection
|
||||
# before a multi-minute gallery-dl subprocess, and a *pooled* idle
|
||||
# connection would be reaped by the server and handed back dead to
|
||||
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
|
||||
# opens a fresh real connection on each checkout, so phase 3 always
|
||||
# reconnects clean; pre_ping is then redundant.
|
||||
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
|
||||
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
|
||||
|
||||
Reference in New Issue
Block a user