bvandeusen
914033db29
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the periodic prune/backfill/recovery tasks churn dead tuples faster than autovacuum always keeps up — so explicit maintenance earns its keep here. - tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn). Scheduled weekly via Beat; also operator-triggerable. - _sync_engine.get_sync_engine(): expose the process engine for the autocommit connection. - GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct + last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a black box. - POST /api/admin/maintenance/vacuum: enqueue the task on demand. Tests: vacuum task runs + reports tables; db-stats shape; trigger queues. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
44 lines
1.4 KiB
Python
44 lines
1.4 KiB
Python
"""Process-level sync engine for Celery task modules.
|
|
|
|
Each task module used to call ``create_engine(...)`` on every invocation,
|
|
which leaked engines (and Postgres connections) — high-fire-rate tasks
|
|
like ``import_media_file`` would exhaust ``max_connections`` within
|
|
minutes during a bulk migration.
|
|
|
|
This module owns one engine per process. Celery prefork forks before any
|
|
task runs, so each worker process lazily initializes its own engine on
|
|
the first task and reuses it for the rest of its life.
|
|
"""
|
|
|
|
from sqlalchemy import create_engine
|
|
from sqlalchemy.orm import sessionmaker
|
|
|
|
from ..config import get_config
|
|
|
|
_ENGINE = None
|
|
_SESSIONMAKER = None
|
|
|
|
|
|
def sync_session_factory():
|
|
"""Return a process-wide ``sessionmaker`` bound to a single engine."""
|
|
global _ENGINE, _SESSIONMAKER
|
|
if _SESSIONMAKER is None:
|
|
cfg = get_config()
|
|
_ENGINE = create_engine(
|
|
cfg.database_url_sync,
|
|
future=True,
|
|
pool_pre_ping=True,
|
|
pool_size=5,
|
|
max_overflow=5,
|
|
pool_recycle=300,
|
|
)
|
|
_SESSIONMAKER = sessionmaker(_ENGINE, expire_on_commit=False)
|
|
return _SESSIONMAKER
|
|
|
|
|
|
def get_sync_engine():
|
|
"""The process-wide sync Engine — for raw work that needs a connection
|
|
directly (e.g. AUTOCOMMIT VACUUM, which can't run inside a transaction)."""
|
|
sync_session_factory() # ensure _ENGINE is initialized
|
|
return _ENGINE
|