fix(infra): bump Postgres shm_size (vacuum DiskFull) + raise DB-backup time limit
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Failing after 12s
CI / frontend-build (push) Successful in 24s
CI / integration (push) Failing after 2m23s

Two more maintenance-queue failures from the operator's 24h list:
- vacuum_analyze died with "could not resize shared memory segment to 67MB: No
  space left on device" — Docker's default /dev/shm is 64MB, too small for
  VACUUM (ANALYZE)'s parallel-worker shared memory. Set the postgres service
  shm_size: 512m.
- backup_db_task timed out at its 12-min limit once the DB grew; a pg_dump can't
  be chunked, so raise it to 30/35 min. (A long backup still briefly holds the
  concurrency-1 lane — the structural fix is a dedicated lane for long one-shots.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 00:08:19 -04:00
parent f2e9ae07dc
commit a183be7e6e
2 changed files with 10 additions and 1 deletions
+5 -1
View File
@@ -41,7 +41,11 @@ def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
bind=True,
autoretry_for=(OperationalError, DBAPIError),
retry_backoff=10, retry_backoff_max=120, max_retries=2,
soft_time_limit=600, time_limit=720,
# A pg_dump can't be chunked; the 12-min limit timed out once the DB grew
# (operator-flagged 2026-06-07). 30/35 min gives real headroom. (A long
# backup still briefly holds the concurrency-1 maintenance lane — the
# structural fix is a dedicated lane for the long one-shots.)
soft_time_limit=1800, time_limit=2100,
)
def backup_db_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
+5
View File
@@ -21,6 +21,11 @@ services:
postgres:
image: pgvector/pgvector:pg16
# Docker's default /dev/shm is 64MB; VACUUM (ANALYZE) and parallel queries
# allocate larger shared-memory segments and fail with
# "could not resize shared memory segment ... No space left on device"
# (operator-flagged 2026-06-07, vacuum_analyze on import_task needed 67MB).
shm_size: 512m
environment:
POSTGRES_USER: ${DB_USER:-fabledcurator}
POSTGRES_PASSWORD: ${DB_PASSWORD:-fabledcurator_dev}