compose(db): lenient healthcheck + stop_grace_period to survive host stalls
Mitigation for the nightly fabledscribe Postgres outage on the vdnt-docker02 Swarm node (incidents 2026-05-15/16/17 around 03:50 UTC). Confirmed kill chain (not the trigger): a brief host-level setns/exec stall makes the Docker healthcheck exec fail with exit 1 → unhealthy → SIGKILL → fast-shutdown can't finish on NFS in 10s → exit 137 → swarm restart_policy.max_attempts: 5 burns out → DB stays dead. Hardens the `db` service so a transient host blip can't escalate to killing the database: - stop_grace_period: 120s (gives PG room to fsync on shutdown) - healthcheck: interval 30s / timeout 10s / retries 10 / start_period 180s (only gates app startup order — not authoritative liveness) - prod: restart_policy condition=on-failure, max_attempts=0, window=120s - quickstart/dev: restart: unless-stopped Host-side trigger (what stalls runc/exec at ~03:50 UTC) is still under investigation — see project_pg_nightly_outage.md. Note: the Portainer prod stack differs from docker-compose.prod.yml here (NFS bind, traefik labels, no ollama). The same `db:` block needs to be pasted into Portainer for the prod mitigation to apply. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,17 +39,21 @@ services:
|
||||
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
stop_grace_period: 120s
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
environment:
|
||||
POSTGRES_USER: fabled
|
||||
POSTGRES_PASSWORD: fabled
|
||||
POSTGRES_DB: fabledassistant
|
||||
# Lenient by design: a transient host exec/healthcheck stall must never
|
||||
# escalate to killing the DB. Health here only gates app startup order.
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U fabled"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 10
|
||||
start_period: 180s
|
||||
restart: unless-stopped
|
||||
|
||||
ollama:
|
||||
|
||||
Reference in New Issue
Block a user