#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the concurrency-1 maintenance_long lane for hours — starving normalize_tags, re-extract, audits, and the new series rescan (which is why #740 "never applied"). Three fixes: - _run_bounded: Popen + bounded post-kill reap; if the child is unkillable (D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The orphan is reaped by the OS once its syscall clears. - backup_db dumps to a LOCAL temp file then moves the finished .sql to the (NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local writes (killable) instead of an NFS write that hangs. backup_images keeps bounded-kill (too big to stage locally). - recover_stalled_backup_runs: split the stall window — db 40 min (was sharing images' 7h), so a hung DB backup is flipped to error promptly. #740 — Standardize tag casing showed "0 groups to change" the instant it was clicked: onNormCommit overwrote the preview with zeros. Keep the real preview visible and disable the button while queued; backend apply was already correct. Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp target; per-kind stall sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -83,8 +83,13 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
|
||||
# small buffer) so the sweep never flags in-flight work.
|
||||
#
|
||||
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
|
||||
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
|
||||
# with a 30-min buffer.
|
||||
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
|
||||
# DB backup/restore is seconds-to-minutes (35-min hard limit). It must NOT share
|
||||
# the images' 7h window — a DB backup wedged on NFS would otherwise sit "running"
|
||||
# for 7 hours holding the concurrency-1 maintenance_long lane (operator-flagged
|
||||
# 2026-06-07). 40 min gives a small buffer over the hard limit.
|
||||
BACKUP_DB_STALL_THRESHOLD_MINUTES = 40
|
||||
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
|
||||
# 2h15m gives a 10-min buffer.
|
||||
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
|
||||
@@ -619,16 +624,20 @@ def recover_stalled_backup_runs() -> int:
|
||||
"""
|
||||
SessionLocal = _sync_session_factory()
|
||||
now = datetime.now(UTC)
|
||||
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
|
||||
msg = (
|
||||
f"stranded by recovery sweep (no terminal status after "
|
||||
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
|
||||
)
|
||||
db_cutoff = now - timedelta(minutes=BACKUP_DB_STALL_THRESHOLD_MINUTES)
|
||||
slow_cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
|
||||
msg = "stranded by recovery sweep (no terminal status within the stall window)"
|
||||
with SessionLocal() as session:
|
||||
result = session.execute(
|
||||
update(BackupRun)
|
||||
.where(BackupRun.status.in_(["running", "restoring"]))
|
||||
.where(BackupRun.started_at < cutoff)
|
||||
# db backups/restores are fast (40-min window); images run hours (7h).
|
||||
.where(
|
||||
or_(
|
||||
and_(BackupRun.kind == "db", BackupRun.started_at < db_cutoff),
|
||||
and_(BackupRun.kind != "db", BackupRun.started_at < slow_cutoff),
|
||||
)
|
||||
)
|
||||
.values(status="error", finished_at=now, error=msg)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
Reference in New Issue
Block a user