fix(backup,tags): unwedge backups on NFS (#739) + tag-standardize "0 groups" (#740)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m10s

#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
  (D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
  orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
  (NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
  writes (killable) instead of an NFS write that hangs. backup_images keeps
  bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
  images' 7h), so a hung DB backup is flipped to error promptly.

#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.

Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 19:20:16 -04:00
parent 19a91a1641
commit daaa7543a8
5 changed files with 188 additions and 27 deletions
+16 -7
View File
@@ -83,8 +83,13 @@ TASK_RUN_KEEP_FAILURE_SECONDS = 7 * 24 * 3600 # 7 days
# small buffer) so the sweep never flags in-flight work.
#
# Backups: images backup has time_limit=23400s (6.5h). 7h covers it
# with a 30-min buffer; db backup at 12 min hard limit fits trivially.
# with a 30-min buffer.
BACKUP_STALL_THRESHOLD_MINUTES = 7 * 60
# DB backup/restore is seconds-to-minutes (35-min hard limit). It must NOT share
# the images' 7h window — a DB backup wedged on NFS would otherwise sit "running"
# for 7 hours holding the concurrency-1 maintenance_long lane (operator-flagged
# 2026-06-07). 40 min gives a small buffer over the hard limit.
BACKUP_DB_STALL_THRESHOLD_MINUTES = 40
# Library audit: scan_library_for_rule has time_limit=7500s (2h5m).
# 2h15m gives a 10-min buffer.
LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES = 135
@@ -619,16 +624,20 @@ def recover_stalled_backup_runs() -> int:
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"{BACKUP_STALL_THRESHOLD_MINUTES // 60}h)"
)
db_cutoff = now - timedelta(minutes=BACKUP_DB_STALL_THRESHOLD_MINUTES)
slow_cutoff = now - timedelta(minutes=BACKUP_STALL_THRESHOLD_MINUTES)
msg = "stranded by recovery sweep (no terminal status within the stall window)"
with SessionLocal() as session:
result = session.execute(
update(BackupRun)
.where(BackupRun.status.in_(["running", "restoring"]))
.where(BackupRun.started_at < cutoff)
# db backups/restores are fast (40-min window); images run hours (7h).
.where(
or_(
and_(BackupRun.kind == "db", BackupRun.started_at < db_cutoff),
and_(BackupRun.kind != "db", BackupRun.started_at < slow_cutoff),
)
)
.values(status="error", finished_at=now, error=msg)
)
session.commit()