fix(backup,tags): unwedge backups on NFS (#739) + tag-standardize "0 groups" (#740)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m10s

#739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min
subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the
concurrency-1 maintenance_long lane for hours — starving normalize_tags,
re-extract, audits, and the new series rescan (which is why #740 "never
applied"). Three fixes:
- _run_bounded: Popen + bounded post-kill reap; if the child is unkillable
  (D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The
  orphan is reaped by the OS once its syscall clears.
- backup_db dumps to a LOCAL temp file then moves the finished .sql to the
  (NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local
  writes (killable) instead of an NFS write that hangs. backup_images keeps
  bounded-kill (too big to stage locally).
- recover_stalled_backup_runs: split the stall window — db 40 min (was sharing
  images' 7h), so a hung DB backup is flipped to error promptly.

#740 — Standardize tag casing showed "0 groups to change" the instant it was
clicked: onNormCommit overwrote the preview with zeros. Keep the real preview
visible and disable the button while queued; backend apply was already correct.

Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp
target; per-kind stall sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 19:20:16 -04:00
parent 19a91a1641
commit daaa7543a8
5 changed files with 188 additions and 27 deletions
+50 -2
View File
@@ -34,16 +34,32 @@ def fake_subprocess_and_images_root(monkeypatch, tmp_path):
stdout = b""
stderr = b""
def _fake_run(cmd, **kwargs):
def _sentinel(cmd):
if cmd[0] == "pg_dump":
i = cmd.index("-f")
Path(cmd[i + 1]).write_bytes(b"-- fake pg_dump\n")
elif cmd[0] == "tar" and "-cf" in cmd:
i = cmd.index("-cf")
Path(cmd[i + 1]).write_bytes(b"fake tar payload")
def _fake_run(cmd, **kwargs):
_sentinel(cmd)
return _FakeProc()
class _FakePopen:
def __init__(self, cmd, **kwargs):
self.returncode = 0
_sentinel(cmd)
def communicate(self, timeout=None):
return (b"", b"")
def kill(self):
self.returncode = -9
# backup_db/backup_images go through _run_bounded (Popen); restore via run.
monkeypatch.setattr("subprocess.run", _fake_run)
monkeypatch.setattr("subprocess.Popen", _FakePopen)
def _seed_backup(db_sync, *, kind, status, started_at, tag=None,
@@ -89,7 +105,8 @@ async def test_backup_db_task_records_failure_on_subprocess_error(db_sync, monke
def _boom(*a, **kw):
raise RuntimeError("synthetic pg_dump fail")
monkeypatch.setattr("subprocess.run", _boom)
# backup_db dumps via _run_bounded → subprocess.Popen.
monkeypatch.setattr("subprocess.Popen", _boom)
with pytest.raises(RuntimeError):
backup_db_task.delay().get()
@@ -237,6 +254,37 @@ def test_prune_backups_never_deletes_running_or_restoring(db_sync):
assert set(statuses) == {"running", "restoring"}
# --- recover_stalled_backup_runs (per-kind threshold, FC #739) -------
def test_stall_sweep_flips_db_fast_but_spares_running_images(db_sync):
from backend.app.tasks.maintenance import recover_stalled_backup_runs
now = datetime.now(UTC)
# A db backup stuck 50 min → past the 40-min db window → flipped to error.
db_id = _seed_backup(
db_sync, kind="db", status="running",
started_at=now - timedelta(minutes=50),
)
# An images backup running 50 min is still well under the 7h window → spared.
img_id = _seed_backup(
db_sync, kind="images", status="running",
started_at=now - timedelta(minutes=50),
)
db_sync.commit()
recover_stalled_backup_runs.apply().get()
db_status = db_sync.execute(
select(BackupRun.status).where(BackupRun.id == db_id)
).scalar_one()
img_status = db_sync.execute(
select(BackupRun.status).where(BackupRun.id == img_id)
).scalar_one()
assert db_status == "error"
assert img_status == "running"
# --- backup_db_nightly ----------------------------------------------