fix(host_agent): a stalled NFS mount no longer freezes the agent (#5760)

#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.

Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

plugins/host_agent/routes.py

@@ -143,7 +143,13 @@ def _expand_sample_to_metrics(
     # inodes, every file create fails with space to spare, so the disk rules
     # and the overview, which read disk_used_pct_worst, see either.
     worst_pct = 0.0
+    stalled = 0
     for disk in sample.get("storage") or []:
+        # A mount that didn't answer in time has no figures (#5760); how many
+        # did is its own metric, so a rule can say "an NFS share stopped".
+        if disk.get("stalled"):
+            stalled += 1
+            continue
         total = disk.get("total_bytes", 0)
         used = disk.get("used_bytes", 0)
         pct = 100.0 * used / total if total else 0.0
@@ -158,7 +164,10 @@ def _expand_sample_to_metrics(
         if pct > worst_pct:
             worst_pct = pct
     if sample.get("storage"):
-        row("disk_used_pct_worst", host_name, worst_pct)
+        # Every mount stalled says nothing about how full they are.
+        if stalled < len(sample["storage"]):
+            row("disk_used_pct_worst", host_name, worst_pct)
+        row("mounts_stalled", host_name, stalled)
 
     # Per-core CPU. Sub-resources carry a ':' so the fleet widget filters them out.
     for i, core_pct in enumerate(sample.get("cpu_cores") or []):