feat(host_agent): each drive's health, read with smartctl, alertable (#5789)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m24s
CI / unit (push) Successful in 1m32s
CI / publish (push) Successful in 1m10s

The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.

install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 09:17:50 -04:00
co-authored by Claude Opus 5.5
parent fdc5c07183
commit 6c4ef368f9
20 changed files with 848 additions and 11 deletions
+23
View File
@@ -266,6 +266,29 @@ The figures are recorded at most once every 30 s, whichever manager sent them, s
A maintenance job on a swarm node opens a second window over `docker` / `swarm` beside its host's (`maintenance._swarm_window`). An alert the drain or reboot causes then waits for the job, and one still firing when the window ends is sent then.
### Drive health (1.20.0, #5789)
Reading SMART needs root, so the root helper does it, as the one action on its fixed list that only reads: `drives`. Hourly, and at start, the agent leaves a `drives` request. The helper runs `smartctl --scan-open` to find the drives itself (the request names none), reads each with `smartctl --json -a -d <type> <dev>`, and writes a summary to `/var/lib/steward-maint/drives.json` (0644). It writes no job result, because a drive read is not a maintenance job. The agent's newest copy rides every sample, as updates do.
smartctl ≥ 7.0 answers in JSON, the same on every distribution and for SATA, SAS and NVMe drives. A missing or older smartctl is reported (`smartctl not installed`, `… too old`) and its text output is never parsed. The helper reports figures only:
- `passed` and `temp_c` for every drive;
- `critical_warning`, `life_used_pct`, `media_errors` and spare % against threshold for NVMe;
- reallocated / pending / offline-uncorrectable sectors for ATA, and life used where the drive reports it;
- grown defects for SCSI.
A drive with no SMART (a virtual disk, a bridge that hides it) reads `available: false`, and one smartctl couldn't open carries `error`.
Steward judges the figures in `drives.py`, so the judgment can change without an agent update:
- **failing**: the drive's self-assessment fails, an NVMe critical warning is raised, or spare blocks fall under threshold;
- **worn**: at least 90% of its rated life is used;
- **watch**: it has bad sectors or media errors but otherwise passes. Old disks carry a few for years, so a count alone isn't a failure.
The figures become metrics on `<host>:drive:<device>`: `drive_failing`, `drive_life_used_pct`, `drive_bad_sectors` and `drive_temp_c`. The host gets `drives_failing` and `drive_life_used_max_pct` for covering rules; they are recorded only when at least one drive was read, so a VM's virtual disks don't claim "0 failing". The registration keeps the newest report (`drive_health`), and the host page lists it under **Drive health**, what needs attention first.
**Installing smartctl on any common distribution.** `files/install-deps.sh` installs smartmontools with whichever package manager the host has: apt (without recommends, which would pull in a mail server), dnf, yum, zypper, pacman or apk. install.sh and the install / provision / update playbooks all fetch and run that one copy. A package that can't be installed is reported and never fails the install.
### What happened to the host (1.14.0, #5756)
A post may carry `events`, said once each, with the next post that gets through: