feat(host_agent): each drive's health, read with smartctl, alertable (#5789)
The root helper gains one read-only action, `drives`: it finds the drives itself (smartctl --scan-open), reads each as JSON, and writes a summary the agent carries on every sample (agent 1.20.0). Steward judges the figures in drives.py (failing / worn / watch / ok / no SMART / unreadable), records them per drive and as host roll-ups for covering rules, keeps the newest report on the registration and lists it on the host page. install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or apk, run by install.sh and every agent playbook; one it can't install is reported, never fatal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -266,6 +266,29 @@ The figures are recorded at most once every 30 s, whichever manager sent them, s
|
||||
|
||||
A maintenance job on a swarm node opens a second window over `docker` / `swarm` beside its host's (`maintenance._swarm_window`). An alert the drain or reboot causes then waits for the job, and one still firing when the window ends is sent then.
|
||||
|
||||
### Drive health (1.20.0, #5789)
|
||||
|
||||
Reading SMART needs root, so the root helper does it, as the one action on its fixed list that only reads: `drives`. Hourly, and at start, the agent leaves a `drives` request. The helper runs `smartctl --scan-open` to find the drives itself (the request names none), reads each with `smartctl --json -a -d <type> <dev>`, and writes a summary to `/var/lib/steward-maint/drives.json` (0644). It writes no job result, because a drive read is not a maintenance job. The agent's newest copy rides every sample, as updates do.
|
||||
|
||||
smartctl ≥ 7.0 answers in JSON, the same on every distribution and for SATA, SAS and NVMe drives. A missing or older smartctl is reported (`smartctl not installed`, `… too old`) and its text output is never parsed. The helper reports figures only:
|
||||
|
||||
- `passed` and `temp_c` for every drive;
|
||||
- `critical_warning`, `life_used_pct`, `media_errors` and spare % against threshold for NVMe;
|
||||
- reallocated / pending / offline-uncorrectable sectors for ATA, and life used where the drive reports it;
|
||||
- grown defects for SCSI.
|
||||
|
||||
A drive with no SMART (a virtual disk, a bridge that hides it) reads `available: false`, and one smartctl couldn't open carries `error`.
|
||||
|
||||
Steward judges the figures in `drives.py`, so the judgment can change without an agent update:
|
||||
|
||||
- **failing**: the drive's self-assessment fails, an NVMe critical warning is raised, or spare blocks fall under threshold;
|
||||
- **worn**: at least 90% of its rated life is used;
|
||||
- **watch**: it has bad sectors or media errors but otherwise passes. Old disks carry a few for years, so a count alone isn't a failure.
|
||||
|
||||
The figures become metrics on `<host>:drive:<device>`: `drive_failing`, `drive_life_used_pct`, `drive_bad_sectors` and `drive_temp_c`. The host gets `drives_failing` and `drive_life_used_max_pct` for covering rules; they are recorded only when at least one drive was read, so a VM's virtual disks don't claim "0 failing". The registration keeps the newest report (`drive_health`), and the host page lists it under **Drive health**, what needs attention first.
|
||||
|
||||
**Installing smartctl on any common distribution.** `files/install-deps.sh` installs smartmontools with whichever package manager the host has: apt (without recommends, which would pull in a mail server), dnf, yum, zypper, pacman or apk. install.sh and the install / provision / update playbooks all fetch and run that one copy. A package that can't be installed is reported and never fails the install.
|
||||
|
||||
### What happened to the host (1.14.0, #5756)
|
||||
|
||||
A post may carry `events`, said once each, with the next post that gets through:
|
||||
|
||||
Reference in New Issue
Block a user