bvandeusen/FabledSteward
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(host_agent): a reboot reads as a reboot, and lost contact as its span (#5756)
CI / lint (push) Successful in 2s
Details
CI / integration (push) Successful in 1m24s
Details
CI / unit (push) Failing after 1m29s
Details
CI / publish (push) Skipped
Details

Browse Source 
#5756 "The agent says when its host rebooted and when it lost contact, so
downtime is recorded as what it was". Agent 1.14.0 posts events with the
next post that gets through: started (with the host's boot time), stopping
(on SIGTERM, with whether systemd says the host is going down) and
lost_contact (from its first unanswered post to the one carrying it).

Steward keeps them as the host's downtime (host_downtime): reboot,
agent_stop, unclean (a boot after the agent was last heard that nobody
announced) or lost_contact. Retried posts apply once.

An announced stop explains the silence for reboot_grace_seconds (Settings
-> Plugins -> host_agent, 10 minutes): seconds_since_seen reads 0 until
then, and the host's other rules go quiet with it (agents_away beside
#5721's registry). Past the grace, agent offline fires.

The host's Agent card lists the last 7 days of downtime. Rows are pruned
at data.retention_days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-14 17:08:16 -04:00
co-authored by Claude Opus 5.5
parent 5dd9956f35
commit c7ff5bc556
16 changed files with 632 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/host-agent-design.md
+11 -1
View File
@@ -113,9 +113,19 @@ Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shut
### Signals and lifecycle
- **SIGHUP** → re-read config file without restart (lets admins change interval/mounts cleanly).
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time → exit.
- **SIGTERM** (from systemd) → finish current poll cycle → flush buffer one last time, with a `stopping` event saying whether the host is going down too (`systemctl is-system-running` says `stopping`) → exit.
- **Systemd restart policy** → `Restart=on-failure`, `RestartSec=10`.
### What happened to the host (1.14.0, #5756)
A post may carry `events`, said once each, with the next post that gets through:
- `started` — the agent starting, with `boot_at` (from `/proc/uptime`).
- `stopping` — on SIGTERM, with `host_shutdown`: true, false, or null when systemd can't say.
- `lost_contact` — `from` the first post that got no answer (or a proxy's 5xx) `to` the post carrying it.
Steward turns them into the host's downtime (`host_downtime`): **reboot** (a stop, then a boot after it), **agent_stop** (the host stayed up), **unclean** (a boot after the agent was last heard that nobody announced: power loss or a crash) and **lost_contact**. An announced stop explains the silence for `reboot_grace_seconds` (Settings → Plugins → host_agent, 10 minutes): `seconds_since_seen` reads 0 until then, and the host's other rules go quiet with it, as when it is offline. Past the grace, agent offline fires. The events live in memory, so a span the agent saw is lost if it restarts mid-outage; the boot time still reads the restart as unclean.
### Logging
To stderr only (systemd captures to journal). No file logging, no log rotation. Levels: